Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14521

KMS client needs retry logic

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.6.0
    • 2.9.0, 2.8.3, 3.0.0
    • None
    • None

    Description

      The kms client appears to have no retry logic – at all. It's completely decoupled from the ipc retry logic. This has major impacts if the KMS is unreachable for any reason, including but not limited to network connection issues, timeouts, the restart during an upgrade.

      This has some major ramifications:

      1. Jobs may fail to submit, although oozie resubmit logic should mask it
      2. Non-oozie launchers may experience higher rates if they do not already have retry logic.
      3. Tasks reading EZ files will fail, probably be masked by framework reattempts
      4. EZ file creation fails after creating a 0-length file – client receives EDEK in the create response, then fails when decrypting the EDEK
      5. Bulk hadoop fs copies, and maybe distcp, will prematurely fail

      Attachments

        1. HADOOP-14521.09.patch
          31 kB
          Xiao Chen
        2. HADOOP-14521.11.patch
          31 kB
          Xiao Chen
        3. HADOOP-14521-branch-2.8.002.patch
          31 kB
          Rushabh Shah
        4. HADOOP-14521-branch-2.8.2.patch
          31 kB
          Rushabh Shah
        5. HADOOP-14521-trunk-10.patch
          31 kB
          Rushabh Shah
        6. HDFS-11804-branch-2.8.patch
          30 kB
          Rushabh Shah
        7. HDFS-11804-trunk.patch
          22 kB
          Rushabh Shah
        8. HDFS-11804-trunk-1.patch
          22 kB
          Rushabh Shah
        9. HDFS-11804-trunk-2.patch
          24 kB
          Rushabh Shah
        10. HDFS-11804-trunk-3.patch
          24 kB
          Rushabh Shah
        11. HDFS-11804-trunk-4.patch
          24 kB
          Rushabh Shah
        12. HDFS-11804-trunk-5.patch
          27 kB
          Rushabh Shah
        13. HDFS-11804-trunk-6.patch
          30 kB
          Rushabh Shah
        14. HDFS-11804-trunk-7.patch
          30 kB
          Rushabh Shah
        15. HDFS-11804-trunk-8.patch
          30 kB
          Rushabh Shah

        Activity

          People

            shahrs87 Rushabh Shah
            shahrs87 Rushabh Shah
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: