Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-326

Add a lifecycle interface for Hadoop components: namenodes, job clients, etc.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None

    Description

      I'd like to propose we have a standard interface for hadoop components, the things that get started or stopped when you bring up a namenode. currently, some of these classes have a stop() or shutdown() method, with no standard name/interface, but no way of seeing if they are live, checking their health of shutting them down reliably. Indeed, there is a tendency for the spawned threads to not want to die; to require the entire process to be killed to stop the workers.

      Having a standard interface would make it easier for

      • management tools to manage the different things
      • monitoring the state of things
      • subclassing

      The latter is interesting as right now TaskTracker and JobTracker start up threads in their constructor; that's very dangerous as subclasses may have their methods called before they are full initialised. Adding this interface would be the right time to clean up the startup process so that subclassing is less risky.

      Attachments

        1. AbstractHadoopComponent.java
          3 kB
          Steve Loughran
        2. hadoop-3628.patch
          49 kB
          Steve Loughran
        3. hadoop-3628.patch
          55 kB
          Steve Loughran
        4. hadoop-3628.patch
          70 kB
          Steve Loughran
        5. hadoop-3628.patch
          55 kB
          Steve Loughran
        6. hadoop-3628.patch
          96 kB
          Steve Loughran
        7. hadoop-3628.patch
          121 kB
          Steve Loughran
        8. hadoop-3628.patch
          147 kB
          Steve Loughran
        9. hadoop-3628.patch
          152 kB
          Steve Loughran
        10. hadoop-3628.patch
          139 kB
          Steve Loughran
        11. hadoop-3628.patch
          143 kB
          Steve Loughran
        12. hadoop-3628.patch
          142 kB
          Steve Loughran
        13. hadoop-lifecycle.sxw
          143 kB
          Steve Loughran
        14. hadoop-lifecycle.pdf
          316 kB
          Steve Loughran
        15. hadoop-3628.patch
          154 kB
          Steve Loughran
        16. hadoop-3628.patch
          165 kB
          Steve Loughran
        17. hadoop-3628.patch
          164 kB
          Steve Loughran
        18. hadoop-lifecycle-tomw.sxw
          144 kB
          Steve Loughran
        19. hadoop-3628.patch
          164 kB
          Steve Loughran
        20. hadoop-3628.patch
          167 kB
          Steve Loughran
        21. hadoop-3628.patch
          162 kB
          Steve Loughran
        22. HADOOP-3628-18.patch
          126 kB
          Steve Loughran
        23. HADOOP-3628-19.patch
          347 kB
          Steve Loughran
        24. hadoop-lifecycle.pdf
          321 kB
          Steve Loughran
        25. HDFS-326.patch
          17 kB
          Steve Loughran

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-213 Provide a way to query job tracker about its daemon thread's status

          • Major - Major loss of function.
          • Resolved
          depends upon

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-5348 Create a ThrowableWritable for serializing exceptions robustly

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3642 add a HadoopIOException that can be thrown in any method that has IOException on its signature

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Test - A new unit, integration or system test. HADOOP-4059 Add tests that try starting the various hadoop command line scripts

          • Major - Major loss of function.
          • Resolved
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3415 JobEndNotifier isnt synchronized, doesnt check state before acting

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-6194 Add service base class and tests to hadoop-common/util

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. HDFS-239 Brief, baseline namenode health check

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3969 Provide Mechanism to optionally expose public org.apache.hadoop.util.Services APIs

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-233 Integrate TaskTracker with the Service base class

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-6915 Add daemon restart option

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              3 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: