Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17861

improve YARN Registry DNS Server qps

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.2
    • None
    • registry

    Description

      There are some points to improve the performance of YARN Registry DNS Server.

      • Do not print unnecessary logs (It just needs change log4j.properties)
        • log4j.logger.org.apache.hadoop.registry.server.dns=WARN
      • Change some loglevels at points which can affect performance degradation.
      • Use "newFixedThreadPool" instead of "newCachedThreadPool" to prevent OOM
      • Use Blocking on TCP handler. Using non-blocking and sleeping some time("Thread.sleep(500)") is meaningless.

      In our environment, QPS of original yarn dns server is about 5000 (UDP), 100 (TCP).
      Now, QPS of our improved yarn dns server is about 47000 (UDP), 500 (TCP).

      I will make a pull request at https://github.com/apache/hadoop soon.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              eub YUBI LEE
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h