Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.2
-
None
Description
There are some points to improve the performance of YARN Registry DNS Server.
- Do not print unnecessary logs (It just needs change log4j.properties)
- log4j.logger.org.apache.hadoop.registry.server.dns=WARN
- Change some loglevels at points which can affect performance degradation.
- Use "newFixedThreadPool" instead of "newCachedThreadPool" to prevent OOM
- Use Blocking on TCP handler. Using non-blocking and sleeping some time("Thread.sleep(500)") is meaningless.
In our environment, QPS of original yarn dns server is about 5000 (UDP), 100 (TCP).
Now, QPS of our improved yarn dns server is about 47000 (UDP), 500 (TCP).
I will make a pull request at https://github.com/apache/hadoop soon.
Attachments
Issue Links
- links to