Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.9.0, 3.0.0-alpha2
-
None
-
Reviewed
Description
Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY (yarn.dispatcher.exit-on-error) always be true no matter what value in configuration files. This misleads users. Two solutions:
- Remove the configuration item and provide a method to allow exitOnDispatchException/shouldExitOnError to be false to enable related unit tests. There is no need for false value in a real daemon since daemons should crash if its dispatcher quit.
- Make it default true instead of false, so that we don't need to hard code it to be true in RM and NM, it is still configurable, and also provide method to enable related unit tests.
Other than that, the code around it needs to refactor. public static final for a variable of interface isn't necessary, and YARN related configure item should be in class YarnConfiguration.
Attachments
Attachments
- YARN-6202.001.patch
- 16 kB
- Yufei Gu
- YARN-6202.002.patch
- 20 kB
- Yufei Gu
- YARN-6202.003.patch
- 21 kB
- Yufei Gu
- YARN-6202.004.patch
- 22 kB
- Yufei Gu
Issue Links
- relates to
-
MAPREDUCE-3634 All daemons should crash instead of hanging around when their EventHandlers get exceptions
- Resolved
Activity
Uploaded patch v1.
Remove configuration items: Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY, and add methods for AsyncDispatcher and EventDispatcher to disable exit on exception/error, which are invoked by unit tests.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 13s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
0 | mvndep | 0m 33s | Maven dependency ordering for branch |
+1 | mvninstall | 15m 46s | trunk passed |
+1 | compile | 19m 2s | trunk passed |
+1 | checkstyle | 2m 13s | trunk passed |
+1 | mvnsite | 3m 14s | trunk passed |
+1 | mvneclipse | 1m 52s | trunk passed |
+1 | findbugs | 5m 9s | trunk passed |
+1 | javadoc | 2m 31s | trunk passed |
0 | mvndep | 0m 19s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 33s | the patch passed |
+1 | compile | 13m 6s | the patch passed |
+1 | javac | 13m 6s | the patch passed |
-0 | checkstyle | 2m 1s | root: The patch generated 2 new + 313 unchanged - 5 fixed = 315 total (was 318) |
+1 | mvnsite | 3m 1s | the patch passed |
+1 | mvneclipse | 1m 50s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 6m 44s | the patch passed |
+1 | javadoc | 2m 28s | the patch passed |
+1 | unit | 2m 37s | hadoop-yarn-common in the patch passed. |
-1 | unit | 0m 39s | hadoop-yarn-server-nodemanager in the patch failed. |
-1 | unit | 0m 45s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | unit | 9m 6s | hadoop-mapreduce-client-app in the patch passed. |
+1 | unit | 3m 16s | hadoop-mapreduce-client-hs in the patch passed. |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. |
124m 28s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:a9ad5d6 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12861109/YARN-6202.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 5886c5f6133c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 4966a6e |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/15426/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15426/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15426/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/15426/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: . |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/15426/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The compiler errors are weird. I don't have them locally. My guess is the Hadoop QA partially compile the project.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 17s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
0 | mvndep | 0m 16s | Maven dependency ordering for branch |
+1 | mvninstall | 14m 19s | trunk passed |
+1 | compile | 17m 19s | trunk passed |
+1 | checkstyle | 2m 9s | trunk passed |
+1 | mvnsite | 3m 8s | trunk passed |
+1 | mvneclipse | 1m 50s | trunk passed |
+1 | findbugs | 5m 54s | trunk passed |
+1 | javadoc | 2m 38s | trunk passed |
0 | mvndep | 0m 18s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 55s | the patch passed |
+1 | compile | 18m 12s | the patch passed |
+1 | javac | 18m 12s | the patch passed |
-0 | checkstyle | 2m 13s | root: The patch generated 2 new + 313 unchanged - 5 fixed = 315 total (was 318) |
+1 | mvnsite | 3m 19s | the patch passed |
+1 | mvneclipse | 2m 9s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 5m 49s | the patch passed |
+1 | javadoc | 2m 49s | the patch passed |
+1 | unit | 2m 38s | hadoop-yarn-common in the patch passed. |
-1 | unit | 12m 29s | hadoop-yarn-server-nodemanager in the patch failed. |
-1 | unit | 40m 27s | hadoop-yarn-server-resourcemanager in the patch failed. |
-1 | unit | 10m 19s | hadoop-mapreduce-client-app in the patch failed. |
+1 | unit | 4m 16s | hadoop-mapreduce-client-hs in the patch passed. |
+1 | asflicense | 0m 51s | The patch does not generate ASF License warnings. |
182m 27s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | |
Timed out junit tests | org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService |
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery | |
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.TestContainer | |
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl | |
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore | |
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore | |
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore | |
org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:a9ad5d6 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12861109/YARN-6202.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 19015b68a570 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 7297e82 |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/15441/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15441/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15441/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15441/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/15441/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: . |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/15441/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 9 new or modified test files. |
0 | mvndep | 0m 16s | Maven dependency ordering for branch |
+1 | mvninstall | 13m 41s | trunk passed |
+1 | compile | 16m 5s | trunk passed |
+1 | checkstyle | 2m 0s | trunk passed |
+1 | mvnsite | 2m 47s | trunk passed |
+1 | mvneclipse | 1m 42s | trunk passed |
+1 | findbugs | 4m 46s | trunk passed |
+1 | javadoc | 2m 14s | trunk passed |
0 | mvndep | 0m 17s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 10s | the patch passed |
+1 | compile | 15m 40s | the patch passed |
+1 | javac | 15m 40s | the patch passed |
-0 | checkstyle | 2m 1s | root: The patch generated 4 new + 359 unchanged - 5 fixed = 363 total (was 364) |
+1 | mvnsite | 3m 2s | the patch passed |
+1 | mvneclipse | 1m 57s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 5m 59s | the patch passed |
+1 | javadoc | 2m 37s | the patch passed |
+1 | unit | 2m 43s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 21s | hadoop-yarn-server-nodemanager in the patch passed. |
-1 | unit | 38m 40s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | unit | 8m 54s | hadoop-mapreduce-client-app in the patch passed. |
+1 | unit | 3m 25s | hadoop-mapreduce-client-hs in the patch passed. |
+1 | asflicense | 0m 42s | The patch does not generate ASF License warnings. |
170m 34s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:a9ad5d6 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12861341/YARN-6202.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 50e9ed2c623a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / b58777a |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/15447/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15447/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/15447/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: . |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/15447/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 17s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 9 new or modified test files. |
0 | mvndep | 0m 14s | Maven dependency ordering for branch |
+1 | mvninstall | 14m 6s | trunk passed |
+1 | compile | 16m 56s | trunk passed |
+1 | checkstyle | 2m 4s | trunk passed |
+1 | mvnsite | 2m 56s | trunk passed |
+1 | mvneclipse | 1m 43s | trunk passed |
+1 | findbugs | 5m 3s | trunk passed |
+1 | javadoc | 2m 17s | trunk passed |
0 | mvndep | 0m 16s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 14s | the patch passed |
+1 | compile | 14m 45s | the patch passed |
+1 | javac | 14m 45s | the patch passed |
-0 | checkstyle | 2m 2s | root: The patch generated 5 new + 359 unchanged - 5 fixed = 364 total (was 364) |
+1 | mvnsite | 3m 5s | the patch passed |
+1 | mvneclipse | 2m 3s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 6m 17s | the patch passed |
+1 | javadoc | 2m 33s | the patch passed |
+1 | unit | 2m 30s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 10s | hadoop-yarn-server-nodemanager in the patch passed. |
-1 | unit | 38m 40s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | unit | 8m 44s | hadoop-mapreduce-client-app in the patch passed. |
+1 | unit | 3m 14s | hadoop-mapreduce-client-hs in the patch passed. |
+1 | asflicense | 0m 40s | The patch does not generate ASF License warnings. |
171m 18s |
Reason | Tests |
---|---|
Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:a9ad5d6 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12861342/YARN-6202.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux e25de5ee9382 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / b58777a |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/15448/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15448/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/15448/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: . |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/15448/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Reading comments above Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY, it seems like DISPATCHER_EXIT_ON_ERROR_KEY is deliberately set to false by default (see comment at https://issues.apache.org/jira/browse/MAPREDUCE-3634?focusedCommentId=13195964&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13195964).
YARN related configure item should be in class YarnConfiguration
This property is not a user-facing property, only frameworks and tests use it, I think. so whether it should go into YarnConfiguration is debatable.
You are right. No matter what default value is, this shouldn't be a configuration item at all. I saw people actually configure it, and thought the items could work! The reality is the daemons should always be true, and some tests need it to be false no matter what configuration is.
I think make the default to be true should be better since it should be true for daemons. The only downside is we need to explicitly set it to false in some test cases, which is mitigated by following cases:
- All unit tests using dispatcher in MockRM is fine since we set it to false in MockRM.
- All unit tests using DrainDispatcher is fine since we set it false in DrainDispatcher as well.
AsyncDispatcher are also used in a lot of other tests besides MockRM, so it will be a lot of test changes.
few tests are crashing without reporting a junit failure due to sys-exit and so are hiding code bugs
Based on this comment in MAPREDUCE-3634, I still think it is better to leave it as false by default, since it is very easy to forget to set this in a new test that uses AsyncDispatcher.
It won't hide code bugs since the unit tests won't pass if AsyncDispatcher crashed. People who use AsyncDispatcher assume it crash on exceptions. Hiding this is more dangerous by making it false by default.
Per offline discussion with Yufei, post to YARN-2917, AsyncDispatcher thread crash will cause JVM to exit(-1), so it is no longer valid that setting DISPATCHER_EXIT_ON_ERROR_KEY to true by default will hide unit test failures. Thus, +1 on this change.
Thanks for tackling this, yufeigu. I agree with the general approach. Can you clean up the unused imports in the checkstyle warning? Also, the unit tests aren't passing:
Running org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.344 sec - in org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Results : Tests run: 1620, Failures: 0, Errors: 0, Skipped: 3 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 38:21.624s [INFO] Finished at: Fri Mar 31 01:15:42 UTC 2017 [INFO] Final Memory: 18M/250M [INFO] ------------------------------------------------------------------------ [WARNING] The requested profile "parallel-tests" could not be activated because it does not exist. [WARNING] The requested profile "native" could not be activated because it does not exist. [WARNING] The requested profile "yarn-ui" could not be activated because it does not exist. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-yarn-server-resourcemanager: ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager && /usr/lib/jvm/java-8-oracle/jre/bin/java -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -jar /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefirebooter8156637588310801099.jar /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire3523297314551279161tmp /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire_124814350333586959477tmp [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Thanks templedf for the review. Uploaded patch v3 to fix style issues. The unit test failure is unrelated.
My bad. I test the wrong uni test, which is TestAMRestart. The failure happens in class TestZKRMStateStore. Uploaded patch v4 for that.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 3m 54s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
0 | mvndep | 1m 46s | Maven dependency ordering for branch |
+1 | mvninstall | 15m 36s | trunk passed |
+1 | compile | 16m 56s | trunk passed |
+1 | checkstyle | 2m 2s | trunk passed |
+1 | mvnsite | 2m 52s | trunk passed |
+1 | mvneclipse | 1m 52s | trunk passed |
+1 | findbugs | 4m 32s | trunk passed |
+1 | javadoc | 2m 19s | trunk passed |
0 | mvndep | 0m 16s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 17s | the patch passed |
+1 | compile | 14m 53s | the patch passed |
+1 | javac | 14m 53s | the patch passed |
-0 | checkstyle | 2m 0s | root: The patch generated 2 new + 365 unchanged - 5 fixed = 367 total (was 370) |
+1 | mvnsite | 3m 20s | the patch passed |
+1 | mvneclipse | 2m 8s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 5m 31s | the patch passed |
+1 | javadoc | 2m 34s | the patch passed |
+1 | unit | 2m 32s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 5s | hadoop-yarn-server-nodemanager in the patch passed. |
-1 | unit | 39m 59s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | unit | 9m 8s | hadoop-mapreduce-client-app in the patch passed. |
+1 | unit | 3m 28s | hadoop-mapreduce-client-hs in the patch passed. |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. |
178m 57s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:612578f |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12863168/YARN-6202.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 7f87b577b9ac 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 0cab572 |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/15616/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/15616/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/15616/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: . |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/15616/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
0 | mvndep | 1m 22s | Maven dependency ordering for branch |
+1 | mvninstall | 13m 59s | trunk passed |
+1 | compile | 16m 32s | trunk passed |
+1 | checkstyle | 2m 1s | trunk passed |
+1 | mvnsite | 3m 9s | trunk passed |
+1 | mvneclipse | 1m 55s | trunk passed |
+1 | findbugs | 4m 45s | trunk passed |
+1 | javadoc | 2m 16s | trunk passed |
0 | mvndep | 0m 18s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 32s | the patch passed |
+1 | compile | 14m 50s | the patch passed |
+1 | javac | 14m 50s | the patch passed |
-0 | checkstyle | 2m 52s | root: The patch generated 2 new + 365 unchanged - 5 fixed = 367 total (was 370) |
+1 | mvnsite | 5m 8s | the patch passed |
+1 | mvneclipse | 3m 30s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 5m 38s | the patch passed |
+1 | javadoc | 2m 36s | the patch passed |
+1 | unit | 2m 36s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 26s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 39m 25s | hadoop-yarn-server-resourcemanager in the patch passed. |
+1 | unit | 8m 43s | hadoop-mapreduce-client-app in the patch passed. |
+1 | unit | 3m 27s | hadoop-mapreduce-client-hs in the patch passed. |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. |
177m 15s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:612578f |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12863168/YARN-6202.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 0a1cbcbf6eec 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 0cab572 |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/15617/artifact/patchprocess/diff-checkstyle-root.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/15617/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: . |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/15617/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
LGTM. Would you mind filing newbie JIRAs to resolve those two checkstyle issues?
Thanks for the patch, yufeigu, and, haibochen, for the review. Committed to trunk. If you want it in branch-2, I'll need another patch for it.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11610 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11610/)
YARN-6202. Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is (templedf: rev dd43b895c2e50fa97cb7327be77509b87dad1823)
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/DrainDispatcher.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java
- (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMDispatcher.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
- (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/Dispatcher.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/EventDispatcher.java
Thanks templedf for the review and commit. Thanks haibochen for the review. I saw other projects use these configuration items, better to not commit this to branch2.
Option 1 sounds good to me. Any thoughts? Vinod Kumar Vavilapalli
I saw other projects use these configuration items, better to not commit this to branch2.
Agree. An example is:
TEZ-2049. Let's keep the patch in trunk only.
Folks, a bunch of downstream are using this. This is not the kind of cleanup that will balance out code-cleanup vs downstream pain. Let's just revert the removals in the patch and instead deprecate the config as well as the constants.
Let's just revert the removals in the patch and instead deprecate the config as well as the constants.
+1. That sounds like a good plan.
I hunted around, and all I could find was Tez and Tajo. Tajo hasn't been touched in 3 years, and after TEZ-2049 it's no longer used by TEZ. In fact, this patch is almost the same as the what was done in TEZ-2049. Do you know of other projects that are using it, vinodkv?
yufeigu I agree that exitOnDispatchException should be true in most scene. But some unimportant component, for examaple dispatcher in CommonNodeLabelsManager. I think that set exitOnDispatchException to true is unnecessary. NodeLabel command failed, should not make resourcemanager crash.Maybe I think we should provide a configuration item to user.
Option 1 sounds good to me. Any thoughts? vinodkv