Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
-
None
Description
When testing HIVE-7629 in build 279 I noticed that the test was taking much longer than normal (2 hours instead of 1) and that TestMiniTezCliDriver was the only test still running. Below is the process listing.
As you can see below, it was running the following .q files:
tez_joins_explain.q vector_decimal_aggregate.q mapjoin_mapjoin.q groupby2.q join1.q bucket_map_join_tez1.q tez_union.q tez_dml.q dynpart_sort_optimization.q orc_merge2.q load_dyn_part1.q disable_merge_for_bucketing.q filter_join_breaktask2.q ctas.q orc_merge3.q
502 15573 0.0 0.0 9236 1224 ? Ss 16:11 0:00 | \_ bash /home/hiveptest//54.221.124.118-hiveptest-2/scratch/hiveptest-TestMiniTezCliDriver-tez_joins_explain.q-vector_decimal_aggregate.q-mapjoin_mapjoin.q-and-12-more.sh 502 15628 0.0 0.0 4128 636 ? S 16:11 0:00 | \_ timeout 2h mvn -B -o test -Dmaven.repo.local=/home/hiveptest//54.221.124.118-hiveptest-2/maven -Phadoop-2 -Phadoop-2 -Dtest=TestMiniTezCliDriver -Dqfile=tez_joins_explain.q,vector_decimal_aggregate.q,mapjoin_mapjoin.q,groupby2.q,join1.q,bucket_map_join_tez1.q,tez_union.q,tez_dml.q,dynpart_sort_optimization.q,orc_merge2.q,load_dyn_part1.q,disable_merge_for_bucketing.q,filter_join_breaktask2.q,ctas.q,orc_merge3.q 502 15629 0.6 4.5 4375516 703136 ? Sl 16:11 0:42 | \_ /usr/java/jdk1.7.0_45-cloudera/bin/java -Xmx2g -XX:MaxPermSize=256M -classpath /usr/local/apache-maven-3.0.5/boot/plexus-classworlds-2.4.jar -Dclassworlds.conf=/usr/local/apache-maven-3.0.5/bin/m2.conf -Dmaven.home=/usr/local/apache-maven-3.0.5 org.codehaus.plexus.classworlds.launcher.Launcher -B -o test -Dmaven.repo.local=/home/hiveptest//54.221.124.118-hiveptest-2/maven -Phadoop-2 -Phadoop-2 -Dtest=TestMiniTezCliDriver -Dqfile=tez_joins_explain.q,vector_decimal_aggregate.q,mapjoin_mapjoin.q,groupby2.q,join1.q,bucket_map_join_tez1.q,tez_union.q,tez_dml.q,dynpart_sort_optimization.q,orc_merge2.q,load_dyn_part1.q,disable_merge_for_bucketing.q,filter_join_breaktask2.q,ctas.q,orc_merge3.q 502 15866 0.0 0.0 106112 1264 ? S 16:12 0:00 | \_ /bin/sh -c cd /home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest && /usr/java/jdk1.7.0_45-cloudera/jre/bin/java -Xmx2048m -XX:MaxPermSize=512m -jar /home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/surefire/surefirebooter8865040196821476054.jar /home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/surefire/surefire2549852945539040510tmp /home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/surefire/surefire_08576952877364320449tmp 502 15867 3.3 4.2 7613168 647740 ? Sl 16:12 3:31 | \_ /usr/java/jdk1.7.0_45-cloudera/jre/bin/java -Xmx2048m -XX:MaxPermSize=512m -jar /home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/surefire/surefirebooter8865040196821476054.jar /home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/surefire/surefire2549852945539040510tmp /home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/surefire/surefire_08576952877364320449tmp 502 1426 0.0 0.0 106264 1332 ? Ss 16:14 0:00 | \_ /bin/bash -c /usr/java/jdk1.7.0_45-cloudera/bin/java -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-1_0/application_1407874332889_0006/container_1407874332889_0006_01_000002 -Dtez.root.logger=INFO,CLA -Xmx1024m -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-1_0/application_1407874332889_0006/container_1407874332889_0006_01_000002/stdout 2>/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-1_0/application_1407874332889_0006/container_1407874332889_0006_01_000002/stderr 502 1442 0.7 1.7 5626068 272824 ? Sl 16:14 0:47 | | \_ /usr/java/jdk1.7.0_45-cloudera/bin/java -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-1_0/application_1407874332889_0006/container_1407874332889_0006_01_000002 -Dtez.root.logger=INFO,CLA -Xmx1024m -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session 502 2828 0.0 0.0 106268 1348 ? Ss 16:15 0:00 | \_ /bin/bash -c /usr/java/jdk1.7.0_45-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-0_3/application_1407874332889_0006/container_1407874332889_0006_01_000003 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-localDir-nm-0_1/usercache/hiveptest/appcache/application_1407874332889_0006/container_1407874332889_0006_01_000003/tmp org.apache.hadoop.mapred.YarnTezDagChild 10.69.139.135 55252 container_1407874332889_0006_01_000003 application_1407874332889_0006 1 1>/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-0_3/application_1407874332889_0006/container_1407874332889_0006_01_000003/stdout 2>/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-0_3/application_1407874332889_0006/container_1407874332889_0006_01_000003/stderr 502 2842 0.6 1.5 3467232 238608 ? Sl 16:15 0:40 | \_ /usr/java/jdk1.7.0_45-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-logDir-nm-0_3/application_1407874332889_0006/container_1407874332889_0006_01_000003 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=/home/hiveptest/54.221.124.118-hiveptest-2/apache-svn-trunk-source/itests/qtest/target/hive/hive-localDir-nm-0_1/usercache/hiveptest/appcache/application_1407874332889_0006/container_1407874332889_0006_01_000003/tmp org.apache.hadoop.mapred.YarnTezDagChild 10.69.139.135 55252 container_1407874332889_0006_01_000003 application_1407874332889_0006 1
The test result which was impacted by this is here: https://issues.apache.org/jira/browse/HIVE-7629?focusedCommentId=14094778&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14094778
