[TAJO-168] infinite loop occurs when QueryMaster is stopping - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.2-incubating
Component/s: QueryMaster, Resource Manager
Labels:
None

Description

See the below log. 'mvn install' sometimes incurs the following infinite loop.

2013-09-09 11:15:10,249 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(153)) - Stopping QueryMasterTask:q_1378692829151_0022
2013-09-09 11:15:10,249 INFO  querymaster.QueryJobManager (QueryJobManager.java:stopQuery(127)) - Stop QueryInProgress:q_1378692829151_0022
2013-09-09 11:15:10,250 INFO  master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:q_1378692829151_0022
2013-09-09 11:15:10,250 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(160)) - Stopped QueryMasterTask:q_1378692829151_0022
2013-09-09 11:15:10,250 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 62808 ms
2013-09-09 11:15:10,250 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 62285 ms
2013-09-09 11:15:10,250 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 63832 ms
2013-09-09 11:15:11,251 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0021 stopped cause query sesstion timeout: 62251 ms
2013-09-09 11:15:11,252 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0022 stopped cause query sesstion timeout: 61742 ms
2013-09-09 11:15:11,252 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 63810 ms
2013-09-09 11:15:11,252 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 63287 ms
2013-09-09 11:15:11,252 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 64834 ms
2013-09-09 11:15:12,253 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0021 stopped cause query sesstion timeout: 63253 ms
2013-09-09 11:15:12,253 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0022 stopped cause query sesstion timeout: 62743 ms
2013-09-09 11:15:12,253 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0025 stopped cause query sesstion timeout: 60704 ms
2013-09-09 11:15:12,254 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(153)) - Stopping QueryMasterTask:q_1378692829151_0025
2013-09-09 11:15:12,254 INFO  querymaster.QueryJobManager (QueryJobManager.java:stopQuery(127)) - Stop QueryInProgress:q_1378692829151_0025
2013-09-09 11:15:12,255 INFO  master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:q_1378692829151_0025
2013-09-09 11:15:12,255 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(160)) - Stopped QueryMasterTask:q_1378692829151_0025
2013-09-09 11:15:12,255 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0026 stopped cause query sesstion timeout: 60197 ms
2013-09-09 11:15:12,255 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(153)) - Stopping QueryMasterTask:q_1378692829151_0026
2013-09-09 11:15:12,255 INFO  master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:q_1378692829151_0026
2013-09-09 11:15:12,255 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(160)) - Stopped QueryMasterTask:q_1378692829151_0026
2013-09-09 11:15:12,255 INFO  querymaster.QueryJobManager (QueryJobManager.java:stopQuery(127)) - Stop QueryInProgress:q_1378692829151_0026
2013-09-09 11:15:12,256 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 64814 ms
2013-09-09 11:15:12,256 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 64291 ms
2013-09-09 11:15:12,256 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 65838 ms
2013-09-09 11:15:13,257 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0021 stopped cause query sesstion timeout: 64257 ms
2013-09-09 11:15:13,257 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0022 stopped cause query sesstion timeout: 63747 ms
2013-09-09 11:15:13,257 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0025 stopped cause query sesstion timeout: 61708 ms
2013-09-09 11:15:13,257 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0026 stopped cause query sesstion timeout: 61199 ms
2013-09-09 11:15:13,257 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0027 stopped cause query sesstion timeout: 60685 ms
2013-09-09 11:15:13,257 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(153)) - Stopping QueryMasterTask:q_1378692829151_0027
2013-09-09 11:15:13,258 INFO  querymaster.QueryJobManager (QueryJobManager.java:stopQuery(127)) - Stop QueryInProgress:q_1378692829151_0027
2013-09-09 11:15:13,258 INFO  master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:q_1378692829151_0027
2013-09-09 11:15:13,258 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(160)) - Stopped QueryMasterTask:q_1378692829151_0027
2013-09-09 11:15:13,258 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 65816 ms
2013-09-09 11:15:13,259 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 65294 ms
2013-09-09 11:15:13,259 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 66841 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0021 stopped cause query sesstion timeout: 65260 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0022 stopped cause query sesstion timeout: 64750 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0025 stopped cause query sesstion timeout: 62711 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0026 stopped cause query sesstion timeout: 62202 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0027 stopped cause query sesstion timeout: 61688 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 66818 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 66295 ms
2013-09-09 11:15:14,260 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 67842 ms
2013-09-09 11:15:15,261 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0021 stopped cause query sesstion timeout: 66261 ms
2013-09-09 11:15:15,262 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0022 stopped cause query sesstion timeout: 65752 ms
2013-09-09 11:15:15,262 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0025 stopped cause query sesstion timeout: 63713 ms
2013-09-09 11:15:15,262 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0026 stopped cause query sesstion timeout: 63204 ms
2013-09-09 11:15:15,262 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0027 stopped cause query sesstion timeout: 62690 ms
2013-09-09 11:15:15,262 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 67820 ms
2013-09-09 11:15:15,262 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 67297 ms
2013-09-09 11:15:15,262 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 68844 ms
2013-09-09 11:15:16,263 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0021 stopped cause query sesstion timeout: 67263 ms
2013-09-09 11:15:16,263 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0022 stopped cause query sesstion timeout: 66753 ms
2013-09-09 11:15:16,263 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0025 stopped cause query sesstion timeout: 64714 ms
2013-09-09 11:15:16,263 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0026 stopped cause query sesstion timeout: 64205 ms
2013-09-09 11:15:16,264 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0027 stopped cause query sesstion timeout: 63692 ms
2013-09-09 11:15:16,264 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 68822 ms
2013-09-09 11:15:16,264 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 68299 ms
2013-09-09 11:15:16,264 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 69846 ms
2013-09-09 11:15:17,265 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0021 stopped cause query sesstion timeout: 68265 ms
2013-09-09 11:15:17,266 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0022 stopped cause query sesstion timeout: 67756 ms
2013-09-09 11:15:17,266 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0025 stopped cause query sesstion timeout: 65717 ms
2013-09-09 11:15:17,266 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0026 stopped cause query sesstion timeout: 65208 ms
2013-09-09 11:15:17,266 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0027 stopped cause query sesstion timeout: 64694 ms
2013-09-09 11:15:17,266 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0018 stopped cause query sesstion timeout: 69824 ms
2013-09-09 11:15:17,266 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0019 stopped cause query sesstion timeout: 69301 ms
2013-09-09 11:15:17,266 WARN  querymaster.QueryMaster (QueryMaster.java:run(325)) - Query q_1378692829151_0017 stopped cause query sesstion timeout: 70848 ms

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

TAJO-168.patch
12/Sep/13 12:33
9 kB
Jinho Kim

Activity

People

Assignee:: Jinho Kim

Reporter:: Hyunsik Choi

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 09/Sep/13 02:17

Updated:: 12/Sep/13 17:29

Resolved:: 12/Sep/13 17:01