[YARN-10850] TimelineService v2 lists containers for all attempts when filtering for one - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.4.0, 3.2.4, 3.3.5
Component/s: timelinereader
Labels:
- pull-request-available

Hadoop Flags:

Reviewed

Description

When using the command

yarn container -list <app_attempt_id>

with an application attempt ID based on the help only the containers for that attempt should be listed.

-list <Application Name or Attempt ID>    List containers for application
                                          attempt when application
                                          attempt ID is provided. When
                                          application name is provided,
                                          then it finds the instances of
                                          the application based on app's
                                          own implementation, and
                                          -appTypes option must be
                                          specified unless it is the
                                          default yarn-service type. With
                                          app name, it supports optional
                                          use of -version to filter
                                          instances based on app version,
                                          -components to filter instances
                                          based on component names,
                                          -states to filter instances
                                          based on instance state.

When TimelineService v2 is enabled all of the containers for the application are returned.

hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn applicationattempt -list application_1625124233002_0007
21/07/01 09:32:23 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
21/07/01 09:32:24 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
21/07/01 09:32:24 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Total number of application attempts :2
         ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
appattempt_1625124233002_0007_000001	              FAILED	container_e43_1625124233002_0007_01_000001	http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/
appattempt_1625124233002_0007_000002	              KILLED	container_e43_1625124233002_0007_02_000001	http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/

Querying the 2 app attempts produces the same output:

hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000001
21/07/01 09:32:35 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
21/07/01 09:32:35 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
21/07/01 09:32:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
21/07/01 09:32:36 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
Total number of containers :12
                  Container-Id	          Start Time	         Finish Time	               State	                Host	   Node Http Address	                            LOG-URL
container_e43_1625124233002_0007_02_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
container_e43_1625124233002_0007_02_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
container_e43_1625124233002_0007_02_000003	                 N/A	                 N/A	                null	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
container_e43_1625124233002_0007_02_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
container_e43_1625124233002_0007_02_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
container_e43_1625124233002_0007_02_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
container_e43_1625124233002_0007_01_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
container_e43_1625124233002_0007_01_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
container_e43_1625124233002_0007_01_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
container_e43_1625124233002_0007_01_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
container_e43_1625124233002_0007_01_000003	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
container_e43_1625124233002_0007_01_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa

hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000002
21/07/01 09:32:50 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
21/07/01 09:32:51 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
21/07/01 09:32:51 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
21/07/01 09:32:51 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
Total number of containers :12
                  Container-Id	          Start Time	         Finish Time	               State	                Host	   Node Http Address	                            LOG-URL
container_e43_1625124233002_0007_02_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
container_e43_1625124233002_0007_02_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
container_e43_1625124233002_0007_02_000003	                 N/A	                 N/A	                null	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
container_e43_1625124233002_0007_02_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
container_e43_1625124233002_0007_02_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
container_e43_1625124233002_0007_02_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
container_e43_1625124233002_0007_01_000005	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
container_e43_1625124233002_0007_01_000004	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
container_e43_1625124233002_0007_01_000002	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
container_e43_1625124233002_0007_01_000006	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
container_e43_1625124233002_0007_01_000003	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
container_e43_1625124233002_0007_01_000001	                 N/A	                 N/A	            COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042	http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa

The reason for this is that the container collector method in TimelineServiceV2 gets the application ID from the attempt ID and collects the containers for the application.

The other implementations however are using the application attempt ID: ApplicationHistoryManagerOnTimelineStore and ApplicationHistoryManagerImpl.

TSv2 should behave similarly to v1.