[FLINK-32552] Mixed up Flink session job deployments - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Not A Bug
Affects Version/s: None
Fix Version/s: None
Component/s: Kubernetes Operator
Labels:
None

Release Note:
Not a bug of the flink k8s operator.

Description

Context

In the scope of end-to-end tests we deploy all the Flink session jobs we have regularly in a staging environment. Some of the jobs are bundled together in one helm chart and therefore deployed at the same time. There are around 40 individual Flink jobs (running on the same Flink session cluster). The session cluster is individual for each e2e test run. The problems described below happen scarcely (1 in ~ 50 run maybe).

Problem

Rarely the operator seems to "mix up" the deployments. This can be seen in the Flink cluster logs as multiple Received JobGraph submission '<JOB NAME>' (<JOB_ID>) logs are created from jobs with the same job_id. This results in errors such as:

DuplicateJobSubmissionException or ClassNotFoundException.

It' also visible in the FlinkSessionJob resource: status.jobStatus.jobName does not match the expected job name of the job being deployed (The job name is passed to the application via argument).

So far we were unable to reliably reproduce the error.

Details

The following lines show the status of 3 jobs form the view point of the Flink cluster dashboard, and the FlinkSessionJob ressource:

aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615

Apache Flink Dashboard:

State: Restarting
ID: a7d36f3881f943a00000000000000002
Exceptions: Cannot load user class: aelps.pipelines.aletsch.smc.SMCUrlMapper

FlinkSessionJob Ressource:

State: RUNNING
jobId: a1221c743367497b0000000000000002
uid: a1221c74-3367-497b-ad2f-8793ab23919d

aletsch_mat_e5730831db8092adb12f5189c4c895ef3a268615

Apache Flink Dashboard:

State: -
ID: -

FlinkSessionJob Ressource:

State: UPGRADING
jobId: -
uid: a7d36f38-81f9-43a0-898f-19b950430e9d

Flink K8s Operator:

Exceptions: DuplicateJobSubmissionException: Job has already been submitted.

aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615

Apache Flink Dashboard:

State: Running
ID: e692c2dfaa18441c0000000000000002
Exceptions: -

FlinkSessionJob Ressource:

State: RUNNING
jobId: e692c2dfaa18441c0000000000000002
uid: e692c2df-aa18-441c-a352-88aefa9a3017

As we can see the aletsch_smc job is presumably running according to the FlinkSessionJob resource, but crash-looping in the cluster and it has the jobID matching the uid of the resource of aletsch_mat. While aletsch_mat is not even running. The following logs also show some suspicious entries: There are several Received JobGraph submission from different jobs with the same jobID.

Logs

The logs are filtered by the 3 jobIds from above.

JobID: a7d36f3881f943a00000000000000002

Flink Cluster
    ...
    023-07-06 10:23:50,552 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002) switched from state RUNNING to RESTARTING.
    2023-07-06 10:23:50	    file: '/tmp/tm_10.0.11.159:6122-e9fadc/blobStorage/job_a7d36f3881f943a00000000000000002/blob_p-40c7a30adef8868254191d2cf2dbc4cb7ab46f0d-8a02a0583d91c5e8e6c94f378aa444c2' (valid JAR)
    2023-07-06 10:23:50,522 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=4}]
    2023-07-06 10:23:50,522 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=3}]
    2023-07-06 10:23:50,522 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=2}]
    2023-07-06 10:23:50,522 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
    2023-07-06 10:23:50,512 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002) switched from state RESTARTING to RUNNING.
    2023-07-06 10:23:48,979 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job a7d36f3881f943a00000000000000002
    2023-07-06 10:23:48,853 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
    2023-07-06 10:23:48,853 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=2}]
    2023-07-06 10:23:48,853 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=3}]
    2023-07-06 10:23:48	    file: '/tmp/tm_10.0.11.159:6122-e9fadc/blobStorage/job_a7d36f3881f943a00000000000000002/blob_p-40c7a30adef8868254191d2cf2dbc4cb7ab46f0d-8a02a0583d91c5e8e6c94f378aa444c2' (valid JAR)
    2023-07-06 10:23:48,661 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002) switched from state RUNNING to RESTARTING.
    2023-07-06 10:23:48,583 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=4}]
    2023-07-06 10:23:48,583 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=3}]
    2023-07-06 10:23:48,583 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=2}]
    2023-07-06 10:23:48,582 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
    2023-07-06 10:23:48,573 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002) switched from state RESTARTING to RUNNING.
    2023-07-06 10:23:47,562 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Received JobGraph submission 'aletsch_mat_e5730831db8092adb12f5189c4c895ef3a268615' (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:47,518 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job a7d36f3881f943a00000000000000002
    2023-07-06 10:23:47,517 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
    2023-07-06 10:23:47,517 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=2}]
    2023-07-06 10:23:47,516 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=3}]
    2023-07-06 10:23:47,463 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:47,463 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job a7d36f3881f943a00000000000000002 is submitted.
    2023-07-06 10:23:47,104 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002) switched from state RUNNING to RESTARTING.
    2023-07-06 10:23:46,804 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Offer reserved slots to the leader of job a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:46,804 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Establish JobManager connection for job a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:46,799 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Successful registration at job manager akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_2 for job a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:46,577 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 221b24b50413805c9e35d7620b8a00b8 for job a7d36f3881f943a00000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:46,577 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 49d3c8cd1080bd38c0144c3d3cc597cd for job a7d36f3881f943a00000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:46,577 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 819f34cc8957066478fb4b3549367d24 for job a7d36f3881f943a00000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:46,574 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Add job a7d36f3881f943a00000000000000002 for job leader monitoring.
    2023-07-06 10:23:46,570 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 36802a7de1487f3fb1b6a3b509bd5e20 for job a7d36f3881f943a00000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:46,560 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a7d36f3881f943a00000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=4}]
    2023-07-06 10:23:46,556 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager aaa9331f70b07a195b5f09d57d1b40c5@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_2 for job a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:46,528 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager aaa9331f70b07a195b5f09d57d1b40c5@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_2 for job a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:46,480 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002) switched from state CREATED to RUNNING.
    2023-07-06 10:23:46,476 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Starting execution of job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a7d36f3881f943a00000000000000002) under job master id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:46,466 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@62877000 for aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:46,079 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Running initialization on master for job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:46,059 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] - Found 0 checkpoints in KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a7d36f3881f943a00000000000000002-config-map'}.
    2023-07-06 10:23:46,051 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] - Recovering checkpoints from KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a7d36f3881f943a00000000000000002-config-map'}.
    2023-07-06 10:23:46,006 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using restart back off time strategy ExponentialDelayRestartBackoffTimeStrategy(initialBackoffMS=1000, maxBackoffMS=300000, backoffMultiplier=2.0, resetBackoffThresholdMS=3600000, jitterFactor=0.5, currentBackoffMS=1000, lastFailureTimestamp=0) for aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:45,987 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Initializing job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:45,966 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Received JobGraph submission 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615' (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:45,965 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Received JobGraph submission 'aletsch_mat_e5730831db8092adb12f5189c4c895ef3a268615' (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:45,915 INFO  org.apache.flink.runtime.jobmanager.DefaultJobGraphStore     [] - Added JobGraph(jobId: a7d36f3881f943a00000000000000002) to KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-cluster-config-map'}.
    2023-07-06 10:23:45,859 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Submitting job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:45,857 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Received JobGraph submission 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a7d36f3881f943a00000000000000002).
    2023-07-06 10:23:45,705 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:45,705 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job a7d36f3881f943a00000000000000002 is submitted.
    2023-07-06 10:23:45,705 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:45,705 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job a7d36f3881f943a00000000000000002 is submitted.
    2023-07-06 10:23:45,705 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
    2023-07-06 10:23:45,705 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job a7d36f3881f943a00000000000000002 is submitted.

    Flink Operator
    2023-07-06 10:26:25,792 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job: a7d36f3881f943a00000000000000002 to session cluster.
    2023-07-06 10:25:05,163 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job: a7d36f3881f943a00000000000000002 to session cluster.
    2023-07-06 10:24:24,553 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job: a7d36f3881f943a00000000000000002 to session cluster.
    2023-07-06 10:24:03,850 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job: a7d36f3881f943a00000000000000002 to session cluster.
    2023-07-06 10:23:53,094 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job: a7d36f3881f943a00000000000000002 to session cluster.
    2023-07-06 10:23:47,346 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job: a7d36f3881f943a00000000000000002 to session cluster.
    2023-07-06 10:23:45,372 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job: a7d36f3881f943a00000000000000002 to session cluster.

JobID: a1221c743367497b0000000000000002

Flink Cluster
    2023-07-06 11:23:48,062 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 1 for job a1221c743367497b0000000000000002 (48548 bytes, checkpointDuration=107 ms, finalizationTime=33 ms).
    2023-07-06 11:23:47,937 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 1 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1688635427922 for job a1221c743367497b0000000000000002.
    2023-07-06 10:23:48,567 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Offer reserved slots to the leader of job a1221c743367497b0000000000000002.
    2023-07-06 10:23:48,567 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Establish JobManager connection for job a1221c743367497b0000000000000002.
    2023-07-06 10:23:48,567 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Successful registration at job manager akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_7 for job a1221c743367497b0000000000000002.
    2023-07-06 10:23:48,009 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request cae6932e2409d5fece3f6b4636e3c71a for job a1221c743367497b0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,003 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 8a57f3ecff07d300aebb33f6b3545aed for job a1221c743367497b0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,003 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 7a4a0cfd16eec4a1cb043cce5f989db0 for job a1221c743367497b0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,002 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Add job a1221c743367497b0000000000000002 for job leader monitoring.
    2023-07-06 10:23:48,002 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 92cbc64513fa703e4acf28bbb3088a58 for job a1221c743367497b0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,999 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job a1221c743367497b0000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=4}]
    2023-07-06 10:23:47,998 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager aaa9331f70b07a195b5f09d57d1b40c5@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_7 for job a1221c743367497b0000000000000002.
    2023-07-06 10:23:47,953 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager aaa9331f70b07a195b5f09d57d1b40c5@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_7 for job a1221c743367497b0000000000000002.
    2023-07-06 10:23:47,922 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a1221c743367497b0000000000000002) switched from state CREATED to RUNNING.
    2023-07-06 10:23:47,887 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Starting execution of job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a1221c743367497b0000000000000002) under job master id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:47,887 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@2222ba4d for aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a1221c743367497b0000000000000002).
    2023-07-06 10:23:47,880 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Running initialization on master for job aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a1221c743367497b0000000000000002).
    2023-07-06 10:23:47,872 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] - Found 0 checkpoints in KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a1221c743367497b0000000000000002-config-map'}.
    2023-07-06 10:23:47,867 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] - Recovering checkpoints from KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a1221c743367497b0000000000000002-config-map'}.
    2023-07-06 10:23:47,832 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using restart back off time strategy ExponentialDelayRestartBackoffTimeStrategy(initialBackoffMS=1000, maxBackoffMS=300000, backoffMultiplier=2.0, resetBackoffThresholdMS=3600000, jitterFactor=0.5, currentBackoffMS=1000, lastFailureTimestamp=0) for aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615 (a1221c743367497b0000000000000002).
    2023-07-06 10:23:47,832 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Initializing job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a1221c743367497b0000000000000002).
    2023-07-06 10:23:47,820 INFO  org.apache.flink.runtime.jobmanager.DefaultJobGraphStore     [] - Added JobGraph(jobId: a1221c743367497b0000000000000002) to KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-cluster-config-map'}.
    2023-07-06 10:23:47,780 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Submitting job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a1221c743367497b0000000000000002).
    2023-07-06 10:23:47,776 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Received JobGraph submission 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615' (a1221c743367497b0000000000000002).
    2023-07-06 10:23:47,668 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=a1221c743367497b0000000000000002.
    2023-07-06 10:23:47,668 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job a1221c743367497b0000000000000002 is submitted.

    Flink Operator
    2023-07-06 10:23:48,007 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-smc-staging-e5730831] Submitted job: a1221c743367497b0000000000000002 to session cluster.
    2023-07-06 10:23:47,505 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-smc-staging-e5730831] Submitting job: a1221c743367497b0000000000000002 to session cluster.
    2023-07-06 10:23:45,416 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-smc-staging-e5730831] Submitting job: a1221c743367497b0000000000000002 to session cluster.

JobID: e692c2dfaa18441c0000000000000002

Flink Cluster
    2023-07-06 11:23:48,004 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 1 for job e692c2dfaa18441c0000000000000002 (8194 bytes, checkpointDuration=125 ms, finalizationTime=28 ms).
    2023-07-06 11:23:47,867 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 1 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1688635427851 for job e692c2dfaa18441c0000000000000002.
    2023-07-06 10:23:48,568 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Offer reserved slots to the leader of job e692c2dfaa18441c0000000000000002.
    2023-07-06 10:23:48,568 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Establish JobManager connection for job e692c2dfaa18441c0000000000000002.
    2023-07-06 10:23:48,568 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Successful registration at job manager akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_6 for job e692c2dfaa18441c0000000000000002.
    2023-07-06 10:23:48,002 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 5e5a0e55fac280bf31abf29a20bce684 for job e692c2dfaa18441c0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,002 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 1cdbce54f4376a1df86430f97dab6858 for job e692c2dfaa18441c0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,002 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request 352db7288d0e4d1775d5f52dd14c769d for job e692c2dfaa18441c0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,001 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Add job e692c2dfaa18441c0000000000000002 for job leader monitoring.
    2023-07-06 10:23:48,000 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive slot request bffed3e4a4c8573049a4119bd7e15f19 for job e692c2dfaa18441c0000000000000002 from resource manager with leader id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:48,998 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job e692c2dfaa18441c0000000000000002: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=4}]
    2023-07-06 10:23:47,998 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager aaa9331f70b07a195b5f09d57d1b40c5@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_6 for job e692c2dfaa18441c0000000000000002.
    2023-07-06 10:23:47,953 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager aaa9331f70b07a195b5f09d57d1b40c5@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_6 for job e692c2dfaa18441c0000000000000002.
    2023-07-06 10:23:47,851 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615 (e692c2dfaa18441c0000000000000002) switched from state CREATED to RUNNING.
    2023-07-06 10:23:47,845 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Starting execution of job 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615' (e692c2dfaa18441c0000000000000002) under job master id aaa9331f70b07a195b5f09d57d1b40c5.
    2023-07-06 10:23:47,844 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@7eeab246 for aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615 (e692c2dfaa18441c0000000000000002).
    2023-07-06 10:23:47,834 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Running initialization on master for job aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615 (e692c2dfaa18441c0000000000000002).
    2023-07-06 10:23:47,825 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] - Found 0 checkpoints in KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-e692c2dfaa18441c0000000000000002-config-map'}.
    2023-07-06 10:23:47,813 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] - Recovering checkpoints from KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-e692c2dfaa18441c0000000000000002-config-map'}.
    2023-07-06 10:23:47,782 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using restart back off time strategy ExponentialDelayRestartBackoffTimeStrategy(initialBackoffMS=1000, maxBackoffMS=300000, backoffMultiplier=2.0, resetBackoffThresholdMS=3600000, jitterFactor=0.5, currentBackoffMS=1000, lastFailureTimestamp=0) for aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615 (e692c2dfaa18441c0000000000000002).
    2023-07-06 10:23:47,781 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Initializing job 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615' (e692c2dfaa18441c0000000000000002).
    2023-07-06 10:23:47,774 INFO  org.apache.flink.runtime.jobmanager.DefaultJobGraphStore     [] - Added JobGraph(jobId: e692c2dfaa18441c0000000000000002) to KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-cluster-config-map'}.
    2023-07-06 10:23:47,703 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Submitting job 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615' (e692c2dfaa18441c0000000000000002).
    2023-07-06 10:23:47,702 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Received JobGraph submission 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615' (e692c2dfaa18441c0000000000000002).
    2023-07-06 10:23:47,650 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=e692c2dfaa18441c0000000000000002.
    2023-07-06 10:23:47,650 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job e692c2dfaa18441c0000000000000002 is submitted.

    Flink Operator
    2023-07-06 10:23:47,973 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-wp-wafer-staging-e5730831] Submitted job: e692c2dfaa18441c0000000000000002 to session cluster.
    2023-07-06 10:23:47,505 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-wp-wafer-staging-e5730831] Submitting job: e692c2dfaa18441c0000000000000002 to session cluster.
    2023-07-06 10:23:45,374 o.a.f.k.o.s.AbstractFlinkService [INFO ][aelps-staging/aletsch-wp-wafer-staging-e5730831] Submitting job: e692c2dfaa18441c0000000000000002 to session cluster.

Attachments

Issue Links

is fixed by

FLINK-32592 (Stream)ExEnv#initializeContextEnvironment isn't thread-safe

Closed

Activity

People

Assignee:: Unassigned

Reporter:: Fabio Wanner

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 06/Jul/23 13:38

Updated:: 19/Jul/23 11:50

Resolved:: 19/Jul/23 11:50