[FLINK-35867] 99% state lost after task auto restart due to "rocksdb block checksum mismatch" exception - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Blocker
Resolution: Unresolved
Affects Version/s: 1.19.0
Fix Version/s: None
Component/s: API / DataStream, Runtime / Checkpointing, Runtime / State Backends
Labels:
None
Environment:
Hide

Flink version: 1.19.0

Linux: Alpine 3.20 runs on k8s

JDK17: OpenJDK 64-Bit Server VM - Alpine - 17/17.0.11+9-alpine-r0

Flink Configuration:

$internal.application.main com.xxxx.XXX $internal.application.program-args [] $internal.pipeline.job-id xxxxxxxxxxxxxxxxxx blob.server.port 6124 dstl.dfs.base-path s3://xxxxxxxxxxxxxxxxxxxxxx/checkpoints dstl.dfs.upload.max-attempts 6 dstl.dfs.upload.next-attempt-delay 5 s dstl.dfs.upload.timeout 5 s env.java.opts.all --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED execution.checkpointing.aligned-checkpoint-timeout 15 s execution.checkpointing.interval 30 s execution.checkpointing.min-pause 30 s execution.checkpointing.timeout 10 min execution.checkpointing.tolerable-failed-checkpoints 5 execution.checkpointing.unaligned true execution.savepoint-restore-mode CLAIM execution.savepoint.ignore-unclaimed-state false execution.target embedded heartbeat.interval 60000 heartbeat.timeout 300000 internal.cluster.execution-mode NORMAL jobmanager.archive.fs.dir s3://xxxxxxxxxxxxxxxxxx/history jobmanager.memory.heap.size 469762048b jobmanager.memory.jvm-metaspace.size 268435456b jobmanager.memory.jvm-overhead.max 201326592b jobmanager.memory.jvm-overhead.min 201326592b jobmanager.memory.off-heap.size 134217728b jobmanager.memory.process.size 1 gb jobmanager.rpc.address xxxxxxx.xxxxxxx jobmanager.rpc.port 6123 kubernetes.cluster-id xxxxxxxxxx kubernetes.internal.jobmanager.entrypoint.class org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint kubernetes.jobmanager.cpu.amount 0.5 kubernetes.jobmanager.labels xxxxxxxxxxxxxxxxxxxxxxxxxx kubernetes.jobmanager.service-account xxxxxxxxxxx kubernetes.namespace xxxxxxxx kubernetes.pod-template-file.jobmanager /tmp/xxxxxxx.tmp kubernetes.pod-template-file.taskmanager /tmp/xxxxxxxx.tmp kubernetes.rest-service.exposed.type ClusterIP kubernetes.taskmanager.cpu.amount 1 metrics.reporter.prom.class org.apache.flink.metrics.prometheus.PrometheusReporter metrics.reporter.prom.port 9249 parallelism.default 1 pipeline.classpaths [] pipeline.jars ['file:/opt/flink/usrlib/xxxxxxxx.jar'] pipeline.operator-chaining false restart-strategy.failure-rate.delay 30 s restart-strategy.failure-rate.failure-rate-interval 2 min restart-strategy.failure-rate.max-failures-per-interval 5 restart-strategy.type failure-rate s3.access-key xxxxxxxxx s3.endpoint xxxxxxxxx s3.path.style.access true s3.secret-key ****** s3.ssl.enabled false state.backend.changelog.enabled true state.backend.changelog.periodic-materialize.interval 5 min state.backend.changelog.storage filesystem state.backend.local-recovery true state.backend.rocksdb.log.dir /opt/flink/log/rocksdb state.backend.rocksdb.log.level WARN_LEVEL state.backend.type rocksdb state.checkpoints.dir s3://xxxxxxxxxxx/checkpoints taskmanager.memory.jvm-metaspace.size 150 mb taskmanager.memory.jvm-overhead.max 200 mb taskmanager.memory.managed.fraction 0.64 taskmanager.memory.network.max 20 mb taskmanager.memory.network.min 16 mb taskmanager.memory.process.size 3 gb taskmanager.memory.task.off-heap.size 200 mb taskmanager.numberOfTaskSlots 1 taskmanager.rpc.port 6122 web.tmpdir /tmp/flink-web-86fc04dd-19fb-4775-8084-52a263c827b7
Show
Flink version: 1.19.0 Linux: Alpine 3.20 runs on k8s JDK17: OpenJDK 64-Bit Server VM - Alpine - 17/17.0.11+9-alpine-r0 Flink Configuration: $internal.application.main com.xxxx.XXX $internal.application.program-args [] $internal.pipeline.job-id xxxxxxxxxxxxxxxxxx blob.server.port 6124 dstl.dfs.base-path s3: //xxxxxxxxxxxxxxxxxxxxxx/checkpoints dstl.dfs.upload.max-attempts 6 dstl.dfs.upload.next-attempt-delay 5 s dstl.dfs.upload.timeout 5 s env.java.opts.all --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED execution.checkpointing.aligned-checkpoint-timeout 15 s execution.checkpointing.interval 30 s execution.checkpointing.min-pause 30 s execution.checkpointing.timeout 10 min execution.checkpointing.tolerable-failed-checkpoints 5 execution.checkpointing.unaligned true execution.savepoint-restore-mode CLAIM execution.savepoint.ignore-unclaimed-state false execution.target embedded heartbeat.interval 60000 heartbeat.timeout 300000 internal.cluster.execution-mode NORMAL jobmanager.archive.fs.dir s3: //xxxxxxxxxxxxxxxxxx/history jobmanager.memory.heap.size 469762048b jobmanager.memory.jvm-metaspace.size 268435456b jobmanager.memory.jvm-overhead.max 201326592b jobmanager.memory.jvm-overhead.min 201326592b jobmanager.memory.off-heap.size 134217728b jobmanager.memory.process.size 1 gb jobmanager.rpc.address xxxxxxx.xxxxxxx jobmanager.rpc.port 6123 kubernetes.cluster-id xxxxxxxxxx kubernetes.internal.jobmanager.entrypoint.class org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint kubernetes.jobmanager.cpu.amount 0.5 kubernetes.jobmanager.labels xxxxxxxxxxxxxxxxxxxxxxxxxx kubernetes.jobmanager.service-account xxxxxxxxxxx kubernetes.namespace xxxxxxxx kubernetes.pod-template-file.jobmanager /tmp/xxxxxxx.tmp kubernetes.pod-template-file.taskmanager /tmp/xxxxxxxx.tmp kubernetes. rest -service.exposed.type ClusterIP kubernetes.taskmanager.cpu.amount 1 metrics.reporter.prom.class org.apache.flink.metrics.prometheus.PrometheusReporter metrics.reporter.prom.port 9249 parallelism. default 1 pipeline.classpaths [] pipeline.jars [ 'file:/opt/flink/usrlib/xxxxxxxx.jar' ] pipeline. operator -chaining false restart-strategy.failure-rate.delay 30 s restart-strategy.failure-rate.failure-rate-interval 2 min restart-strategy.failure-rate.max-failures-per-interval 5 restart-strategy.type failure-rate s3.access-key xxxxxxxxx s3.endpoint xxxxxxxxx s3.path.style.access true s3.secret-key ****** s3.ssl.enabled false state.backend.changelog.enabled true state.backend.changelog.periodic-materialize.interval 5 min state.backend.changelog.storage filesystem state.backend.local-recovery true state.backend.rocksdb.log.dir /opt/flink/log/rocksdb state.backend.rocksdb.log.level WARN_LEVEL state.backend.type rocksdb state.checkpoints.dir s3: //xxxxxxxxxxx/checkpoints taskmanager.memory.jvm-metaspace.size 150 mb taskmanager.memory.jvm-overhead.max 200 mb taskmanager.memory.managed.fraction 0.64 taskmanager.memory.network.max 20 mb taskmanager.memory.network.min 16 mb taskmanager.memory.process.size 3 gb taskmanager.memory.task.off-heap.size 200 mb taskmanager.numberOfTaskSlots 1 taskmanager.rpc.port 6122 web.tmpdir /tmp/flink-web-86fc04dd-19fb-4775-8084-52a263c827b7

Description

I hava a Flink task encounters an "rocksdb block checksum mismatch" exception, and it auto restart successfully as I have enabled the task failure restart strategy, but after the task restarted, I found that almost all the state was lost. here is the job log:

2024-07-12 18:28:24,818 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36218 for job 108a64cca6f69d6185679243cf2614c1 (5561263810 bytes, checkpointDuration=69663 ms, finalizationTime=0 ms).
2024-07-12 18:28:54,819 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36219 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780134818 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:29:21,329 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36219 for job 108a64cca6f69d6185679243cf2614c1 (5557919089 bytes, checkpointDuration=26510 ms, finalizationTime=1 ms).
2024-07-12 18:29:51,330 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36220 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780191330 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:29:51,669 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36220 for job 108a64cca6f69d6185679243cf2614c1 (5559503775 bytes, checkpointDuration=339 ms, finalizationTime=0 ms).
2024-07-12 18:30:21,669 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36221 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780221669 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:30:21,881 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36221 for job 108a64cca6f69d6185679243cf2614c1 (5560859350 bytes, checkpointDuration=211 ms, finalizationTime=1 ms).
2024-07-12 18:30:51,881 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36222 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780251881 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:31:01,769 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36222 for job 108a64cca6f69d6185679243cf2614c1 (5562228091 bytes, checkpointDuration=9888 ms, finalizationTime=0 ms).
2024-07-12 18:31:31,769 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36223 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780291769 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:31:32,040 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36223 for job 108a64cca6f69d6185679243cf2614c1 (5561271317 bytes, checkpointDuration=271 ms, finalizationTime=0 ms).
2024-07-12 18:32:02,041 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36224 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780322041 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:32:16,495 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36224 for job 108a64cca6f69d6185679243cf2614c1 (5562726599 bytes, checkpointDuration=14454 ms, finalizationTime=0 ms).
2024-07-12 18:32:46,495 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36225 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780366495 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:32:46,762 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36225 for job 108a64cca6f69d6185679243cf2614c1 (5562130180 bytes, checkpointDuration=267 ms, finalizationTime=0 ms).
2024-07-12 18:33:01,688 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 2 disconnected.
2024-07-12 18:33:16,762 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36226 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780396762 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:34:23,110 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36226 for job 108a64cca6f69d6185679243cf2614c1 (5563534332 bytes, checkpointDuration=66348 ms, finalizationTime=0 ms).
2024-07-12 18:34:53,110 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36227 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780493110 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:35:52,759 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36227 for job 108a64cca6f69d6185679243cf2614c1 (5564780617 bytes, checkpointDuration=59649 ms, finalizationTime=0 ms).
2024-07-12 18:36:22,759 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36228 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780582759 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:36:41,048 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36228 for job 108a64cca6f69d6185679243cf2614c1 (5564705694 bytes, checkpointDuration=18288 ms, finalizationTime=1 ms).
2024-07-12 18:37:11,048 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36229 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780631048 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:37:11,306 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36229 for job 108a64cca6f69d6185679243cf2614c1 (5561636978 bytes, checkpointDuration=257 ms, finalizationTime=0 ms).
2024-07-12 18:37:41,305 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36230 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780661305 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:37:48,013 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36230 for job 108a64cca6f69d6185679243cf2614c1 (5562888186 bytes, checkpointDuration=6708 ms, finalizationTime=0 ms).
2024-07-12 18:38:01,789 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 3 disconnected.
2024-07-12 18:38:18,014 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36231 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780698013 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:38:18,281 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36231 for job 108a64cca6f69d6185679243cf2614c1 (5562168310 bytes, checkpointDuration=268 ms, finalizationTime=0 ms).
2024-07-12 18:38:48,281 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36232 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780728281 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:38:48,490 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36232 for job 108a64cca6f69d6185679243cf2614c1 (5563567463 bytes, checkpointDuration=209 ms, finalizationTime=0 ms).
2024-07-12 18:39:18,490 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36233 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780758490 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:39:18,711 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36233 for job 108a64cca6f69d6185679243cf2614c1 (5564807753 bytes, checkpointDuration=221 ms, finalizationTime=0 ms).
2024-07-12 18:39:48,711 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36234 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780788711 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:39:54,232 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36234 for job 108a64cca6f69d6185679243cf2614c1 (5566129925 bytes, checkpointDuration=5521 ms, finalizationTime=0 ms).
2024-07-12 18:40:24,232 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36235 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780824232 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:40:35,807 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36235 for job 108a64cca6f69d6185679243cf2614c1 (5561884010 bytes, checkpointDuration=11574 ms, finalizationTime=1 ms).
2024-07-12 18:41:05,808 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36236 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780865807 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:42:24,718 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36236 for job 108a64cca6f69d6185679243cf2614c1 (5557617450 bytes, checkpointDuration=78911 ms, finalizationTime=0 ms).
2024-07-12 18:42:54,718 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36237 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780974718 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:43:01,891 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 5 disconnected.
2024-07-12 18:43:24,560 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36237 for job 108a64cca6f69d6185679243cf2614c1 (5558955281 bytes, checkpointDuration=29838 ms, finalizationTime=4 ms).
2024-07-12 18:43:54,560 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36238 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781034560 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:43:54,812 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36238 for job 108a64cca6f69d6185679243cf2614c1 (5558537398 bytes, checkpointDuration=252 ms, finalizationTime=0 ms).
2024-07-12 18:44:24,812 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36239 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781064812 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:44:25,022 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36239 for job 108a64cca6f69d6185679243cf2614c1 (5559598066 bytes, checkpointDuration=210 ms, finalizationTime=0 ms).
2024-07-12 18:44:55,023 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36240 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781095022 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:45:25,766 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36240 for job 108a64cca6f69d6185679243cf2614c1 (5560552320 bytes, checkpointDuration=30744 ms, finalizationTime=0 ms).
2024-07-12 18:45:55,766 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36241 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781155766 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:46:07,411 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36241 for job 108a64cca6f69d6185679243cf2614c1 (5560190293 bytes, checkpointDuration=11645 ms, finalizationTime=0 ms).
2024-07-12 18:46:37,411 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36242 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781197411 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:46:37,631 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36242 for job 108a64cca6f69d6185679243cf2614c1 (5559238902 bytes, checkpointDuration=219 ms, finalizationTime=1 ms).
2024-07-12 18:47:07,632 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36243 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781227631 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:47:07,843 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36243 for job 108a64cca6f69d6185679243cf2614c1 (5560131966 bytes, checkpointDuration=211 ms, finalizationTime=1 ms).
2024-07-12 18:47:37,843 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36244 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781257843 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:48:01,993 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 4 disconnected.
2024-07-12 18:48:32,551 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_0) switched from RUNNING to FAILED on iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613).
org.apache.flink.util.SerializedThrowable: org.rocksdb.RocksDBException: block checksum mismatch: stored = 2324967102, computed = 955729931  in /tmp/tm_iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1/tmp/job_108a64cca6f69d6185679243cf2614c1_op_KeyedProcessOperator_996b015f3163ca08af0c30ed8f6247a7__1_1__uuid_e0d4925d-c24b-4d3a-a693-2df28f090798/db/012566.sst offset 16764263 size 1149
    at org.rocksdb.RocksDB.write0(Native Method) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.rocksdb.RocksDB.write(RocksDB.java:1784) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flush(RocksDBWriteBatchWrapper.java:126) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:586) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.state.changelog.ChangelogKeyedStateBackend.initMaterialization(ChangelogKeyedStateBackend.java:796) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.state.common.PeriodicMaterializationManager.lambda$triggerMaterialization$1(PeriodicMaterializationManager.java:225) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist-1.19.0.jar:1.19.0]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-07-12 18:48:32,569 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - 13 tasks will be restarted to recover the failed task 3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_0.
2024-07-12 18:48:32,569 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job Flink Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RUNNING to RESTARTING.
2024-07-12 18:48:32,579 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 36244 for job 108a64cca6f69d6185679243cf2614c1. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator is suspending.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.stopCheckpointScheduler(CheckpointCoordinator.java:2056) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator.jobStatusChanges(CheckpointCoordinatorDeActivator.java:46) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifyJobStatusChange(DefaultExecutionGraph.java:1580) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1146) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1118) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.transitionExecutionGraphState(SchedulerBase.java:600) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.addVerticesToRestartPending(DefaultScheduler.java:382) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.restartTasksWithDelay(DefaultScheduler.java:358) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeRestartTasks(DefaultScheduler.java:326) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:268) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:261) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:488) ~[flink-dist-1.19.0.jar:1.19.0]
    at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) [?:?]
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) [?:?]
2024-07-12 18:48:32,584 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_0) switched from RUNNING to CANCELING.
2024-07-12 18:48:36,934 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:36,934 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:36,934 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:36,934 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:38,440 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:39,931 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:40,447 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:40,646 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_0) switched from CANCELING to CANCELED.
2024-07-12 18:48:40,648 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job 108a64cca6f69d6185679243cf2614c1
2024-07-12 18:48:40,648 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job Flink Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RESTARTING to RUNNING.
2024-07-12 18:48:40,648 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker [] - Clear all pending allocations for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:48:40,650 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Restoring job 108a64cca6f69d6185679243cf2614c1 from Checkpoint 36243 @ 1720781227631 for 108a64cca6f69d6185679243cf2614c1 located at s3://xxxxxxxxxxx/checkpoints/108a64cca6f69d6185679243cf2614c1/chk-36243.
2024-07-12 18:48:40,652 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No master state to restore
2024-07-12 18:48:40,654 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,654 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1) switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,656 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job 108a64cca6f69d6185679243cf2614c1: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2024-07-12 18:48:40,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: MySource (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1 and vertex id bc764cd8ddf7a0cff126f51c16239658_0 to iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying MySource (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1 and vertex id d24cade8b1c6d9528ce19407131a7eda_0 to iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,658 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying ProcessB (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1 and vertex id b4596501f3bb302b1c7aedb1a85dd7d4_0 to iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,658 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,658 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 497 (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1 and vertex id 0f257274a8483e4e1c50200b13c3ef28_0 to iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,658 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,658 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 530 (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1 and vertex id e43de375578634774108d582ac68fec1_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,659 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,659 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 531 (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1 and vertex id 996b015f3163ca08af0c30ed8f6247a7_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,659 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,659 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 532 (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1 and vertex id fa9967a8a4db458887f74772e7a3c05c_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,659 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,659 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 500 (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1 and vertex id 2f010967a103f39821dd1dc8dfbcabe7_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,660 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,660 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 528 (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1 and vertex id 005bcfbd5a31ac889c9be64a694eb949_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,660 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,660 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 529 (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1 and vertex id 361c54102873653ce8bbbfb96909315f_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,660 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,660 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying ProcessA (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1 and vertex id 777214ea96c0589f14708a68600868b9_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,661 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,661 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying GlobalWindows (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1 and vertex id 242e525045a332cf5da0422bf5582b5d_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,662 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,662 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying MySink: Writer (1/1) (attempt #1) with attempt id 3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1 and vertex id a4eb636d5723c707af0c320f39db4e52_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,759 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,759 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,834 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,933 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:40,934 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,934 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,936 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,140 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,236 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,236 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,237 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:41,332 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,337 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,340 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,536 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:41,737 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:42,029 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:42,633 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:43,432 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:43,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:43,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:44,032 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:44,036 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:44,337 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:48:45,030 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1) switched from INITIALIZING to RUNNING.
2024-07-12 18:49:10,649 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36245 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781350649 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:49:11,196 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36245 for job 108a64cca6f69d6185679243cf2614c1 (5565178747 bytes, checkpointDuration=546 ms, finalizationTime=0 ms).
2024-07-12 18:49:41,195 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36246 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781381195 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:49:41,384 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36246 for job 108a64cca6f69d6185679243cf2614c1 (5565960808 bytes, checkpointDuration=189 ms, finalizationTime=0 ms).
2024-07-12 18:50:11,384 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36247 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781411384 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:50:11,580 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36247 for job 108a64cca6f69d6185679243cf2614c1 (5566901598 bytes, checkpointDuration=196 ms, finalizationTime=0 ms).
2024-07-12 18:50:41,581 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36248 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781441580 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:51:16,833 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1) switched from RUNNING to FAILED on my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613).
org.apache.flink.util.SerializedThrowable: org.rocksdb.RocksDBException: block checksum mismatch: stored = 2324967102, computed = 955729931  in /tmp/tm_my-task-esoy-taskmanager-1-1/tmp/job_108a64cca6f69d6185679243cf2614c1_op_KeyedProcessOperator_996b015f3163ca08af0c30ed8f6247a7__1_1__uuid_97a224f6-fba4-4733-8709-55fba7b4f020/db/012566.sst offset 16764263 size 1149
    at org.rocksdb.RocksDB.write0(Native Method) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.rocksdb.RocksDB.write(RocksDB.java:1784) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flush(RocksDBWriteBatchWrapper.java:126) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:586) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.state.changelog.ChangelogKeyedStateBackend.initMaterialization(ChangelogKeyedStateBackend.java:796) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.state.common.PeriodicMaterializationManager.lambda$triggerMaterialization$1(PeriodicMaterializationManager.java:225) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist-1.19.0.jar:1.19.0]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-07-12 18:51:16,834 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - 13 tasks will be restarted to recover the failed task 3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1.
2024-07-12 18:51:16,834 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job Flink Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RUNNING to RESTARTING.
2024-07-12 18:51:16,834 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 36248 for job 108a64cca6f69d6185679243cf2614c1. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator is suspending.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.stopCheckpointScheduler(CheckpointCoordinator.java:2056) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator.jobStatusChanges(CheckpointCoordinatorDeActivator.java:46) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifyJobStatusChange(DefaultExecutionGraph.java:1580) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1146) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1118) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.transitionExecutionGraphState(SchedulerBase.java:600) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.addVerticesToRestartPending(DefaultScheduler.java:382) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.restartTasksWithDelay(DefaultScheduler.java:358) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeRestartTasks(DefaultScheduler.java:326) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:268) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:261) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:488) ~[flink-dist-1.19.0.jar:1.19.0]
    at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) [?:?]
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) [?:?]
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1) switched from RUNNING to CANCELING.
2024-07-12 18:51:22,531 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:22,531 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:22,531 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:22,630 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:22,630 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:22,631 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:22,631 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:23,231 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:24,132 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:24,232 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:24,330 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:24,345 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1) switched from CANCELING to CANCELED.
2024-07-12 18:51:24,346 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job 108a64cca6f69d6185679243cf2614c1
2024-07-12 18:51:24,346 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job Flink Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RESTARTING to RUNNING.
2024-07-12 18:51:24,346 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker [] - Clear all pending allocations for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:51:24,346 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Restoring job 108a64cca6f69d6185679243cf2614c1 from Checkpoint 36247 @ 1720781411384 for 108a64cca6f69d6185679243cf2614c1 located at s3://xxxxxxxxxxx/checkpoints/108a64cca6f69d6185679243cf2614c1/chk-36247.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No master state to restore
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2) switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,349 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job 108a64cca6f69d6185679243cf2614c1: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2024-07-12 18:51:24,349 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,349 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: MySource (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2 and vertex id bc764cd8ddf7a0cff126f51c16239658_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,350 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,350 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying MySource (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2 and vertex id d24cade8b1c6d9528ce19407131a7eda_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,350 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,350 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying ProcessB (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2 and vertex id b4596501f3bb302b1c7aedb1a85dd7d4_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,350 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,350 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 497 (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2 and vertex id 0f257274a8483e4e1c50200b13c3ef28_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 530 (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2 and vertex id e43de375578634774108d582ac68fec1_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 531 (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2 and vertex id 996b015f3163ca08af0c30ed8f6247a7_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 532 (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2 and vertex id fa9967a8a4db458887f74772e7a3c05c_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 500 (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2 and vertex id 2f010967a103f39821dd1dc8dfbcabe7_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 528 (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2 and vertex id 005bcfbd5a31ac889c9be64a694eb949_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,352 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,352 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 529 (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2 and vertex id 361c54102873653ce8bbbfb96909315f_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,352 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,352 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying ProcessA (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2 and vertex id 777214ea96c0589f14708a68600868b9_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,353 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,353 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying GlobalWindows (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2 and vertex id 242e525045a332cf5da0422bf5582b5d_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,353 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,353 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying MySink: Writer (1/1) (attempt #2) with attempt id 3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2 and vertex id a4eb636d5723c707af0c320f39db4e52_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,433 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,437 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,445 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,529 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,534 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,537 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,630 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,638 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,731 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:24,732 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,733 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:24,842 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,933 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,938 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,940 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:25,230 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:25,232 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:25,434 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:26,031 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:54,346 INFO  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger checkpoint for job 108a64cca6f69d6185679243cf2614c1 since Checkpoint triggering task 497 (1/1) of job 108a64cca6f69d6185679243cf2614c1 is not being executed at the moment. Aborting checkpoint. Failure reason: Not all required tasks are currently running..
2024-07-12 18:51:56,413 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:57,636 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:57,729 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:57,734 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:58,043 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:58,448 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:51:58,630 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2) switched from INITIALIZING to RUNNING.
2024-07-12 18:52:24,346 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36249 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781544346 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:52:24,763 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36249 for job 108a64cca6f69d6185679243cf2614c1 (5571063521 bytes, checkpointDuration=417 ms, finalizationTime=0 ms).
2024-07-12 18:52:54,763 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36250 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781574763 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:52:55,134 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36250 for job 108a64cca6f69d6185679243cf2614c1 (5572066981 bytes, checkpointDuration=370 ms, finalizationTime=1 ms).
2024-07-12 18:53:02,064 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 1 disconnected.
2024-07-12 18:53:25,135 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36251 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781605134 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:53:25,336 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36251 for job 108a64cca6f69d6185679243cf2614c1 (5573138583 bytes, checkpointDuration=201 ms, finalizationTime=1 ms).
2024-07-12 18:53:55,337 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36252 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781635337 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:54:30,532 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2) switched from RUNNING to FAILED on my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613).
org.apache.flink.util.SerializedThrowable: org.rocksdb.RocksDBException: block checksum mismatch: stored = 2324967102, computed = 955729931  in /tmp/tm_my-task-esoy-taskmanager-1-1/tmp/job_108a64cca6f69d6185679243cf2614c1_op_KeyedProcessOperator_996b015f3163ca08af0c30ed8f6247a7__1_1__uuid_7a62e596-fca3-4f5e-8905-55fa2f15254e/db/012566.sst offset 16764263 size 1149
    at org.rocksdb.RocksDB.write0(Native Method) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.rocksdb.RocksDB.write(RocksDB.java:1784) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flush(RocksDBWriteBatchWrapper.java:126) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:586) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.state.changelog.ChangelogKeyedStateBackend.initMaterialization(ChangelogKeyedStateBackend.java:796) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.state.common.PeriodicMaterializationManager.lambda$triggerMaterialization$1(PeriodicMaterializationManager.java:225) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist-1.19.0.jar:1.19.0]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-07-12 18:54:30,535 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - 13 tasks will be restarted to recover the failed task 3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2.
2024-07-12 18:54:30,535 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job Flink Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RUNNING to RESTARTING.
2024-07-12 18:54:30,535 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 36252 for job 108a64cca6f69d6185679243cf2614c1. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator is suspending.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.stopCheckpointScheduler(CheckpointCoordinator.java:2056) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator.jobStatusChanges(CheckpointCoordinatorDeActivator.java:46) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifyJobStatusChange(DefaultExecutionGraph.java:1580) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1146) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1118) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.transitionExecutionGraphState(SchedulerBase.java:600) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.addVerticesToRestartPending(DefaultScheduler.java:382) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.restartTasksWithDelay(DefaultScheduler.java:358) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeRestartTasks(DefaultScheduler.java:326) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:268) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:261) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:488) ~[flink-dist-1.19.0.jar:1.19.0]
    at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83) ~[flink-dist-1.19.0.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168) ~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253) [flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) [?:?]
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) [?:?]
2024-07-12 18:54:30,537 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2) switched from RUNNING to CANCELING.
2024-07-12 18:54:37,032 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:37,032 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:37,034 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:37,439 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:39,845 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:40,047 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:40,630 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:41,218 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2) switched from CANCELING to CANCELED.
2024-07-12 18:54:41,218 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job Flink Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RESTARTING to RUNNING.
2024-07-12 18:54:41,219 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job 108a64cca6f69d6185679243cf2614c1
2024-07-12 18:54:41,219 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker [] - Clear all pending allocations for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:54:41,219 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Restoring job 108a64cca6f69d6185679243cf2614c1 from Checkpoint 36251 @ 1720781605134 for 108a64cca6f69d6185679243cf2614c1 located at s3://xxxxxxxxxxx/checkpoints/108a64cca6f69d6185679243cf2614c1/chk-36251.
2024-07-12 18:54:41,220 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No master state to restore
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3) switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job 108a64cca6f69d6185679243cf2614c1: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2024-07-12 18:54:41,222 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,222 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: MySource (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3 and vertex id bc764cd8ddf7a0cff126f51c16239658_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,222 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,222 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying MySource (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3 and vertex id d24cade8b1c6d9528ce19407131a7eda_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,223 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,223 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying ProcessB (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3 and vertex id b4596501f3bb302b1c7aedb1a85dd7d4_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,223 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,223 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 497 (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3 and vertex id 0f257274a8483e4e1c50200b13c3ef28_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,224 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 530 (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3 and vertex id e43de375578634774108d582ac68fec1_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,224 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 531 (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3 and vertex id 996b015f3163ca08af0c30ed8f6247a7_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,224 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 532 (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3 and vertex id fa9967a8a4db458887f74772e7a3c05c_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 500 (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3 and vertex id 2f010967a103f39821dd1dc8dfbcabe7_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 528 (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3 and vertex id 005bcfbd5a31ac889c9be64a694eb949_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying 529 (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3 and vertex id 361c54102873653ce8bbbfb96909315f_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying ProcessA (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3 and vertex id 777214ea96c0589f14708a68600868b9_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,227 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,227 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying GlobalWindows (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3 and vertex id 242e525045a332cf5da0422bf5582b5d_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,227 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3) switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,227 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying MySink: Writer (1/1) (attempt #3) with attempt id 3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3 and vertex id a4eb636d5723c707af0c320f39db4e52_0 to my-task-esoy-taskmanager-1-1 @ 10.18.218.80 (dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,246 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,248 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,258 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,334 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySource (1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,335 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,339 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,431 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,547 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,548 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,631 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,633 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,637 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,732 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,733 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3) switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,739 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessB (1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,931 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - MySink: Writer (1/1) (3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 531 (1/1) (3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,937 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 497 (1/1) (3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,133 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ProcessA (1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,231 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 500 (1/1) (3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,332 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 528 (1/1) (3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 529 (1/1) (3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 532 (1/1) (3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - 530 (1/1) (3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:54:43,072 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindows (1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3) switched from INITIALIZING to RUNNING.
2024-07-12 18:55:11,219 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36253 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781711219 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:55:11,668 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36253 for job 108a64cca6f69d6185679243cf2614c1 (5579943671 bytes, checkpointDuration=448 ms, finalizationTime=0 ms).
2024-07-12 18:55:41,667 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36254 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781741667 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:55:41,932 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36254 for job 108a64cca6f69d6185679243cf2614c1 (5581813906 bytes, checkpointDuration=265 ms, finalizationTime=0 ms).
2024-07-12 18:56:11,932 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36255 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781771932 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:56:12,232 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36255 for job 108a64cca6f69d6185679243cf2614c1 (5583516024 bytes, checkpointDuration=300 ms, finalizationTime=0 ms).
2024-07-12 18:56:42,233 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36256 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781802232 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:56:42,533 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36256 for job 108a64cca6f69d6185679243cf2614c1 (4016415422 bytes, checkpointDuration=300 ms, finalizationTime=1 ms).
2024-07-12 18:57:12,534 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36257 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781832533 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:57:12,802 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36257 for job 108a64cca6f69d6185679243cf2614c1 (2652007752 bytes, checkpointDuration=269 ms, finalizationTime=0 ms).
2024-07-12 18:57:42,802 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36258 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781862802 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:57:43,112 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36258 for job 108a64cca6f69d6185679243cf2614c1 (1773173152 bytes, checkpointDuration=309 ms, finalizationTime=1 ms).
2024-07-12 18:58:02,165 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 2 disconnected.
2024-07-12 18:58:13,111 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36259 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781893111 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:58:13,347 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36259 for job 108a64cca6f69d6185679243cf2614c1 (895142299 bytes, checkpointDuration=236 ms, finalizationTime=0 ms).
2024-07-12 18:58:43,347 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36260 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781923347 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:58:43,638 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36260 for job 108a64cca6f69d6185679243cf2614c1 (894663617 bytes, checkpointDuration=291 ms, finalizationTime=0 ms).
2024-07-12 18:59:13,638 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36261 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781953638 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:59:13,946 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36261 for job 108a64cca6f69d6185679243cf2614c1 (896368454 bytes, checkpointDuration=308 ms, finalizationTime=0 ms).
2024-07-12 18:59:43,947 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36262 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781983946 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:59:44,174 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36262 for job 108a64cca6f69d6185679243cf2614c1 (14651880 bytes, checkpointDuration=228 ms, finalizationTime=0 ms).
2024-07-12 19:00:14,174 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36263 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782014174 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:00:14,407 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36263 for job 108a64cca6f69d6185679243cf2614c1 (16554103 bytes, checkpointDuration=232 ms, finalizationTime=1 ms).
2024-07-12 19:00:44,407 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36264 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782044407 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:00:44,653 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36264 for job 108a64cca6f69d6185679243cf2614c1 (18614579 bytes, checkpointDuration=246 ms, finalizationTime=0 ms).
2024-07-12 19:01:14,653 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36265 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782074653 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:01:14,890 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36265 for job 108a64cca6f69d6185679243cf2614c1 (20589849 bytes, checkpointDuration=236 ms, finalizationTime=1 ms).
2024-07-12 19:01:44,891 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36266 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782104890 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:01:45,096 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36266 for job 108a64cca6f69d6185679243cf2614c1 (18483980 bytes, checkpointDuration=206 ms, finalizationTime=0 ms).
2024-07-12 19:02:15,097 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36267 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782135097 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:02:15,330 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36267 for job 108a64cca6f69d6185679243cf2614c1 (15499141 bytes, checkpointDuration=232 ms, finalizationTime=1 ms).
2024-07-12 19:02:45,330 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36268 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782165330 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:02:45,561 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36268 for job 108a64cca6f69d6185679243cf2614c1 (17239129 bytes, checkpointDuration=230 ms, finalizationTime=1 ms).
2024-07-12 19:03:02,254 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 3 disconnected.
2024-07-12 19:03:15,561 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36269 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782195561 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:03:15,806 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36269 for job 108a64cca6f69d6185679243cf2614c1 (17200016 bytes, checkpointDuration=245 ms, finalizationTime=0 ms).
2024-07-12 19:03:45,806 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36270 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782225806 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:03:46,085 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36270 for job 108a64cca6f69d6185679243cf2614c1 (19206137 bytes, checkpointDuration=278 ms, finalizationTime=1 ms).
2024-07-12 19:04:16,086 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36271 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782256085 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:04:16,298 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36271 for job 108a64cca6f69d6185679243cf2614c1 (20850283 bytes, checkpointDuration=213 ms, finalizationTime=0 ms).
2024-07-12 19:04:46,299 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36272 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782286299 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:04:46,544 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36272 for job 108a64cca6f69d6185679243cf2614c1 (20564375 bytes, checkpointDuration=245 ms, finalizationTime=0 ms).
2024-07-12 19:05:16,544 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36273 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782316544 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:05:16,805 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36273 for job 108a64cca6f69d6185679243cf2614c1 (22748704 bytes, checkpointDuration=261 ms, finalizationTime=0 ms).
2024-07-12 19:05:46,805 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36274 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782346805 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:05:47,091 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36274 for job 108a64cca6f69d6185679243cf2614c1 (24839853 bytes, checkpointDuration=286 ms, finalizationTime=0 ms).
2024-07-12 19:06:17,091 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 36275 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782377091 for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:06:17,324 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 36275 for job 108a64cca6f69d6185679243cf2614c1 (26767845 bytes, checkpointDuration=233 ms, finalizationTime=0 ms).

There are some exceptions when checkpointing, I have found a similar report on FLINK-19016, in my case, it seems that rocksDB write local sst file failed.

But please see time from "2024-07-12 18:55:11,668" to "2024-07-12 19:00:14,174" of checkpoint 36253-36262 , the task has restarted successfully but the state size decreased from 5.5Gb to 14Mb, I didn't set any state ttlConfig for this task and there is no state delete in my business code. I am sure the state was realy lost as I have checked the sink data. My state is majority in the KeyedProcessFunction, I am not sure wether it is due to the "block checksum mismatch" exception and I am worried about this problem happens again, because there isn't any other exceptions throws and Flink tell me the task is restarted sucessfully , so when I realized this issue, it was already 3 days later, there is no more logs left.

The task has enabled the changelog state feature and it runs well for half a month before, I have set "state.backend.local-recovery=true", I don't know whether the configuration of my task is appropriate, please point me how to avoid this or even let the job crash down after it happens. I can't reproduce this problem still now.

by the way, the task is running on k8s and there is no k8s pod restarted during this problem happen.

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: heiya07

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 19/Jul/24 02:52

Updated:: 14/Aug/24 01:35