Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
2.6.0
-
None
Description
To reproduce:
- Try to run a Pipeline that takes longer than 50 seconds to go from JOB_STATE_PENDING to JOB_STATE_RUNNING using "with Pipeline() as pipeline:" (automatically executes pipeline.run().wait_until_finish() via Pipeline._exit_) on Dataflow.
- Your script will raise "AssertionError: Job did not reach to a terminal state after waiting indefinitely." (from https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1152-L1153)
I think the issue is that the block at https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L147-L160 unintentionally exits if it takes > 50 seconds for a job to go from JOB_STATE_PENDING to JOB_STATE_RUNNING.
Attachments
Issue Links
- links to