Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8103

Stop job service when pipeline execution finishes

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • runner-flink

    Description

      Currently, job servers are shut down when the Python script exits [1]. A better long-term solution would be to shut them down instead when the pipeline is finished executing, such as [2]. This will put resource management in a common code path that is less error-prone.

       

      [1] https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/job_server.py#L73

      [2] https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/portable_runner.py#L451

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ibzib Kyle Weaver
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: