Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4635

Dataflow runner deletes the binary specified in flag --worker_binary

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • 2.6.0
    • sdk-go
    • None

    Description

      Currently a binary compiled for the dataflow runner compiles the source code again in order to produce a worker binary.  In a production environment I will not have the source code around so I need to provide the worker binary myself.

      The good news: there is a flag for that.

      The bad news: the dataflow runner deletes the binary after launching the pipeline.

      Expected behaviour: don't delete the binary. I am the owner of the worker binary. It is not a temporary artifact. I would like to reuse the same binary to launch multiple dataflow pipelines

       

      Better yet, maybe add a flag "–self_worker_binary", so the currently running binary is used as the worker binary. This way I don't have to copy my binary twice into the docker image (once as the entrypoint, and once as data for the command line flag).

      Attachments

        Issue Links

          Activity

            People

              herohde Henning Rohde
              exm Eduardo Morales
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m