Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5495

PipelineResources algorithm is not working in most environments

    XMLWordPrintableJSON

Details

    Description

      Issue are:
      1. it assumes the classloader is an URLClassLoader (not always true and java >= 9 breaks that as well for the app loader)
      2. it uses loader.getURLs() which leads to including the JRE itself in the staged file

      Looks like this detect resource algorithm can't work and should be replaced by a SPI rather than a built-in and not extensible algorithm. Another valid alternative is to just drop that "guess" logic and force the user to set staged files.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2530 Make Beam compatible with next Java LTS version (Java 11)

          • P3 - Non-urgent bugs, features, and improvements
          • Triage Needed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-7478 Remote cluster submission from Flink Runner broken due to staging issues

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-6656 org.apache.beam.runners.core.construction.PipelineResources#detectClassPathResourcesToStage broken on java > 8

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          links to

          Web Link GitHub Pull Request #10268

          Web Link GitHub Pull Request #10410

          Web Link GitHub Pull Request #10420

          Web Link GitHub Pull Request #10423

          Activity

            People

              ŁukaszG Lukasz Gajowy
              romain.manni-bucau Romain Manni-Bucau
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 16h 10m
                  16h 10m