Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7093 Support Spark 3 in Spark runner
  3. BEAM-12591

Spark Structured Streaming runner: code source is not visible from the IDE

Details

    Description

      When spark 3 was introduced, the spark structured streaming runner sources were moved to spark/2 module (https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe) and the build system was changed. Now spark runners are built per spark version module (2 or 3) and the version specific sources are copied from spark-version/src to spark-version/build/source-overrides and gradle source sets are set to main-spark src + spark-version/build/source-overrides. The problem is that  spark-version/src is outside of gradle source sets and thus the IDE does not compile this source, that is not an issue when there is very few classes but in the case of the spark structured streaming runner, it is the whole runner code that is rendered invisible to the IDE which is impracticable for runner development.

       

      As ide project settings are gerenated with gradle import, changing the project settings manually would not work as the settings would be erased with each gradle import

      Attachments

        Issue Links

          Activity

            People

              echauchot Etienne Chauchot
              echauchot Etienne Chauchot
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m