Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7268

Make external sorter Hadoop free

Details

    • Improvement
    • Status: Triage Needed
    • P3
    • Resolution: Fixed
    • 2.13.0
    • 2.14.0
    • extensions-java-sorter

    Description

      Right now the Java sorter extension depends on Hadoop SequenceFile for external sort. It'll be nice to re-implement it without the dependency to avoid conflicts.

      Pull request adds a new option to switch to native sorting, the default behavior remains Hadoop sorting.

          final BufferedExternalSorter.Options sorterOpts =
              BufferedExternalSorter
                  .options()
                  .withExternalSorterType(ExternalSorter.Options.SorterType.HADOOP);
      
      ...
      
              .apply(SortValues.create(sorterOpts));
      
      

      Attachments

        Issue Links

          Activity

            People

              sinisa_lyh Neville Li
              sinisa_lyh Neville Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h