Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22386 Data Source V2 improvements
  3. SPARK-23889

DataSourceV2: Add interfaces to pass required sorting and clustering for writes

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Implemented
    • 2.3.0
    • None
    • SQL
    • None

    Description

      From a discussion on the dev list, there is consensus around adding interfaces to pass required sorting and clustering to Spark. The proposal is to add:

      interface RequiresClustering {
  Set<Expression> requiredClustering();
}

interface RequiresSort {
  List<SortOrder> requiredOrdering();
}

      When only RequiresSort is present, the sort would produce a global sort. The partitioning introduced by that sort would be overridden by RequiresClustering, making the sort local to each partition.

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue SPARK-33779 DataSource V2: API to request distribution and ordering on write

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-33808 DataSource V2: Build logical writes in the optimizer

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-34026 DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-34049 DataSource V2: Use Write abstraction in StreamExecution

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #29066 [Github] Pull Request #29066 (aokolnychyi)

          Pull request #30706 [Github] Pull Request #30706 (aokolnychyi)

          Pull request #30706 [Github] Pull Request #30706 (aokolnychyi)

          Activity

            People

              Unassigned Unassigned
              rdblue Ryan Blue
              Votes:
              3 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: