THIS IS A TEST INSTANCE THIS IS A TEST INSTANCE THIS IS A TEST INSTANCE THIS IS A TEST INSTANCE THIS IS A TEST INSTANCE

DO NOT USE THIS INSTANCE FOR LIVE DATA!!!!

  1. Sub-task SPARK-38987
    Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true
  2. Sub-task SPARK-36705
    Disable push based shuffle when IO encryption is enabled or serializer is not relocatable
  3. Sub-task SPARK-36558
    Stage has all tasks finished but with ongoing finalization can cause job hang
  4. Sub-task SPARK-36530
    Avoid finalizing when there's no push at all in a shuffle
  5. Sub-task SPARK-36374
    Push-based shuffle documentation
  6. Sub-task SPARK-36266
    Rename classes in shuffle RPC used for block push operations
  7. Sub-task SPARK-36255
    FileNotFoundException from the shuffle push can cause the executor to terminate
  8. Sub-task SPARK-35917
    Disable push-based shuffle until the feature is complete
  9. Sub-task SPARK-35671
    Add Support in the ESS to serve merged shuffle block meta and data to executors
  10. Sub-task SPARK-35546
    Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way
  11. Sub-task SPARK-34840
    Fix cases of corruption in merged shuffle blocks that are pushed
  12. Sub-task SPARK-33350
    Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data
  13. Sub-task SPARK-32923
    Add support to properly handle different type of stage retries
  14. Sub-task SPARK-32922
    Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
  15. Sub-task SPARK-32921
    Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario
  16. Sub-task SPARK-32920
    Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
  17. Sub-task SPARK-32919
    Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions
  18. Sub-task SPARK-32918
    RPC implementation to support control plane coordination for push-based shuffle
  19. Sub-task SPARK-32917
    Add support for executors to push shuffle blocks after successful map task completion
  20. Sub-task SPARK-32916
    Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle
  21. Sub-task SPARK-32915
    RPC implementation to support pushing and merging shuffle blocks
Refresh results