Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-996

Unnecessary shuffle / block mismatch in frame right indexing

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • SystemML 0.11
    • SystemML 0.11
    • None
    • None

    Description

      The implementation of our spark frame right indexing instruction unnecessarily tries to create aligned partial blocks, which causes unnecessary shuffle and merge issues (on merging these partial blocks) as frame can have variable block sizes. We should avoid merge/shuffle altogether and simply slice out the relevant sub blocks.

      cc mwdusenb@us.ibm.com acs_s

      Attachments

        Activity

          This issue results in errors such as:

          org.apache.sysml.runtime.DMLRuntimeException: Mismatched frame block sizes for: 15 65539 11 65539	at org.apache.sysml.runtime.instructions.spark.utils.FrameRDDAggregateUtils$MergeBlocksFunction.call(FrameRDDAggregateUtils.java:82)	at org.apache.sysml.runtime.instructions.spark.utils.FrameRDDAggregateUtils$MergeBlocksFunction.call(FrameRDDAggregateUtils.java:64)
          
          dusenberrymw Mike Dusenberry added a comment - This issue results in errors such as: org.apache.sysml.runtime.DMLRuntimeException: Mismatched frame block sizes for : 15 65539 11 65539 at org.apache.sysml.runtime.instructions.spark.utils.FrameRDDAggregateUtils$MergeBlocksFunction.call(FrameRDDAggregateUtils.java:82) at org.apache.sysml.runtime.instructions.spark.utils.FrameRDDAggregateUtils$MergeBlocksFunction.call(FrameRDDAggregateUtils.java:64)
          mboehm7 Matthias Boehm added a comment -

          mwdusenb@us.ibm.com ok, this issue should now be fixed along with the performance (right indexing is now a simple narrow transformation as it is ought to be).

          mboehm7 Matthias Boehm added a comment - mwdusenb@us.ibm.com ok, this issue should now be fixed along with the performance (right indexing is now a simple narrow transformation as it is ought to be).

          Great, that fixed this issue!

          dusenberrymw Mike Dusenberry added a comment - Great, that fixed this issue!

          People

            mboehm7 Matthias Boehm
            mboehm7 Matthias Boehm
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: