Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-2152

Spark mapmm fails for broadcast with larger than integer dims

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 1.1
    • None
    • None

    Description

      
              at org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.getBroadcastForVariable(SparkExecutionContext.java:545)
              at org.apache.sysml.runtime.instructions.spark.MapmmSPInstruction.processInstruction(MapmmSPInstruction.java:128)
              at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:264)
              ... 17 more
      Caused by: java.io.IOException: org.apache.sysml.runtime.DMLRuntimeException: Matrix dimensions too large for CP runtime: 514 x 5129281161
              at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:211)
              at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:171)
              at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:434)
              at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:59)
              at org.apache.sysml.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:981)
              at org.apache.sysml.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:417)
              ... 20 more
      Caused by: org.apache.sysml.runtime.DMLRuntimeException: Matrix dimensions too large for CP runtime: 514 x 5129281161
              at org.apache.sysml.runtime.io.MatrixReader.createOutputMatrixBlock(MatrixReader.java:80)
              at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readMatrixFromHDFS(ReaderBinaryBlockParallel.java:59)
              at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:207)
              ... 25 more
      

      Attachments

        Activity

          People

            mboehm7 Matthias Boehm
            mboehm7 Matthias Boehm
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: