Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-2029

Perftest L2SVM icpt 1 sparse 800GB w/ codegen failing

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 1.0.0
    • None
    • None

    Description

      Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 141 and 161 -- Error evaluating instruction: CP°ba+*°_mVar169·MATRIX·DOUBLE°_mVar170·MATRIX·DOUBLE°_mVar176·MATRIX·DOUBLE°16
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:294)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:218)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
      	at org.apache.sysml.runtime.controlprogram.WhileProgramBlock.execute(WhileProgramBlock.java:115)
      	... 14 more
      Caused by: org.apache.sysml.runtime.controlprogram.caching.CacheException: Reading of scratch_space//_p2037254_9.1.44.28//_t0/temp66_56 (656476731) failed.
      	at org.apache.sysml.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:436)
      	at org.apache.sysml.runtime.controlprogram.context.ExecutionContext.getMatrixInput(ExecutionContext.java:274)
      	at org.apache.sysml.runtime.controlprogram.context.ExecutionContext.getMatrixInput(ExecutionContext.java:252)
      	at org.apache.sysml.runtime.instructions.cp.AggregateBinaryCPInstruction.processInstruction(AggregateBinaryCPInstruction.java:71)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:264)
      	... 17 more
      Caused by: java.io.IOException: Failed parallel read of binary block input.
      	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readBinaryBlockMatrixFromHDFS(ReaderBinaryBlockParallel.java:113)
      	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readMatrixFromHDFS(ReaderBinaryBlockParallel.java:70)
      	at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:204)
      	at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:168)
      	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:434)
      	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:59)
      	at org.apache.sysml.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:977)
      	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromRDD(MatrixObject.java:491)
      	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromRDD(MatrixObject.java:59)
      	at org.apache.sysml.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:426)
      	... 21 more
      Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 46
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readBinaryBlockMatrixFromHDFS(ReaderBinaryBlockParallel.java:103)
      	... 30 more
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 46
      	at org.apache.sysml.runtime.matrix.data.SparseRowVector.append(SparseRowVector.java:197)
      	at org.apache.sysml.runtime.matrix.data.SparseBlockMCSR.append(SparseBlockMCSR.java:264)
      	at org.apache.sysml.runtime.matrix.data.MatrixBlock.appendToSparse(MatrixBlock.java:758)
      	at org.apache.sysml.runtime.matrix.data.MatrixBlock.appendToSparse(MatrixBlock.java:721)
      	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel$ReadFileTask.call(ReaderBinaryBlockParallel.java:183)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      Attachments

        Activity

          People

            mboehm7 Matthias Boehm
            mboehm7 Matthias Boehm
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: