Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-1265

ArrayIndexOutOfBoundsException in interQuartileMean(X, W)

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • SystemML 0.13
    • None
    • None

    Description

      The interQuartileMean(X, W) function seems to be broken. The following script throws an

       ArrayIndexOutOfBoundsException 
      A = matrix("0.0 1.0 2.0 1.0 0.0", rows=5, cols=1)
      W = matrix("0.0 0.5 1.0 0.5 0.0", rows=5, cols=1)
      
      b = interQuartileMean(A, W)
      print("b is: " + b)
      

      Error:

      17/02/14 13:56:09 INFO api.DMLScript: BEGIN DML run 02/14/2017 13:56:09
      17/02/14 13:56:10 INFO api.DMLScript: SystemML Statistics:
      Total execution time:		0.368 sec.
      Number of executed MR Jobs:	0.
      
      17/02/14 13:56:10 INFO api.DMLScript: END DML run 02/14/2017 13:56:10
      17/02/14 13:56:10 ERROR api.DMLScript: Failed to execute DML script.
      org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 1 and 5 -- Error evaluating instruction: CP°qpick°_mVar4·MATRIX·DOUBLE°_Var6·SCALAR·DOUBLE°IQM°true
      	at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
      	at org.apache.sysml.api.DMLScript.execute(DMLScript.java:684)
      	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:360)
      	at org.apache.sysml.api.DMLScript.main(DMLScript.java:221)
      Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 1 and 5 -- Error evaluating instruction: CP°qpick°_mVar4·MATRIX·DOUBLE°_Var6·SCALAR·DOUBLE°IQM°true
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:320)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:221)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:168)
      	at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
      	... 3 more
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 9
      	at org.apache.sysml.runtime.matrix.data.MatrixBlock.quickGetValue(MatrixBlock.java:576)
      	at org.apache.sysml.runtime.matrix.data.MatrixBlock.interQuartileMean(MatrixBlock.java:4693)
      	at org.apache.sysml.runtime.instructions.cp.QuantilePickCPInstruction.processInstruction(QuantilePickCPInstruction.java:177)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290)
      	... 6 more
      Exception in thread "main" org.apache.sysml.api.DMLException: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 1 and 5 -- Error evaluating instruction: CP°qpick°_mVar4·MATRIX·DOUBLE°_Var6·SCALAR·DOUBLE°IQM°true
      	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:374)
      	at org.apache.sysml.api.DMLScript.main(DMLScript.java:221)
      Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 1 and 5 -- Error evaluating instruction: CP°qpick°_mVar4·MATRIX·DOUBLE°_Var6·SCALAR·DOUBLE°IQM°true
      	at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
      	at org.apache.sysml.api.DMLScript.execute(DMLScript.java:684)
      	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:360)
      	... 1 more
      Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 1 and 5 -- Error evaluating instruction: CP°qpick°_mVar4·MATRIX·DOUBLE°_Var6·SCALAR·DOUBLE°IQM°true
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:320)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:221)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:168)
      	at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
      	... 3 more
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 9
      	at org.apache.sysml.runtime.matrix.data.MatrixBlock.quickGetValue(MatrixBlock.java:576)
      	at org.apache.sysml.runtime.matrix.data.MatrixBlock.interQuartileMean(MatrixBlock.java:4693)
      	at org.apache.sysml.runtime.instructions.cp.QuantilePickCPInstruction.processInstruction(QuantilePickCPInstruction.java:177)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290)
      	... 6 more
      
      

      Attachments

        Activity

          mboehm7 Matthias Boehm added a comment -

          Yes, and we actually have tests for all of them. As it turned out, the problem was wrong input data - we expect the weights to be integers, so the two 0.5 are actually invalid. So far we only checked the total sum for integer but in your case the two 0.5 canceled each other out. I'll add a more robust error handling.

          Before updating the documentation, we should also think about making all weighted operations consistent - there are some operations like cov and cm that allow arbitrary weights, whereas quantile and iqm require integer data.

          mboehm7 Matthias Boehm added a comment - Yes, and we actually have tests for all of them. As it turned out, the problem was wrong input data - we expect the weights to be integers, so the two 0.5 are actually invalid. So far we only checked the total sum for integer but in your case the two 0.5 canceled each other out. I'll add a more robust error handling. Before updating the documentation, we should also think about making all weighted operations consistent - there are some operations like cov and cm that allow arbitrary weights, whereas quantile and iqm require integer data.
          fschueler Felix Schueler added a comment -

          Yeah, there are a couple of functions that take weights (quantile, median, ...). Isn't it the same as doing interQuartileMean(W * X)?

          fschueler Felix Schueler added a comment - Yeah, there are a couple of functions that take weights (quantile, median, ...). Isn't it the same as doing interQuartileMean(W * X)?
          mboehm7 Matthias Boehm added a comment -

          thanks for reporting this fschueler - it's probably due to the weights which are rather uncommon. I"ll have a look later tonight.

          mboehm7 Matthias Boehm added a comment - thanks for reporting this fschueler - it's probably due to the weights which are rather uncommon. I"ll have a look later tonight.

          People

            mboehm7 Matthias Boehm
            fschueler Felix Schueler
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: