Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-2026

Perftest OOM on genStratStatsData 80GB w/ codegen

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 1.0.0
    • None
    • None

    Description

      java.lang.OutOfMemoryError: Java heap space
      	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
      	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
      	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$3.apply(TorrentBroadcast.scala:271)
      	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$3.apply(TorrentBroadcast.scala:271)
      	at org.apache.spark.util.io.ChunkedByteBufferOutputStream.allocateNewChunkIfNeeded(ChunkedByteBufferOutputStream.scala:87)
      	at org.apache.spark.util.io.ChunkedByteBufferOutputStream.write(ChunkedByteBufferOutputStream.scala:75)
      	at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
      	at net.jpountz.lz4.LZ4BlockOutputStream.finish(LZ4BlockOutputStream.java:235)
      	at net.jpountz.lz4.LZ4BlockOutputStream.close(LZ4BlockOutputStream.java:175)
      	at java.io.ObjectOutputStream$BlockDataOutputStream.close(ObjectOutputStream.java:1828)
      	at java.io.ObjectOutputStream.close(ObjectOutputStream.java:742)
      	at org.apache.spark.serializer.JavaSerializationStream.close(JavaSerializer.scala:57)
      	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$blockifyObject$1.apply$mcV$sp(TorrentBroadcast.scala:278)
      	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1346)
      	at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:277)
      	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:126)
      	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
      	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
      	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:56)
      	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1488)
      	at org.apache.spark.api.java.JavaSparkContext.broadcast(JavaSparkContext.scala:650)
      	at org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.getBroadcastForVariable(SparkExecutionContext.java:559)
      	at org.apache.sysml.runtime.instructions.spark.SpoofSPInstruction.processInstruction(SpoofSPInstruction.java:125)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:264)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:218)
      	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
      	at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:118)
      	at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:102)
      	at org.apache.sysml.api.DMLScript.execute(DMLScript.java:747)
      	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:511)
      	at org.apache.sysml.api.DMLScript.main(DMLScript.java:238)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      

      Attachments

        Issue Links

          Activity

            People

              mboehm7 Matthias Boehm
              mboehm7 Matthias Boehm
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: