Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-2162

Deserialization of compressed blocks w/ shared dictionary fails in Spark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 1.1
    • None
    • None

    Description

      For very large datasets that exceed aggregate cluster memory, we automatically apply compression. However, in case blocks with shared DDC1 dictionaries are evicted and subsequently deserialized this deserialization fails with mismatching number of bytes.

      java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)

      Attachments

        Activity

          People

            mboehm7 Matthias Boehm
            mboehm7 Matthias Boehm
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: