Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3630

Identify cause of Kryo+Snappy PARSING_ERROR

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Done
    • 1.1.0, 1.2.0
    • None
    • Spark Core
    • None

    Description

      A recent GraphX commit caused non-deterministic exceptions in unit tests so it was reverted (see SPARK-3400).

      Separately, aash observed the same exception stacktrace in an application-specific Kryo registrator:

      com.esotericsoftware.kryo.KryoException: java.io.IOException: failed to uncompress the chunk: PARSING_ERROR(2)
com.esotericsoftware.kryo.io.Input.fill(Input.java:142) com.esotericsoftware.kryo.io.Input.require(Input.java:169) 
com.esotericsoftware.kryo.io.Input.readInt(Input.java:325) com.esotericsoftware.kryo.io.Input.readFloat(Input.java:624) 
com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.read(DefaultSerializers.java:127) 
com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.read(DefaultSerializers.java:117) 
com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) 
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) 
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) 
com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
...

      This ticket is to identify the cause of the exception in the GraphX commit so the faulty commit can be fixed and merged back into master.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-6977 PARSING_ERROR(2) in Spark Streaming

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-3743 noisy logging when context is stopped

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-3426 Sort-based shuffle compression behavior is inconsistent

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-3948 Sort-based shuffle can lead to assorted stream-corruption exceptions

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-3958 Possible stream-corruption issues in TorrentBroadcast

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-4019 Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cause deserialization errors if at least one partition is empty

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-4105 FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-4133 PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-4056 Upgrade snappy-java to 1.1.1.5

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-3400 GraphX unit tests fail nondeterministically

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              joshrosen Josh Rosen
              aash Andrew Ash
              Votes:
              9 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: