Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36681

Fail to load Snappy codec

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.3.0
    • Spark Core
    • None

    Description

      snappy-java as a native library should not be relocated in Hadoop shaded client libraries. Currently we use Hadoop shaded client libraries in Spark. If trying to use SnappyCodec to write sequence file, we will encounter the following error:

      [info]   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native Method)                                                                                                 
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)                                                                                                        
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
[info]   at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
[info]   at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.writeBuffer(SequenceFile.java:1589) 
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.sync(SequenceFile.java:1605)
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1629)

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17891 lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-17125 Using snappy-java in SnappyCodec

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #35784 [Github] Pull Request #35784 (viirya)

          Pull request #36136 [Github] Pull Request #36136 (HyukjinKwon)

          Pull request #36136 [Github] Pull Request #36136 (HyukjinKwon)

          Pull request #36687 [Github] Pull Request #36687 (peter-toth)

          Activity

            People

              viirya L. C. Hsieh
              viirya L. C. Hsieh
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: