Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-2196

Support LZ4_RAW codec

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Resolved
    • None
    • 1.13.0
    • parquet-mr
    • None

    Description

      There is a long history about the LZ4 interoperability of parquet files between parquet-mr and parquet-cpp (which is now in the Apache Arrow). Attached links are the evidence. In short, a new LZ4_RAW codec type has been introduced since parquet format v2.9.0. However, only parquet-cpp supports LZ4_RAW. The parquet-mr library still uses the old Hadoop-provided LZ4 codec and cannot read parquet files with LZ4_RAW.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-43273 Support lz4raw compression codec for Parquet

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. PARQUET-1974 LZ4 decoding is not working over hadoop

          • Blocker - Blocks development and/or testing work, production could not run
          • Open

          Bug - A problem which impairs or prevents the functions of the product. ARROW-9177 [C++][Parquet] Tracking issue for cross-implementation LZ4 Parquet compression compatibility

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-1878 [C++] lz4 codec is not compatible with Hadoop Lz4Codec

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-1996 [Format] Add interoperable LZ4 codec, deprecate existing LZ4 codec

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              wgtmac Gang Wu
              wgtmac Gang Wu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: