Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12417

Orc bloom filter options are not propagated during file write in spark

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.0.0
    • SQL
    • None

    Description

      ORC bloom filter is supported by the version of hive used in Spark 1.5.2. However, when trying to create orc file with bloom filter option, it does not make use of it.

      E.g, following orc output does not create the bloom filter even though the options are specified.

          Map<String, String> orcOption = new HashMap<String, String>();
    orcOption.put("orc.bloom.filter.columns", "*");
    hiveContext.sql("select * from accounts where effective_date='2015-12-30'").write().
        format("orc").options(orcOption).save("/tmp/accounts");

      Attachments

        1. SPARK-12417.1.patch
          2 kB
          Rajesh Balamohan

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20901 Feature parity for ORC with Parquet

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25427 Add BloomFilter creation test cases

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. ORC-137 Disable bloomfilter PPD for timestamps for files created before ORC-135

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ORC-101 Correct the use of the default charset in the bloomfilter

          • Major - Major loss of function.
          • Closed
          links to

          Pull request #10375 [Github] Pull Request #10375 (rajeshbalamohan)

          Pull request #10842 [Github] Pull Request #10842 (rajeshbalamohan)

          Activity

            People

              apachespark Apache Spark
              rajesh.balamohan Rajesh Balamohan
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: