Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-1288

[C++] Invalid memory freeing with ZLIB compression

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.8.0
    • 1.8.1
    • None
    • None

    Description

      The simple example code ends with a segfault/munmap_chunk(): invalid pointer:

      #include "orc/Common.hh"
      #include "orc/OrcFile.hh"
      
      using namespace orc;
      
      int main(void) {
          ORC_UNIQUE_PTR<OutputStream> outStream = writeLocalFile("test_file.orc");
          ORC_UNIQUE_PTR<Type> schema(Type::buildTypeFromString("struct<c0:int>"));
          WriterOptions options;
          options.setCompression(orc::CompressionKind_ZLIB);
          options.setStripeSize(4096);
          options.setCompressionBlockSize(4096);
          ORC_UNIQUE_PTR<Writer> writer = createWriter(*schema, outStream.get(), options);
          uint64_t batchSize = 65535, rowCount = 10000000;
          ORC_UNIQUE_PTR<ColumnVectorBatch> batch = writer->createRowBatch(batchSize);
          StructVectorBatch *root = dynamic_cast<StructVectorBatch *>(batch.get());
          LongVectorBatch *c0 = dynamic_cast<LongVectorBatch *>(root->fields[0]);
          uint64_t rows = 0;
          
          for (uint64_t i = 0; i < rowCount; ++i) {
              c0->data[rows] = i;
              rows++;
              if (rows == batchSize) {
                  root->numElements = rows;
                  c0->numElements = rows;
                  writer->add(*batch);
                  rows = 0;
              }
          }
          if (rows != 0) {
              root->numElements = rows;
              c0->numElements = rows;
              writer->add(*batch);
              rows = 0;
          }
          writer->close();
          return 0;
      }
      

      The bug depends on the stripe size, compression size, and the record number written to the file as well. I wasn't able to reproduce the error with other compression strategies than ZLIB.

      It looks like to me that it's related to ORC-1130 somehow, but I couldn't comprehend how. (Reverting that modification on the main branch solved the issue).

      Attachments

        Issue Links

          Activity

            People

              rex_xinzh ZhangXin
              noirello noirello
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: