Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13138

Unable to append to a SequenceFile with Compression.NONE.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.7.2
    • 2.8.0, 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      Hi,

      I'm trying to use the append functionnality to an existing SequenceFile.

      If I set Compression.NONE, it works when the file is created, but when the file already exists I've a NullPointerException, by the way it works if I specify a compression with a codec.

      Failing code
      Option compression = compression(CompressionType.NONE);
Option keyClass = keyClass(LongWritable.class);
Option valueClass = valueClass(BytesWritable.class);
Option out = file(dfs);
Option append = appendIfExists(true);

writer = createWriter(conf,
             out,
             append,
             compression,
             keyClass,
             valueClass);

      The following exeception is thrown when the file exists because compression option is checked:

      Exception in thread "main" java.lang.NullPointerException
	at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1119)
	at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)

      This is due to the codec which is null:

      SequenceFile.java
       if (readerCompressionOption.value != compressionTypeOption.value
                || !readerCompressionOption.codec.getClass().getName()
                    .equals(compressionTypeOption.codec.getClass().getName())) {
              throw new IllegalArgumentException(
                  "Compression option provided does not match the file");
            }

      Thansk

      Mickaël

      Attachments

        1. HADOOP-13138-01.patch
          5 kB
          Vinayakumar B
        2. HADOOP-13138-02.patch
          5 kB
          Vinayakumar B

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13216 Append SequenceFile with compressionType(NONE,RECORD,BLOCK) throws NullPointerException

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16021 SequenceFile.createWriter appendIfExists codec cause NullPointerException

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-7139 Allow appending to existing SequenceFiles

          • Major - Major loss of function.
          • Closed

          Activity

            People

              vinayakumarb Vinayakumar B
              mgervais Gervais Mickaël
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: