Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-25484

NULL cannot be inserted as a Parquet map (and as key in a map)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      not sure if this is a bug or a feature request, but I haven't been able to test NULL as map in the scope of HIVE-23688:

      CREATE TABLE parquet_map_type_string (
      id int,
      stringMap map<string, string>
      ) stored as parquet;
      
      insert into parquet_map_type_string SELECT 1, MAP('k1', null, 'k2', 'v2'); -- NULL as value, works
      insert into parquet_map_type_string (id) VALUES (2); -- NULL as map, fails
      insert into parquet_map_type_string SELECT 3, MAP(null, 'k3', 'k4', 'v4'); -- NULL as key, fails
      

      leads to:

      Caused by: java.lang.NullPointerException
      	at org.apache.parquet.io.api.Binary$FromStringBinary.encodeUTF8(Binary.java:218)
      	at org.apache.parquet.io.api.Binary$FromStringBinary.<init>(Binary.java:209)
      	at org.apache.parquet.io.api.Binary.fromString(Binary.java:537)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$StringDataWriter.write(DataWritableWriter.java:474)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MapDataWriter.write(DataWritableWriter.java:354)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:228)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:251)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:115)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:76)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:35)
      	at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128)
      	at org.apache.parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:182)
      	at org.apache.parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:44)
      	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:161)
      	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:174)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1160)
      	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
      	at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
      	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
      	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:173)
      	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:155)
      	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
      

      Attachments

        Issue Links

          Activity

            People

              abstractdog László Bodor
              abstractdog László Bodor
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: