Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12175

Upgrade Kryo version to 3.0.x

    XMLWordPrintableJSON

Details

    Description

      Current version of kryo (2.22) has some issue (refer exception below and in HIVE-12174) with serializing ArrayLists generated using Arrays.asList(). We need to either replace all occurrences of Arrays.asList() or change the current StdInstantiatorStrategy. This issue is fixed in later versions and kryo community recommends using DefaultInstantiatorStrategy with fallback to StdInstantiatorStrategy. More discussion about this issue is here https://github.com/EsotericSoftware/kryo/issues/216. Alternatively, custom serilization/deserilization class can be provided for Arrays.asList.

      Also, kryo 3.0 introduced unsafe based serialization which claims to have much better performance for certain types of serialization.

      Exception:

      Caused by: java.lang.NullPointerException
	at java.util.Arrays$ArrayList.size(Arrays.java:2847)
	at java.util.AbstractList.add(AbstractList.java:108)
	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
	at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	... 57 more

      Attachments

        1. HIVE-12175.6.patch
          20 kB
          Prasanth Jayachandran
        2. HIVE-12175.5.patch
          20 kB
          Prasanth Jayachandran
        3. HIVE-12175.4.patch
          13 kB
          Prasanth Jayachandran
        4. HIVE-12175.3.patch
          11 kB
          Prasanth Jayachandran
        5. HIVE-12175.3.patch
          10 kB
          Prasanth Jayachandran
        6. HIVE-12175.2.patch
          10 kB
          Prasanth Jayachandran
        7. HIVE-12175.1.patch
          8 kB
          Prasanth Jayachandran

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11519 kryo.KryoException: Encountered unregistered

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12174 Error: java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 112, Size: 11

          • Blocker - Blocks development and/or testing work, production could not run
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12551 Fix several kryo exceptions in branch-1

          • Major - Major loss of function.
          • Resolved
          is required by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12302 Use KryoPool instead of thread-local caching

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-10793 Make spark's use/subclassing of hive more maintainable

          • Major - Major loss of function.
          • Resolved

          Wish - General wishlist item. SPARK-11416 Upgrade kryo package to version 3.0

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              prasanth_j Prasanth Jayachandran
              prasanth_j Prasanth Jayachandran
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: