Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-11342

Hive UDFs are unable to use classes from the same Jar in catalogd

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 4.1.0
    • Impala 4.2.0, Impala 4.1.1
    • Catalog
    • None
    • ghx-label-14

    Description

      While it is a known limitations that Hive UDFs in Impala cannot load dependencies from classpath, they should be able to load classes from the same jar (and thus creating an shaded jar with the dependencies should be a valid workaround). Meanwhile I got an error when the parent class of the UDF was a class defined in another package in the jar.

      The exception occurred when catalogd tried to instantiate the UDF class:

      I0608 14:50:21.520296 22464 jni-util.cc:286] java.lang.NoClassDefFoundError: com/esri/core/geometry/Point
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getConstructor0(Class.java:3075)
        at java.lang.Class.getConstructor(Class.java:1825)
        at org.apache.impala.hive.executor.HiveLegacyJavaFunction.instantiateUDFInstance(HiveLegacyJavaFunction.java:115)
        at org.apache.impala.hive.executor.HiveLegacyJavaFunction.<init>(HiveLegacyJavaFunction.java:74)
        at org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:50)
        at org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:67)
        at org.apache.impala.service.CatalogOpExecutor.createFunction(CatalogOpExecutor.java:2117)
        at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:453)
        at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:245)
Caused by: java.lang.ClassNotFoundException: com.esri.core.geometry.Point
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 11 more

      The issue seems to be cause by closing the UrlClassLoader too early, after loading the class but before doing the actual instantiation:
      https://github.com/apache/impala/blob/13bbff4e4e5fc5d459cc6f7a5512f84ceba897cd/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfLoader.java#L71

      Note that this is an issue only in the catalogd, not the executors, as the class loader is kept open longer in that case.

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-10997 Refactor code for UDF implementation

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-11774 CatalogD failed to start due to loading UDF failure

          • Major - Major loss of function.
          • Resolved
          relates to

          Task - A task that needs to be done. IMPALA-8016 UDFs unable to use other functions defined in same jar

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              csringhofer Csaba Ringhofer
              csringhofer Csaba Ringhofer
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: