Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14380

Queries on tables with remote HDFS paths fail in "encryption" checks.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.2.0
    • Encryption
    • None

    Description

      If a table has table/partition locations set to remote HDFS paths, querying them will cause the following IAException:

      2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1867)) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected: hdfs://bar.ygrid.yahoo.com:8020
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274)
...

      This is because of the following code in SessionState:

      SessionState.java
       public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws HiveException {
    if (hdfsEncryptionShim == null) {
      try {
        FileSystem fs = FileSystem.get(sessionConf);
        if ("hdfs".equals(fs.getUri().getScheme())) {
          hdfsEncryptionShim = ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf);
        } else {
          LOG.debug("Could not get hdfsEncryptionShim, it is only applicable to hdfs filesystem.");
        }
      } catch (Exception e) {
        throw new HiveException(e);
      }
    }

    return hdfsEncryptionShim;
  }

      When the FileSystem instance is created, using the sessionConf implies that the current HDFS is going to be used. This call should instead fetch the FileSystem instance corresponding to the path being checked.

      A fix is forthcoming...

      (Note to self: YHIVE-860)

      Attachments

        1. HIVE-14380.branch-1.2.patch
          17 kB
          Mithun Radhakrishnan
        2. HIVE-14380.1.patch
          8 kB
          Mithun Radhakrishnan

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HIVE-14379 Queries on tables with remote HDFS paths fail in "encryption" checks.

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-16874 qurey fail when try to read file from remote hdfs

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11116 Can not select data from table which points to remote hdfs location

          • Major - Major loss of function.
          • Patch Available

          Bug - A problem which impairs or prevents the functions of the product. HIVE-14688 Hive drop call fails in presence of TDE

          • Major - Major loss of function.
          • Patch Available
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31675 Fail to insert data to a table with remote location which causes by hive encryption check

          • Major - Major loss of function.
          • In Progress

          Bug - A problem which impairs or prevents the functions of the product. HIVE-16874 qurey fail when try to read file from remote hdfs

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              mithun Mithun Radhakrishnan
              mithun Mithun Radhakrishnan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: