Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-2435

GetHDFSSequenceFile sequence reader throwing away authenticated filesystem and creating its own

    XMLWordPrintableJSON

Details

    Description

      Hadoop Kerberos has a way of accessing things as an authenticated user
      UserGroupInformation.doAs
      NIFI is doing that to get access to the filesystem in order to basically tell hadoop to use the correct credentials so our processor does that correctly and then passes the results into the sequence reader unfortunately, the sequence reader then throws away our authenticated filesystem and creates its own without wrapping in a UGI.doAs

      More context:
      The constructor used for org.apache.hadoop.io.SequenceFile.Reader in both KeyValueReader and ValueReader is deprecated and has been turned into a callthrough that only uses the passed in FileSystem to get the qualified path[1]. I've modified those classes to do that logic and then wrapped the callsite in the processor in a UGI.doAs if it is configured to use a UGI

      [1]: https://github.com/apache/hadoop/commit/0c462b223f151208ff7bd5148cee0e436c23d795#diff-e70823f17863445da52875999e4b19a2L1592

      Attachments

        Issue Links

          Activity

            People

              bryanrosander@gmail.com Bryan Rosander
              digitalplummer Juan Sequeiros
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: