Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21232 Kerberos delegation token framework
  3. FLINK-29622

KerberosDelegationTokenManager fails to load DelegationTokenProvider due to NoClassDefFoundError in various ITCases

    XMLWordPrintableJSON

Details

    Description

      There are multiple ITCases (e.g. EventTimeWindowCheckpointingITCase) that print an error when trying to load the HadoopFSDelegationTokenProvider which is on the classpath through flink-runtime but the corresponding hadoop dependency seems to be missing:

      186348 02:25:25,492 [                main] INFO  org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - Loading delegation token providers
 186349 02:25:25,493 [                main] ERROR org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - Failed to initialize delegation token provider hadoopfs
 186350 java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration
 186351         at org.apache.flink.runtime.security.token.HadoopFSDelegationTokenProvider.init(HadoopFSDelegationTokenProvider.java:68) ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186352         at org.apache.flink.runtime.security.token.KerberosDelegationTokenManager.loadProviders(KerberosDelegationTokenManager.java:124) ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186353         at org.apache.flink.runtime.security.token.KerberosDelegationTokenManager.<init>(KerberosDelegationTokenManager.java:109) ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186354         at org.apache.flink.runtime.security.token.KerberosDelegationTokenManager.<init>(KerberosDelegationTokenManager.java:91) ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186355         at org.apache.flink.runtime.security.token.KerberosDelegationTokenManagerFactory.create(KerberosDelegationTokenManagerFactory.java:47) ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186356         at org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:431) ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186357         at org.apache.flink.runtime.testutils.MiniClusterResource.startMiniCluster(MiniClusterResource.java:234) ~[flink-runtime-1.16-SNAPSHOT-tests.jar:1.16-SNAPSHOT]
 186358         at org.apache.flink.runtime.testutils.MiniClusterResource.before(MiniClusterResource.java:109) ~[flink-runtime-1.16-SNAPSHOT-tests.jar:1.16-SNAPSHOT]
 186359         at org.apache.flink.test.util.MiniClusterWithClientResource.before(MiniClusterWithClientResource.java:64) ~[flink-test-utils-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186360         at org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.setupTestCluster(EventTimeWindowCheckpointingITCase.java:253) ~[test-classes/:?]
 186361         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
[...]

      This error might be misleading/confusing to people investigating the logs. It looks like this error is actually expected since the tests not necessarily require Kerberos delegation tokens.

      Attachments

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue FLINK-25908 Move Hadoop FS token obtain functionality into HadoopFSDelegationTokenProvider

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #21098

          Web Link GitHub Pull Request #21114

          Activity

            People

              gaborgsomogyi Gabor Somogyi
              mapohl Matthias Pohl
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: