Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2008

kafka jars must be part of the storm-kafka topology

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • None
    • None
    • storm-kafka
    • None

    Description

      As of now the dependency is present but it's scope is provided, this is incorrect.
      https://github.com/apache/storm/blob/master/external/storm-kafka/pom.xml#L121

              <dependency>
                  <groupId>org.apache.kafka</groupId>
                  <artifactId>${storm.kafka.artifact.id}</artifactId>
                  <version>${storm.kafka.version}</version>
                  <scope>provided</scope>
              </dependency>
      

      This leads to following stacktrace in worker, when we try to run TridentKafkaWordCount topology.

          This is for solving following exception:
          Exception in thread "main" java.lang.NoClassDefFoundError: kafka/api/OffsetRequest
              at org.apache.storm.kafka.KafkaConfig.<init>(KafkaConfig.java:48)
              at org.apache.storm.kafka.trident.TridentKafkaConfig.<init>(TridentKafkaConfig.java:30)
              at org.apache.storm.starter.trident.TridentKafkaWordCount.createTransactionalKafkaSpout(TridentKafkaWordCount.java:102)
              at org.apache.storm.starter.trident.TridentKafkaWordCount.addTridentState(TridentKafkaWordCount.java:131)
              at org.apache.storm.starter.trident.TridentKafkaWordCount.buildConsumerTopology(TridentKafkaWordCount.java:149)
              at org.apache.storm.starter.trident.TridentKafkaWordCount.runMain(TridentKafkaWordCount.java:249)
              at org.apache.storm.starter.trident.TridentKafkaWordCount.main(TridentKafkaWordCount.java:240)
          Caused by: java.lang.ClassNotFoundException: kafka.api.OffsetRequest
              at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
              ... 7 more
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              raghavgautam Raghav Kumar Gautam
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: