Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9210

kafka stream loss data

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.1
    • None
    • streams
    • None

    Description

      kafka broker: 2.0.1

      kafka stream client: 2.1.0

      1. two applications run at the same time
      2. after some days,I stop one application(in k8s)
      3. The flollowing log occured and I check the data and find that value is less than what I expected.

      Partitions [flash-app-xmc-worker-share-store-minute-repartition-1]Partitions [flash-app-xmc-worker-share-store-minute-repartition-1] for changelog flash-app-xmc-worker-share-store-minute-changelog-12019-11-19 05:50:49.816|WARN |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting StreamTasks stores to recreate from scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19 05:50:49.817|INFO |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1 ProcessorTopology: KSTREAM-SOURCE-0000000070: topics: [flash-app-xmc-worker-share-store-minute-repartition] children: [KSTREAM-AGGREGATE-0000000067] KSTREAM-AGGREGATE-0000000067: states: [worker-share-store-minute] children: [KTABLE-TOSTREAM-0000000071] KTABLE-TOSTREAM-0000000071: children: [KSTREAM-SINK-0000000072] KSTREAM-SINK-0000000072: topic: StaticTopicNameExtractor(xmc-worker-share-minute)Partitions [flash-app-xmc-worker-share-store-minute-repartition-1] for changelog flash-app-xmc-worker-share-store-minute-changelog-12019-11-19 05:50:49.842|WARN |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting StreamTasks stores to recreate from scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19 05:50:49.842|INFO |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1 ProcessorTopology: KSTREAM-SOURCE-0000000070: topics: [flash-app-xmc-worker-share-store-minute-repartition] children: [KSTREAM-AGGREGATE-0000000067] KSTREAM-AGGREGATE-0000000067: states: [worker-share-store-minute] children: [KTABLE-TOSTREAM-0000000071] KTABLE-TOSTREAM-0000000071: children: [KSTREAM-SINK-0000000072] KSTREAM-SINK-0000000072: topic: StaticTopicNameExtractor(xmc-worker-share-minute)Partitions [flash-app-xmc-worker-share-store-minute-repartition-1] for changelog flash-app-xmc-worker-share-store-minute-changelog-12019-11-19 05:50:49.905|WARN |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting StreamTasks stores to recreate from scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19 05:50:49.906|INFO |flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1 ProcessorTopology: KSTREAM-SOURCE-0000000070: topics: [flash-app-xmc-worker-share-store-minute-repartition] children: [KSTREAM-AGGREGATE-0000000067] KSTREAM-AGGREGATE-0000000067: states: [worker-share-store-minute] 

       

      Attachments

        1. screenshot-1.png
          137 kB
          panpan.liu
        2. app.log
          271 kB
          panpan.liu

        Issue Links

          Activity

            People

              Unassigned Unassigned
              panpan.liu panpan.liu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: