Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1064

Standalone Samza with Zookeeper for Coordination

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 0.13.0
    • None
    • None

    Description

      In this use-case, we propose using Zookeeper for coordinating processor liveness and task distribution in a Samza job. Additionally, it opens up the possibility of allowing a flexible number of participating processors (no fixed container count).

      Attachments

        1. SamzaStandalone1.pdf
          75 kB
          Boris Shkolnik
        2. SamzaStandalone1.md
          30 kB
          Boris Shkolnik
        3. Samza Standalone-0.pdf
          322 kB
          Navina Ramesh
        4. Samza Standalone-0.md
          30 kB
          Navina Ramesh
        5. image_1.png
          62 kB
          Navina Ramesh
        6. image_0.png
          5 kB
          Navina Ramesh

        Issue Links

          depends upon

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1203 Use ProcessorId instead of Zk node Id in JobModel generation

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1147 ZkJobCoordinator does not honor the interface contract of JobCoordinator

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1173 Evaluate CoordinationService design and resolve unresolved questions from PR#91

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1150 Handling Error propagation between ZkJobCoordinator & DebounceTimer

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1193 Fix the flaky test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1201 Zk path setup for leader elector and barrier should be separated

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1212 Refactor interaction between StreamProcessor, JobCoordinator and SamzaContainer

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1241 ScheduleAfterDebounceTime thread is not shutdown when ZkJobCoordinator shutsdown

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1272 ZkCoordinationUtils deletes the entire Zk tree on reset

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue SAMZA-555 Write a ZK-based samza-standalone JobRunner

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1279 list of children in onProcessorsChange throws NPE when there an exception in ZK

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1282 Spinning up more containers than the number of tasks kills leader

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SAMZA-1125 Container.shutdown() only stops the runloop

          • Major - Major loss of function.
          • Resolved

          Test - A new unit, integration or system test. SAMZA-1343 Modify the existing failure-testing script for Zookeeper based deployment

          • Major - Major loss of function.
          • Open
          is required by

          New Feature - A new feature of the product, which has yet to be developed. SAMZA-1063 Samza Standalone Project

          • Major - Major loss of function.
          • In Progress
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1344 Add retry in ApplicationRunner for failed StreamProcessors

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1348 Run stop() in the StreamProcessor in the same thread as other events.

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1301 Investigate ZKClient exceptions in ZkJobCoordinator.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1382 Support for ZK communication version protocol.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1302 Improve Zookeeper error handling

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1304 Improve duplicate processors detection

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1338 Number of processors greater then number of tasks results in an incorrect behavior.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1368 make sure that a new job model will be generated if zk barrier times out.

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. SAMZA-1041 Multi-stage feature for Samza

          • Major - Major loss of function.
          • In Progress

          Improvement - An improvement or enhancement to an existing feature or task. SAMZA-1349 Fix stop() sequence in the StreamProcessor

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SAMZA-1303 Request for waitForFinish(time, TimeUnit) api in LocalApplicationRunner

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          requires

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1084 User thread does not see errors from the processor thread when using the StreamProcessor API

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #64

          Web Link GitHub Pull Request #180

          Activity

            People

              navina Navina Ramesh
              navina Navina Ramesh
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: