Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Currently, the only way to run Samza containers in distributed mode is using Yarn. However, there has been interest to run Samza on top of other resource managers with the recent explosion in the # of such systems. Users have also requested us to run Samza as a library, and to run Samza on Docker containers managed by Kubernetes.
We must re-think the JobCoordinator functionality as follows:
1. ID assignment: Provide an ID to each SamzaContainer.
2. JobModel agreement: Ensure containers agree on a JobModel.
3. Re-start the SamzaContainer when the job model changes.
This will arguably require some leader election (depending on how users choose to run Samza).
Attachments
Attachments
Issue Links
- relates to
-
SAMZA-680 Invert the JobCoordinator and AM logic
-
- Resolved
-
-
SAMZA-375 Investigate Mesos Job Support
-
- Open
-
-
-
- Open
-
-
-
- Resolved
-
-
-
- Resolved
-
1.
|
Refactor all unit tests to work with the new JobCoordinator APIs |
|
Open | Unassigned |