Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Currently, the YARN AM pretty much dictates how the JobCoordinator works. This creates lot of inflexibility on how we can control failures or even integrate with new system (Mesos).
For e.g., https://issues.apache.org/jira/browse/SAMZA-465?focusedCommentId=14522043&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14522043
It would nice to invert the logic to JobCoordinator, and JobCoordinator has a global view of container failures, config changes etc. This simplifies lot of implementation specifics (for e.g., dynamic scaling becomes easier).
A another nice to have, would be make this logic pluggable.
e.g., job.clustermanager = org.apache.samza.YarnClusterManager job.clustermanager = org.apache.samza.MesosClusterManager
Attachments
Attachments
Issue Links
- is related to
-
SAMZA-881 Re-think the Samza Job Coordinator
- Open