Details
-
Sub-task
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
None
-
None
-
Committer (Medium) - This is for regular contributors/committers
Description
If a user wants to run various jobs using different technologies behind the resource manager, then there needs to be a technology to choose between them.
For example (scheduler is the key component of the resource manager backend):
"mesos job" -> Mesos scheduler et. al.
"spark job -> Spark scheduler et. al.
"normal job" -> Standard resource manager scheduler et. al.
"torque job" -> Torque scheduler et. al.
The natural place to make the switch is based on the queue that that job is in. Thus we can create a resource manager backend that multiplexes jobs based on the queue that they are in.
This can be accomplished by using XML configuration to map queues to backend sets, and create a custom set to multiplex between what is specified in XML. The set must include the following:
Muxing components:
-Scheduler
-Batchmgr
-Monitor
Supporting components:
-Backend Manager - keeps track of children (Scheduler, Monitor, Batchmgr) and which queue these sets are associated.
-Backend repository - to setup BackendManager via config