Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
taskHeartbeat currently handles 3 different operations
1. The heartbeat itself to indicate that the container is alive.
2. Reporting new events sent over by the task
3. Fetching events which are to be consumed by the task.
All three can be independent.
This does not affect the Tez wire protocol itself, which will continue to use a single call for all three. However this would get split within the default Tez AM plugin.
The eventual intent is to allow plugins to control this functionality independently.
An example: The addition of a eventsAvailable(Task*) on the TaskCommunicator allows a Plugin to get events the moment they're available and push them into the executors. Meanwhile, the heartbeat becomes just that - a heartbeat - and the frequency can be reduced to multiple seconds instead of the current milliseconds.