Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3044

Split TaskCommunicatorContext.taskHeartbeat into separate methods

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      taskHeartbeat currently handles 3 different operations
      1. The heartbeat itself to indicate that the container is alive.
      2. Reporting new events sent over by the task
      3. Fetching events which are to be consumed by the task.

      All three can be independent.

      This does not affect the Tez wire protocol itself, which will continue to use a single call for all three. However this would get split within the default Tez AM plugin.

      The eventual intent is to allow plugins to control this functionality independently.
      An example: The addition of a eventsAvailable(Task*) on the TaskCommunicator allows a Plugin to get events the moment they're available and push them into the executors. Meanwhile, the heartbeat becomes just that - a heartbeat - and the frequency can be reduced to multiple seconds instead of the current milliseconds.

      Attachments

        1. TEZ-3044.1.wip.txt
          61 kB
          Siddharth Seth

        Activity

          People

            sseth Siddharth Seth
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: