Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1563

TezClient.submitDAGSession alters DAG local resources regardless of DAG submission

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.5.0
    • 0.5.1
    • None
    • None
    • Reviewed

    Description

      In TezClient#submitDAGSesssion(Dag), a DAGPlan is created from the DAG before the DAGClientAMProtocolBlockingPB is instantiated. When the application isn't running, waitForProxy() will throw a SessionNotRunning Exception.

      The problem is that the internal state of the DAG is modified, regardless of whether the DAG is actually run or not.

      DAGPlan dagPlan = dag.createDag(amConfig.getTezConfiguration());
      

      The createDag method will ultimately call addTaskLocalFiles for each Vertex in the DAG

      // add common task files for this DAG
      vertex.addTaskLocalFiles(commonTaskLocalFiles);
      

      Because the DAG's state is modified, Vertex#addTaskLocalFiles(Map) will fail if any resources are added multiple times. As such, if the application is not running and SessionNotRunning is thrown, that same DAG cannot be passed in to run the DAG after the application is started again.

      Additionally, DAG is missing a getTaskLocalFiles method as compared to Vertex which would be good to add to make the two classes more uniform.

      Attachments

        1. TEZ-1563.2.patch
          10 kB
          Bikas Saha
        2. TEZ-1563.1.patch
          4 kB
          Bikas Saha

        Issue Links

          Activity

            People

              bikassaha Bikas Saha
              elserj Josh Elser
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: