Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1204

Visualize StreamGraph and ExecutionPlan

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.13.0
    • 0.13.0
    • None
    • None

    Description

      Based on SAMZA-1178, we are able to generate the JSON representation of StreamGraph and ExecutionPlan. We need to provide a visualizer to render the json as DAG and let users to view it.

      First look: https://xinyuiscool.github.io/visualizer/plan.html. This is based on the example graph JSON generated in TestJobGraphJsonGenerator.

      For the api-test job:
      ![image](https://cloud.githubusercontent.com/assets/24738423/25154090/00f6cd76-2444-11e7-84aa-efa7d5fb735f.png)

      For the join-test job:
      ![image](https://cloud.githubusercontent.com/assets/24738423/25154115/19de1182-2444-11e7-9283-3b68a122201d.png)

      How to visualize the plan (locally):
      Once a Samza application (using fluent API) is deployed, an execution plan will be generated by the ExecutionPlanner. The plan JSON will be written to a file (plan.json) under the bin directory, which also contains the plan.html and javscripts (js folder). So the bin dir has the following for visualization:
      ./bin

      • plan.json
      • plan.html
      • /js
      • d3.js
      • dagre.js
      • planToDagre.js

      Then deployed locally, the user can directly view the plan by enter the following in a browser:
      **file:///my/samza/app/dir/bin/plan.html**

      To visualize the plan remotely, the user can either scp the files into local box and view the plan there, or bring up a simple http server to serve the page.

      In the future when we have the Samza dashboard, we can add the plan visualization there to avoid the above manual steps.

      Attachments

        1. dag1.png
          138 kB
          Xinyu Liu
        2. dag2.png
          159 kB
          Xinyu Liu

        Issue Links

          Activity

            People

              xinyu Xinyu Liu
              xinyu Xinyu Liu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: