Uploaded image for project: 'Falcon'
  1. Falcon
  2. FALCON-285

Support Lineage information capture

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.5
    • None
    • None

    Description

      We would want to capture enough information from entities and the associated executions to drive lineage tracing and visualization.
      The plan is to capture lineage specific information - which is all the inputs, process and associated workflows, outputs for each execution in Falcon post processing step. Post message consumption, this information is acted upon and persisted in falcon server for each output generated. This should work for replication as well. I need to think about eviction which was not invoking post processing but will soon.

      Attachments

        Issue Links

          1.
          Capture information in process entity about the user workflow Sub-task Resolved Venkatesh Seetharam
          2.
          Record lineage information in post processing Sub-task Resolved Venkatesh Seetharam
          3.
          Persist lineage information into a persistent store Sub-task Resolved Venkatesh Seetharam
          4.
          Provide REST APIs for discovering lineage metadata over the store Sub-task Resolved Venkatesh Seetharam
          5.
          Visualize lineage information on the dashboard Sub-task Closed Haohui Mai
          6.
          Document lineage feature Sub-task Resolved Sowmya Ramesh
          7.
          Process lineage information for Replication policies Sub-task Resolved Sowmya Ramesh
          8.
          Add indexing to the graph property keys Sub-task Resolved Venkatesh Seetharam
          9.
          Clean up historical data periodically Sub-task Open Unassigned
          10.
          Add existing entities from store at startup if graphed is empty for backwards compatibility Sub-task Resolved Ajay Yadav
          11.
          Bug when MetadataMappingService is not configured as one of the application services Sub-task Resolved Venkatesh Seetharam
          12.
          REST API does not conform to Rexster Sub-task Resolved Venkatesh Seetharam
          13.
          Instance id's captured are of different formats in process and feed Sub-task Resolved Venkatesh Seetharam
          14.
          Lineage recording fails with NPE for processes with >1 inputs Sub-task Resolved Venkatesh Seetharam
          15.
          Add a REST API to get properties for a given vertex Sub-task Resolved Venkatesh Seetharam
          16.
          Remove Graph dump option in CLI Sub-task Resolved Venkatesh Seetharam
          17.
          Show vertex information in the web UI Sub-task Closed Haohui Mai
          18.
          Display lineage link only for jobs that are succeeded in the web UI Sub-task Closed Haohui Mai
          19.
          Lineage breaks if feed.xml doesn't have the date pattern in feed path location Sub-task Resolved Sowmya Ramesh
          20.
          Preserve data type for properties in a vertex Sub-task Resolved Ajay Yadav
          21.
          Process lineage information for Retention policies Sub-task Resolved Sowmya Ramesh
          22.
          Enable metrics for Titan Sub-task Closed Ajay Yadav
          23.
          Upgrade Blueprints to latest and Titan to latest (0.5) Sub-task Open Unassigned

          Activity

            People

              Unassigned Unassigned
              svenkat Venkatesh Seetharam
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: