Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
Description
When adding task from a running Helix streaming job, we need the capability to split an existing workunit.
We can reuse most of the attributes within the old workunit, and just need to update following props:
task.id
writer.output.dir
partition.id
gobblin.kafka.streaming.numPartitions
Since we get the partition watermark from state store, so we don't need to re-calculate that within the workunit. All kafka related properties can also be reused.