Details
-
New Feature
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
The Spark runner provides a convenient feature which is the metric sink.
By configuration, it allows us to configure a metric sink using metrics.properties configuration containing:
driver.sink.graphite.class=org.apache.beam.runners.spark.metrics.sink.GraphiteSink driver.sink.graphite.host=localhost driver.sink.graphite.port=2003 driver.sink.graphite.prefix=spark driver.sink.graphite.period=1 driver.sink.graphite.unit=SECONDS
This approach is very convenient to send the metric to any sink. I think we can apply this logic in generic way working with any runner.
The idea is to use a metric poll thread in the pipeline (that we can enable via PipelineOptions) and send to a sink.
I started a PoC about that.
Attachments
Issue Links
- is superceded by
-
BEAM-3310 Push metrics to a backend in an runner agnostic way
- Open