Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
There are multiple Hdfs Output Operator in Malhar lib. A few of them are inefficient because they immediately flush after they write a tuple to output stream. This makes the operator very slow.
For a POC we created a high-throughput fault-tolerant hdfs output operator. We need the capabilities of that operator in library and refactor the existing implementations to avoid duplicating functionality.