Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
TEZ-1228 adds support to IFile for storing K,L<V>. Currently KeyValueWriter allows write of K,V
public void write(Object key, Object value) throws IOException;
We should add support for
public void write(Object key, Iterable<Object> values) throws IOException;
taking advantage of TEZ-1228. In few cases, pig unwraps key, list<values> and writes them as separate K,V pairs. This can avoid that overhead. That may enable us to even add something similar to hash based partial aggregation for join like what we do for groupby.
Attachments
Attachments
Issue Links
- requires
-
TEZ-1228 Prototype IFile : Define a memory & merge optimized vertex-intermediate file format for Tez
- Closed