Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1260

Allow KeyValueWriter to support writing list of values also

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • None
    • None
    • Reviewed

    Description

      TEZ-1228 adds support to IFile for storing K,L<V>. Currently KeyValueWriter allows write of K,V

      public void write(Object key, Object value) throws IOException;

      We should add support for

      public void write(Object key, Iterable<Object> values) throws IOException;

      taking advantage of TEZ-1228. In few cases, pig unwraps key, list<values> and writes them as separate K,V pairs. This can avoid that overhead. That may enable us to even add something similar to hash based partial aggregation for join like what we do for groupby.

      Attachments

        1. TEZ-1260.2.patch
          10 kB
          Rajesh Balamohan
        2. TEZ-1260.1.patch
          5 kB
          Rajesh Balamohan

        Issue Links

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rohini Rohini Palaniswamy
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: