Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • llap
    • None
    • None
    • None

    Description

      This is patch for HIVE-7428. For now this will be in llap branch as hive has not bumped up the tez version yet.

      Attachments

        1. HIVE-10397.trunk.patch
          7 kB
          Prasanth Jayachandran
        2. HIVE-10397.patch
          17 kB
          Prasanth Jayachandran
        3. HIVE-10397.2.patch
          7 kB
          Prasanth Jayachandran

        Issue Links

          Activity

            ekoifman Eugene Koifman added a comment -

            Is there plan to upgrade Tez in branch-1? Is that risky?

            ekoifman Eugene Koifman added a comment - Is there plan to upgrade Tez in branch-1? Is that risky?

            IIRC, this needs tez 0.7.0 or greater version of tez. I don't think we updated the tez version in branch-1 yet.

            prasanth_j Prasanth Jayachandran added a comment - IIRC, this needs tez 0.7.0 or greater version of tez. I don't think we updated the tez version in branch-1 yet.
            ekoifman Eugene Koifman added a comment -

            prasanth_j gopalv Is this portable to branch-1 or will it pull a bunch of other changes?

            ekoifman Eugene Koifman added a comment - prasanth_j gopalv Is this portable to branch-1 or will it pull a bunch of other changes?

            The old code in llap branch was reverted and .2 patch (same as .2 patch from HIVE-7428) is committed to llap branch.

            prasanth_j Prasanth Jayachandran added a comment - The old code in llap branch was reverted and .2 patch (same as .2 patch from HIVE-7428 ) is committed to llap branch.

            This patch needs to be reverted and the new patch from HIVE-7428 should be applied.

            prasanth_j Prasanth Jayachandran added a comment - This patch needs to be reverted and the new patch from HIVE-7428 should be applied.

            gopalv I made a patch based on your suggestions. This patch removes OrcInputFormat implementing SplitSizeEstimator. Instead a new generic ColumnSizeEstimator is added to tez code path. But this will still not solve the hadoop-1 issue or compatible with older tez installs.

            prasanth_j Prasanth Jayachandran added a comment - gopalv I made a patch based on your suggestions. This patch removes OrcInputFormat implementing SplitSizeEstimator. Instead a new generic ColumnSizeEstimator is added to tez code path. But this will still not solve the hadoop-1 issue or compatible with older tez installs.

            prasanth_j: this is fine for LLAP branch, but this patch might break hadoop-1 builds.

            Because of the inheritance dependencies for Tez, you cannot make the InputFormat implement SplitSizeEstimator in trunk.

            The original idea to do this via an extra object (because java has no mixins) was to implement OrcSplit -> ColumnarSplit + instantiate a generic ColumnarSplitEstimator only inside Tez codepath would do the trick.

            gopalv Gopal Vijayaraghavan added a comment - prasanth_j : this is fine for LLAP branch, but this patch might break hadoop-1 builds. Because of the inheritance dependencies for Tez, you cannot make the InputFormat implement SplitSizeEstimator in trunk. The original idea to do this via an extra object (because java has no mixins) was to implement OrcSplit -> ColumnarSplit + instantiate a generic ColumnarSplitEstimator only inside Tez codepath would do the trick.

            Committed to llap branch.

            prasanth_j Prasanth Jayachandran added a comment - Committed to llap branch.

            We should port this back to trunk later when hive moves to new version of tez which has SplitSizeEstimator interface.

            prasanth_j Prasanth Jayachandran added a comment - We should port this back to trunk later when hive moves to new version of tez which has SplitSizeEstimator interface.
            prasanth_j Prasanth Jayachandran added a comment - FYI.. gopalv and seth.siddharth@gmail.com ..

            People

              prasanth_j Prasanth Jayachandran
              prasanth_j Prasanth Jayachandran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: