Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6144

Implement non-staged MapJoin

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 0.13.0
    • Query Processor

    Description

      For map join, all data in small aliases are hashed and stored into temporary file in MapRedLocalTask. But for some aliases without filter or projection, it seemed not necessary to do that. For example.

      select a.* from src a join src b on a.key=b.key;
      

      makes plan like this.

      STAGE PLANS:
        Stage: Stage-4
          Map Reduce Local Work
            Alias -> Map Local Tables:
              a 
                Fetch Operator
                  limit: -1
            Alias -> Map Local Operator Tree:
              a 
                TableScan
                  alias: a
                  HashTable Sink Operator
                    condition expressions:
                      0 {key} {value}
                      1 
                    handleSkewJoin: false
                    keys:
                      0 [Column[key]]
                      1 [Column[key]]
                    Position of Big Table: 1
      
        Stage: Stage-3
          Map Reduce
            Alias -> Map Operator Tree:
              b 
                TableScan
                  alias: b
                  Map Join Operator
                    condition map:
                         Inner Join 0 to 1
                    condition expressions:
                      0 {key} {value}
                      1 
                    handleSkewJoin: false
                    keys:
                      0 [Column[key]]
                      1 [Column[key]]
                    outputColumnNames: _col0, _col1
                    Position of Big Table: 1
                    Select Operator
                      File Output Operator
            Local Work:
              Map Reduce Local Work
        Stage: Stage-0
          Fetch Operator
      

      table src(a) is fetched and stored as-is in MRLocalTask. With this patch, plan can be like below.

        Stage: Stage-3
          Map Reduce
            Alias -> Map Operator Tree:
              b 
                TableScan
                  alias: b
                  Map Join Operator
                    condition map:
                         Inner Join 0 to 1
                    condition expressions:
                      0 {key} {value}
                      1 
                    handleSkewJoin: false
                    keys:
                      0 [Column[key]]
                      1 [Column[key]]
                    outputColumnNames: _col0, _col1
                    Position of Big Table: 1
                    Select Operator
                        File Output Operator
            Local Work:
              Map Reduce Local Work
                Alias -> Map Local Tables:
                  a 
                    Fetch Operator
                      limit: -1
                Alias -> Map Local Operator Tree:
                  a 
                    TableScan
                      alias: a
                Has Any Stage Alias: false
        Stage: Stage-0
          Fetch Operator
      

      Attachments

        1. HIVE-6144.9.patch.txt
          1010 kB
          Navis Ryu
        2. HIVE-6144.8.patch.txt
          1011 kB
          Navis Ryu
        3. HIVE-6144.7.patch.txt
          1013 kB
          Navis Ryu
        4. HIVE-6144.6.patch.txt
          1.04 MB
          Navis Ryu
        5. HIVE-6144.5.patch.txt
          1.14 MB
          Navis Ryu
        6. HIVE-6144.4.patch.txt
          1.14 MB
          Navis Ryu
        7. HIVE-6144.3.patch.txt
          67 kB
          Navis Ryu
        8. HIVE-6144.2.patch.txt
          66 kB
          Navis Ryu
        9. HIVE-6144.1.patch.txt
          65 kB
          Navis Ryu

        Issue Links

          Activity

            People

              navis Navis Ryu
              navis Navis Ryu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: