Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-7602

Possibility to force repartition on read/select

    XMLWordPrintableJSON

Details

    Description

      It will be nice and usefull in certain situations to have the capacity to do repartition like in spark (https://spark.apache.org/docs/latest/rdd-programming-guide.html)

      either an automatically repartition in certain limit or possibility to indicate the desired repartition or both options.

      The only way (that I know now) to do that with Drill is to change store.parquet.block-size and regenerate the input file then do the request (but it will be nice to have the ability to do that on read)

      illustration : with 2 Parquets files file1 of 50Mo (1 million rows) and file2 of 1Mo (5000 rows)

      CREATE TABLE dfs.test.`result_from_1_parquet` AS 
      (SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1` ON ...)
      => ~ 50min
      
      -- Today we have to change the parquet block size to force multiple parquet files
      ALTER SESSION SET `store.parquet.block-size` = 1048576;
      -- Repartition data
      CREATE TABLE dfs.test.`file1_bis` AS (SELECT * FROM dfs.test.`file1`);
      -- then Launch the request
      CREATE TABLE dfs.test.`result_from_1_parquet` AS 
      (SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1_bis` ON ...)
      => ~ 1min
      

      So it's possible to save a lot of time (depending of configuration of cluster) by simply forcing more input file.
      it would be useful not to have to regenerate the files with the ideal fragmentation before request.

      This situation easily appears when making inequality JOIN (to lookup ip in ip range for example) on not so big dataset:

      ALTER SESSION SET `planner.enable_nljoin_for_scalar_only` = false;
      SELECT  * 
      FROM dfs.test.`a_pqt` AS a
      INNER JOIN dfs.test.`b_pqt` AS b
      ON inet_aton(b.ip) >= inet_aton(a.ip_first) AND inet_aton(b.ip) <= inet_aton(a.ip_last);
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            benj641 benj
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: