Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-4231

On update operation with 3.1v, cloned spark session is used and set properties are lost.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.2.0
    • None
    • None

    Description

      Update operation with bad records property fails with 3.1v.

      [Steps to reproduce]:

      0: jdbc:hive2://linux-221:22550/> set carbon.options.bad.records.action=force;

      ------------------------------------------+

      key value

      ------------------------------------------+

      carbon.options.bad.records.action force

      ------------------------------------------+

      1 row selected (0.04 seconds)

      0: jdbc:hive2://linux-221:22550/> create table t_carbn1(item_type_cd int, sell_price bigint, profit decimal(10,4), item_name string, update_time timestamp) stored a

      ---------

      Result

      ---------

      ---------

      No rows selected (2.117 seconds)

      0: jdbc:hive2://linux-221:22550/> insert into t_carbn1 select 2, 100000,23.3,'Apple','2012-11-11 11:11:11';

      INFO : Execution ID: 858

      -------------

      Segment ID

      -------------

      0

      -------------

      1 row selected (4.278 seconds)

      0: jdbc:hive2://linux-221:22550/> update t_carbn1 set (item_type_cd) = (item_type_cd/1);

      Error: org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.RuntimeException: Update operation failed. DataLoad failure

      [Root cause]:

      On update command, persist is called and with latest 3.1 spark changes, spark returns a cloned SparkSession from cacheManager with all specified configurations disabled. As now its using different sparkSession for 3.1 which is not initialized in CarbonEnv. So CarbonEnv.init is called where new CarbonSessionInfo is created with no sessionParams. So, the properties set were not accessible.

      Spark creates cloned spark session based on following properties:

      1. spark.sql.optimizer.canChangeCachedPlanOutputPartitioning

      2. spark.sql.sources.bucketing.autoBucketedScan.enabled

      3.  spark.sql.adaptive.enabled

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            Shreelekhya SHREELEKHYA GAMPA
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: