Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22938

Assert that SQLConf.get is accessed only on the driver.

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.1
    • 2.3.0
    • SQL
    • None

    Description

      Assert if code tries to access SQLConf.get on executor.
      This can lead to hard to detect bugs, where the executor will read fallbackConf, falling back to default config values, ignoring potentially changed non-default configs.
      If a config is to be passed to executor code, it needs to be read on the driver, and passed explicitly.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-38328 SQLConf.get flaky causes NON-default spark session settings being lost

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23894 Flaky Test: BucketedWriteWithoutHiveSupportSuite

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • In Progress
          links to

          Pull request #20136 [Github] Pull Request #20136 (juliuszsompolski)

          Pull request #21190 [Github] Pull Request #21190 (cloud-fan)

          Pull request #21341 [Github] Pull Request #21341 (cloud-fan)

          Activity

            People

              juliuszsompolski Juliusz Sompolski
              juliuszsompolski Juliusz Sompolski
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: