Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31684

Overwrite partition failed with 'WRONG FS' when the target partition is not belong to the filesystem as same as the table

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.3, 2.2.3, 2.3.4, 2.4.5, 3.0.0, 3.1.0
    • 3.1.0
    • SQL
    • None

    Description

      With https://issues.apache.org/jira/browse/SPARK-18107, we will disable the underlying replace(overwrite) and instead do delete in spark side and only do copy in hive side to bypass the performance issue - https://issues.apache.org/jira/browse/HIVE-11940

      Conditionally, if the table location and partition location do not belong to the same [[FileSystem]], We should not disable hive overwrite. Otherwise, hive will use the [[FileSystem]] instance belong to the table location to copy files, which will fail [FileSystem#checkPath]
      see https://github.com/apache/hive/blob/rel/release-2.3.7/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1648-L1659

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31675 Fail to insert data to a table with remote location which causes by hive encryption check

          • Major - Major loss of function.
          • In Progress
          links to

          Pull request #28511 [Github] Pull Request #28511 (yaooqinn)

          Activity

            People

              Qin Yao Kent Yao 2
              Qin Yao Kent Yao 2
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: