Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4687

SparkContext#addFile doesn't keep file folder information

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 1.3.0, 1.4.0
    • Spark Core
    • None

    Description

      Files added with SparkContext#addFile are loaded with Utils#fetchFile before a task starts. However, Utils#fetchFile puts all files under the Spart root on the worker node. We should have an option to keep the folder information.

      Attachments

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. SPARK-6144 When in cluster mode using ADD JAR with a hdfs:// sourced jar will fail

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          is depended upon by

          Sub-task - The sub-task of the issue HIVE-8851 Broadcast files for small tables via SparkContext.addFile() and SparkFiles.get() [Spark Branch]

          • Major - Major loss of function.
          • Open

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-3145 Hive on Spark umbrella

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #3670 [Github] Pull Request #3670 (sryza)

          Activity

            People

              sandyr Sandy Ryza
              jxiang Jimmy Xiang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: