Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36603

Use WeakReference over SoftReference in LevelDB to avoid retaining iterator refs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.1.2
    • 3.2.0, 3.1.3
    • Spark Core
    • None

    Description

      See discussion at https://github.com/apache/spark/pull/28769#issuecomment-906722390

      The SoftReference used to track iterators in LevelDB can prevent them from being GCed early enough, thus finalized and closed. A WeakReference would be more readily collected, and solves an apparent actual issue running a 3.1.x SHS.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31929 local cache size exceeding "spark.history.store.maxDiskUsage" triggered "java.io.IOException" in history server on Windows

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #33859 [Github] Pull Request #33859 (srowen)

          Pull request #33859 [Github] Pull Request #33859 (srowen)

          Activity

            People

              srowen Sean R. Owen
              srowen Sean R. Owen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: