Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6936

SQLContext.sql() caused deadlock in multi-thread env

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.5.0
    • SQL
    • JDK 1.8.x, RedHat
      Linux version 2.6.32-431.23.3.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Wed Jul 16 06:12:23 EDT 2014

    • Important

    Description

      Doing (the same query) in more than one threads with SQLConext.sql may lead to deadlock. Here is a way to reproduce it (since this is multi-thread issue, the reproduction may or may not be so easy).

      1. Register a relatively big table.

      2. Create two different classes and in the classes, do the same query in a method and put the results in a set and print out the set size.

      3. Create two threads to use an object from each class in the run method. Start the threads. For my tests, it can have a deadlock just in a few runs.

      Attachments

        Activity

          People

            marmbrus Michael Armbrust
            zwu.net@gmail.com Paul Wu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: