Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11737

String may not be serialized correctly with Kyro

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.5.1, 1.6.0
    • 1.5.3, 1.6.0
    • SQL
    • None

    Description

      When run in cluster mode, the driver may have different memory (and configs) than executor, also if Kyro is used, then string can not be collected back to driver:

      >>> sqlContext.range(10).selectExpr("repeat(cast(id as string), 9)").show()
+----------------------------+
|repeat(cast(id as string),9)|
+----------------------------+
|                   	0|
|                   	1|
|                   	2|
|                   	3|
|                   	4|
|                   	5|
|                   	6|
|                   	7|
|                   	8|
|                   	9|
+----------------------------+

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. SPARK-11657 Bad Dataframe data read from parquet

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-14524 In SparkSQL, it can't be select column of String type because of UTF8String when setting more than 32G for executors.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Pull request #9704 [Github] Pull Request #9704 (davies)

          Activity

            People

              davies Davies Liu
              davies Davies Liu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: