Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Duplicate
-
1.5.2
-
None
-
Centos
Description
(related Issue:https://github.com/apache/spark/pull/8210/files)
When we set 32G(or more) for executor, select the column of String type, it shows the Wrong result, such as:
'abcde' (less than 8 chars) => '' (it will show nothing)
'abcdefghijklmn' (more than 8 chars) =>'ijklmn' ( it will cut the the front of 8 chars)
However, when we set 31G( or less) for executor, all is good.
We also have debugged this problem, we found that SparkSQL uses UTF8String internally, it depends on some properties of locally JVM Memmory allocation ( see class 'org.apache.spark.unsafe.Platform').
Attachments
Issue Links
- is duplicated by
-
SPARK-11737 String may not be serialized correctly with Kyro
- Resolved