Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.0.0
-
None
-
None
Description
I have a Dataframe with a struct and I need to rename some fields to lower case before saving it to cassandra.
It turns out that it's not possible to cast a boolean field of a struct to another boolean field in the renamed struct:
case class ClassWithBoolean(flag: Boolean)
case class Parent(cwb: ClassWithBoolean)val structCwb: DataType = StructType(Seq(
StructField("flag", BooleanType, true)
))Seq(Parent(ClassWithBoolean(true)))
.toDF
.withColumn("cwb", $"cwb".cast(structCwb))
.collectscala.MatchError: BooleanType (of class org.apache.spark.sql.types.BooleanType$)
A workaround is to temporarily cast the field to an Integer and back:
val structCwbTmp: DataType = StructType(Seq(
StructField("flag", IntegerType, true)
))Seq(Parent(ClassWithBoolean(true)))
.toDF
.withColumn("cwb", $"cwb".cast(structCwbTmp))
.withColumn("cwb", $"cwb".cast(structCwb))
.collect