Description
We create new Alias with new exprId in Dataset.dropDuplicates now. However it causes problem when we want to select the columns as follows:
val ds = Seq(("a", 1), ("a", 2), ("b", 1), ("a", 1)).toDS() // ds("_2") will cause analysis exception ds.dropDuplicates("_1").select(ds("_1").as[String], ds("_2").as[Int])
Attachments
Issue Links
- breaks
-
SPARK-19065 dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute replacement
-
- Resolved
-
- links to
