Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.3.3, 2.4.1, 2.4.2, 2.4.3, 2.4.4
-
None
Description
In spark version (2.3.3 , 2.4.1 , 2.4.2 , 2.4.3 , 2.4.4) use Broadcast.destroy() method can not destroy the broadcast data, the driver and executor storage memory in spark ui is continuous increase。
val batch = Seq(1 to 9999: _*)
val strSeq = batch.map(i => s"xxh-$i")
val rdd = sc.parallelize(strSeq)
rdd.cache()
batch.foreach(_ => {
val broc = sc.broadcast(strSeq)
rdd.map(id => broc.value.contains(id)).collect()
broc.destroy()
})
