Blocker Description
Reproduce:
df = spark.range(0, 1000) from pyspark.sql.functions import udf def foo(x): raise StopIteration() df.withColumn('v', udf(foo)).show() # Results # +---+---+ # | id| v| # +---+---+ # +---+---+
I think the task should fail in this case
Attachments
Issue Links
- is duplicated by
-
SPARK-24034 StopIteration in pyspark mapper gives partial results
-
- Resolved
-
- links to
(1 links to)