Description
Currently nondeterministic expression is broken without a explicit initialization phase.
Let me take `MonotonicallyIncreasingID` as an example. This expression need a mutable state to remember how many times it has been evaluated, so we use `@transient var count: Long` there. By being transient, the `count` will be reset to 0 and *only* to 0 when serialize and deserialize it, as deserialize transient variable will result to default value. There is no way to use another initial value for `count`, until we add the explicit initialization phase.
For now no nondeterministic expression need this feature, but we may add new ones with the need of a different initial value for mutable state in the future.
Another use case is local execution for LocalRelation, there is no serialize and deserialize phase and thus we can't reset mutable states for it.
Attachments
Issue Links
- is related to
-
SPARK-9462 Initialize nondeterministic expressions in code gen fallback mode
- Resolved
- links to