Description
In some cases when timestamps contain nanoseconds they will be parsed incorrectly.
Examples:
"2016-05-14T15:12:14.0034567Z" -> "2016-05-14 15:12:14.034567"
"2016-05-14T15:12:14.000345678Z" -> "2016-05-14 15:12:14.345678"
The issue seems to be happening in DateTimeUtils.stringToTimestamp(). It assumes that only 6 digit fraction of a second will be passed.
With this being the case I would suggest either discarding nanoseconds automatically, or throw an exception prompting to pre-format timestamps to microsecond precision first before casting to the Timestamp.
Attachments
Issue Links
- relates to
-
SPARK-14428 [SQL] Allow more flexibility when parsing dates and timestamps in json datasources
- Resolved
- links to