Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Duplicate
-
None
-
None
-
None
-
None
Description
I've found it quite cumbersome to work with dates thus far in Spark, it can be hard to reason about the timeformat and what type you're working with, for instance:
say that I have a date in the format
2017-20-12
// Y-D-M
In order to parse that into a Date, I have to perform several conversions.
to_date( unix_timestamp(col("date"), dateFormat) .cast("timestamp")) .alias("date")
I propose simplifying this by adding a to_date function (exists) but adding one that accepts a format for that date. I also propose a to_timestamp function that also supports a format.
so that you can avoid entirely the above conversion.
It's also worth mentioning that many other databases support this. For instance, mysql has the STR_TO_DATE function, netezza supports the to_timestamp semantic.
Attachments
Issue Links
- duplicates
-
SPARK-16609 Single function for parsing timestamps/dates
- Resolved