Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
I see that HoodieTableConfig has a config to let users to override timezone for commit time generation. but looks like there are some places where we use current system's zone instead of honoring the config.
public static Date parseDateFromInstantTime(String timestamp) throws ParseException { try { // Enables backwards compatibility with non-millisecond granularity instants String timestampInMillis = timestamp; if (isSecondGranularity(timestamp)) { // Add milliseconds to the instant in order to parse successfully timestampInMillis = timestamp + DEFAULT_MILLIS_EXT; } else if (timestamp.length() > MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH) { // compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with ms granularity timestampInMillis = timestamp.substring(0, MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH); } LocalDateTime dt = LocalDateTime.parse(timestampInMillis, MILLIS_INSTANT_TIME_FORMATTER); return Date.from(dt.atZone(ZoneId.systemDefault()).toInstant()); } catch (DateTimeParseException e) { throw new ParseException(e.getMessage(), e.getErrorIndex()); } }
also, we are adding a guard rail to archival that archival should not go past clean commits. when clean is based on hours, we have a fix which may not work for other time zones. Ref patch: https://github.com/apache/hudi/pull/8422