Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-6097

Follow up all timeline related apis and methods to honor timezone if overridden

    XMLWordPrintableJSON

Details

    Description

      I see that HoodieTableConfig has a config to let users to override timezone for commit time generation. but looks like there are some places where we use current system's zone instead of honoring the config. 

      public static Date parseDateFromInstantTime(String timestamp) throws ParseException {
        try {
          // Enables backwards compatibility with non-millisecond granularity instants
          String timestampInMillis = timestamp;
          if (isSecondGranularity(timestamp)) {
            // Add milliseconds to the instant in order to parse successfully
            timestampInMillis = timestamp + DEFAULT_MILLIS_EXT;
          } else if (timestamp.length() > MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH) {
            // compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with ms granularity
            timestampInMillis = timestamp.substring(0, MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH);
          }
      
          LocalDateTime dt = LocalDateTime.parse(timestampInMillis, MILLIS_INSTANT_TIME_FORMATTER);
          return Date.from(dt.atZone(ZoneId.systemDefault()).toInstant());
        } catch (DateTimeParseException e) {
          throw new ParseException(e.getMessage(), e.getErrorIndex());
        }
      } 

       

      also, we are adding a guard rail to archival that archival should not go past clean commits. when clean is based on hours, we have a fix which may not work for other time zones. Ref patch: https://github.com/apache/hudi/pull/8422

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            shivnarayan sivabalan narayanan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: