Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5563

Timezone lookup may be ambiguous

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • Impala 2.0
    • Impala 3.1.0
    • Backend

    Description

      When using functions like to_utc_timestamp that take a string timezone parameter, if the timezone is not a 'region specifier' [1] (i.e. a key into the timezone database entries), then Impala attempts to match the string parameter against a number of other properties of the timezones in the database [2]:

      • a zone's std abbreviation
      • a zone's std full name
      • a zone's dst abbreviation
      • a zone's dst full name
      time_zone_ptr TimezoneDatabase::FindTimezone(
    const string& tz, const TimestampValue& tv, bool tv_in_utc) {
  ...
  // See if they specified a zone id
  time_zone_ptr tzp = tz_database_.time_zone_from_region(tz);
  if (tzp != NULL) return tzp;

  for (vector<string>::const_iterator iter = tz_region_list_.begin();
       iter != tz_region_list_.end(); ++iter) {
    time_zone_ptr tzp = tz_database_.time_zone_from_region(*iter);
    DCHECK(tzp != NULL);
    if (tzp->dst_zone_abbrev() == tz) return tzp;
    if (tzp->std_zone_abbrev() == tz) return tzp;
    if (tzp->dst_zone_name() == tz) return tzp;
    if (tzp->std_zone_name() == tz) return tzp;
  }
  return time_zone_ptr();
}

      This can result in ambiguous zones being used because the properties listed above are not unique, e.g.

      mj@mj-desktop:~/dev/Impala$ grep CEST be/src/exprs/timezone_db.cc 
\"Africa/Ceuta\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Antarctica/Troll\",\"UTC\",\"Coordinated Universal Time\",\"CEST\",\"Central European Summer Time\",\"+00:00:00\",\"+02:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Arctic/Longyearbyen\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Atlantic/Jan_Mayen\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"CET\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+02:00:00\",\"-1;0;10\",\"+02:00:00\"\n\
\"ECT\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Amsterdam\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Andorra\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Belgrade\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Berlin\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Bratislava\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Brussels\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Budapest\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Busingen\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Copenhagen\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Gibraltar\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Ljubljana\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Luxembourg\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Madrid\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Malta\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Monaco\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Oslo\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Paris\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Podgorica\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Prague\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Rome\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/San_Marino\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Sarajevo\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Skopje\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Stockholm\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Tirane\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Vaduz\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Vatican\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Vienna\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Warsaw\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Zagreb\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Europe/Zurich\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\
\"Poland\",\"CET\",\"Central European Time\",\"CEST\",\"Central European Summer Time\",\"+01:00:00\",\"+01:00:00\",\"-1;0;3\",\"+01:00:00\",\"-1;0;10\",\"+01:00:00\"\n\

      Based on the code, CEST may match a zone with many different GMT offsets or DST adjustments/dates. Depending on the order in which the list of zones happens to be iterated, the results will be different.

      1: http://www.boost.org/doc/libs/1_64_0/doc/html/date_time/local_time.html
      2: https://github.com/apache/incubator-impala/blob/2.9.0/be/src/exprs/timezone_db.cc#L45

      Attachments

        Issue Links

          is part of

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3307 add support for IANA time zone database

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2546 Invalid time zones are ignored but should be rejected

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2542 PDT and PST use the same offset

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3307 add support for IANA time zone database

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3169 to_utc_timestamp blank value

          • Major - Major loss of function.
          • Closed

          Activity

            People

              attilaj Attila Jeges
              mjacobs Matthew Jacobs
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: