Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27948

Wrong results when using materialized views with non-deterministic/dynamic functions

    XMLWordPrintableJSON

Details

    Description

      There are certain SQL functions that return different results across different executions. Usually we refer to these functions as non-deterministic or dynamic. Some examples are: UNIX_TIMESTAMP(), CURRENT_TIMESTAMP, CURRENT_DATE, etc.

      When a materialized view definition contains such functions the queries that are using this view may return wrong results.

      Consider the following scenario where we populate the employee table with timestamps representing the future. For making this easily reproable in self-contained test the timestamps are only a few seconds apart.

      CREATE TABLE EMPS (ENAME STRING, BIRTH_EPOCH_SECS INT) STORED AS ORC TBLPROPERTIES ('transactional'='true');
      
      INSERT INTO EMPS
      VALUES ('Victor', UNIX_TIMESTAMP()),
             ('Alex', UNIX_TIMESTAMP() + 2),
             ('Bob', UNIX_TIMESTAMP() + 5),
             ('Alice', UNIX_TIMESTAMP() + 10);
      
      CREATE MATERIALIZED VIEW v_emp AS SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP();
      

      When the materialized view is created it is populated with only the rows that match the timestamp at the given time.

      To demonstrate the problem run the following queries with view based rewritting disabled and enabled.

      set hive.materializedview.rewriting.sql=false;
      SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP();
      
      Victor	1702302786
      Alex	1702302788
      Bob	1702302791
      
      set hive.materializedview.rewriting.sql=true;
      SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP();
      
      Victor	1702302786
      Alex	1702302788
      

      Naturally the second query should return more rows than the first one since UNIX_TIMESTAMP is constantly growing. However, when view based rewritting is in use the second query will use the results from the materialized view which are by now obsolete (missing Bob entry).

      Attachments

        1. materialized_view_unix_timestamp.q
          0.7 kB
          Stamatis Zampetakis

        Issue Links

          Activity

            People

              kkasa Krisztian Kasa
              zabetak Stamatis Zampetakis
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: