Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
4.0.0-beta-1
Description
There are certain SQL functions that return different results across different executions. Usually we refer to these functions as non-deterministic or dynamic. Some examples are: UNIX_TIMESTAMP(), CURRENT_TIMESTAMP, CURRENT_DATE, etc.
When a materialized view definition contains such functions the queries that are using this view may return wrong results.
Consider the following scenario where we populate the employee table with timestamps representing the future. For making this easily reproable in self-contained test the timestamps are only a few seconds apart.
CREATE TABLE EMPS (ENAME STRING, BIRTH_EPOCH_SECS INT) STORED AS ORC TBLPROPERTIES ('transactional'='true'); INSERT INTO EMPS VALUES ('Victor', UNIX_TIMESTAMP()), ('Alex', UNIX_TIMESTAMP() + 2), ('Bob', UNIX_TIMESTAMP() + 5), ('Alice', UNIX_TIMESTAMP() + 10); CREATE MATERIALIZED VIEW v_emp AS SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP();
When the materialized view is created it is populated with only the rows that match the timestamp at the given time.
To demonstrate the problem run the following queries with view based rewritting disabled and enabled.
set hive.materializedview.rewriting.sql=false; SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP();
Victor 1702302786 Alex 1702302788 Bob 1702302791
set hive.materializedview.rewriting.sql=true; SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP();
Victor 1702302786 Alex 1702302788
Naturally the second query should return more rows than the first one since UNIX_TIMESTAMP is constantly growing. However, when view based rewritting is in use the second query will use the results from the materialized view which are by now obsolete (missing Bob entry).
Attachments
Attachments
Issue Links
- is caused by
-
HIVE-24274 Implement Query Text based MaterializedView rewrite
- Resolved
- is related to
-
HIVE-27291 Constant reduction in CBO does not work for UNIX_TIMESTAMP
- Open
- links to