Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.17.1
-
None
-
None
Description
Say we have a Flink SQL view where
- we use clock dependent function like `UNIX_TIMESTAMP()` in query filter, eg. WHERE clause, eg. table.timestamp < UNIX_TIMESTAMP()
- source record is retracted at a time where the filter is evaluated as false
we expect a retraction is produced from the view, but in practice nothing happen.
We are using kafka as a source, here's a small snippet that shows the problem.
CREATE TEMPORARY VIEW my_view AS SELECT key, someData, expiry FROM upstream WHERE expiry > UNIX_TIMESTAMP(); select * from my_view where key = 5574332;
The actual query is a bit more complicated but this simplified one should illustrate the issue. Below is the event happen in chronological order:
- Run this query as a stream
- Create a record in upstream where key = 5574332, and expiry to be in 3 minutes into the future.
- Observe insertion of the record, as expected
- Wait for 3 minutes
- Now the record should expired, but given there's no update, there's no change to the stream output just yet
- Delete the upstream record (using tombstone in kafka)
- Observe no change in the stream output, but we are expecting retraction(aka deletion)
Is this a known issue? I've search Jira but could find any, I observed this in 1.15 until 1.17, havent tested with 1.18 and above though