Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-35341

Retraction stop working with Clock dependent function in Filter

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.17.1
    • None
    • Table SQL / Runtime
    • None

    Description

       

      Say we have a Flink SQL view where

      1. we use clock dependent function like `UNIX_TIMESTAMP()` in query filter, eg. WHERE clause, eg. table.timestamp < UNIX_TIMESTAMP()
      2. source record is retracted at a time where the filter is evaluated as false

      we expect a retraction is produced from the view, but in practice nothing happen.

       

      We are using kafka as a source, here's a small snippet that shows the problem.

       

      CREATE TEMPORARY VIEW my_view AS
          SELECT key,
                  someData,
                  expiry
          FROM upstream 
          WHERE expiry > UNIX_TIMESTAMP();
      
      
      select * from my_view where key = 5574332;

       

       

      The actual query is a bit more complicated but this simplified one should illustrate the issue. Below is the event happen in chronological order:

       

      1. Run this query as a stream
      2. Create a record in upstream where key = 5574332, and expiry to be in 3 minutes into the future.
      3. Observe insertion of the record, as expected
      4. Wait for 3 minutes
      5. Now the record should expired, but given there's no update, there's no change to the stream output just yet
      6. Delete the upstream record (using tombstone in kafka)
      7. Observe no change in the stream output, but we are expecting retraction(aka deletion)

       

      Is this a known issue? I've search Jira but could find any, I observed this in 1.15 until 1.17, havent tested with 1.18 and above though

      Attachments

        Activity

          People

            Unassigned Unassigned
            qingwei91 Lim Qing Wei
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: