Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33137 FLIP-312: Prometheus Sink Connector
  3. FLINK-36404

PrometheusSinkWriteException thrown by the response callback may not cause job to fail

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      Issue
      PrometheusSinkWriteException thrown by HttpResponseCallback do not cause the httpclient IOReactor to fail, being actually swallowed, and preventing the job from failing.
      Also, related: exceptions from the IOReactor eventually causes the response callback failed to be called. Allowing the user to set DISCARD_AND_CONTINUE on generic exceptions thrown by the client may hide rethrown exceptions. Also, there is really no use of not failing on a generic unhandled exceptions from the client.

      Solution
      1. Intercept PrometheusSinkWriteException up the httpclient stack, adding to the client a IOSessionListener to that can rethow those exceptions, causing the reactor to actually fail, and consequently also the operator to fail.
      2. Remove the ability to configure of error handling behaviour on generic exceptions thrown by the httpclient. The job should always fail.
      3. When the httpclient IOReactor fail, a long chain of exceptions is logged. To keep the actual root cause evident, the response callback should log to ERROR when the exception happens

      Attachments

        Activity

          People

            Unassigned Unassigned
            nicusX Lorenzo Nicora
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: