Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-26049

The tolerable-failed-checkpoints logic is invalid when checkpoint trigger failed

    XMLWordPrintableJSON

Details

    Description

      After triggerCheckpoint, if checkpoint failed, flink will execute the tolerable-failed-checkpoints logic. But if triggerCheckpoint failed, flink won't execute the tolerable-failed-checkpoints logic.

      How to reproduce this issue?

      In our online env, hdfs sre deletes the flink base dir by mistake, and flink job don't have permission to create checkpoint dir. So cause flink trigger checkpoint failed.

      There are some didn't meet expectations:

      • JM just log "Failed to trigger checkpoint for job 6f09d4a15dad42b24d52c987f5471f18 since Trigger checkpoint failure" . Don't show the root cause or exception.
      • user set tolerable-failed-checkpoints=0, but if triggerCheckpoint failed, flink won't execute the tolerable-failed-checkpoints logic. 
      • When triggerCheckpoint failed, numberOfFailedCheckpoints is always 0
      • When triggerCheckpoint failed, we can't find checkpoint info in checkpoint history page.

       

       

       

      All metrics are normal, so the next day we found out that the checkpoint failed, and the checkpoint has been failing for a day. it's not acceptable to the flink user.

      I have some ideas:

      1. Should tolerable-failed-checkpoints logic be executed when triggerCheckpoint fails?
      2. When triggerCheckpoint failed, should increase numberOfFailedCheckpoints?
      3. When triggerCheckpoint failed, should show checkpoint info in checkpoint history page?
      4. JM just show "Failed to trigger checkpoint", should we show detailed exception to easy find the root cause?

       

      Masters, could we do these changes? Please correct me if I'm wrong.

      Attachments

        1. image-2022-02-09-18-08-17-868.png
          751 kB
          Rui Fan
        2. image-2022-02-09-18-08-34-992.png
          70 kB
          Rui Fan
        3. image-2022-02-09-18-08-42-920.png
          80 kB
          Rui Fan
        4. image-2022-02-18-11-28-53-337.png
          670 kB
          Rui Fan
        5. image-2022-02-18-11-33-28-232.png
          261 kB
          Rui Fan
        6. image-2022-02-18-11-44-52-745.png
          148 kB
          Rui Fan
        7. image-2022-02-22-10-27-43-731.png
          582 kB
          Rui Fan
        8. image-2022-02-22-10-31-05-012.png
          27 kB
          Rui Fan

        Issue Links

          Activity

            People

              fanrui Rui Fan
              fanrui Rui Fan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: