Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-30998

Add optional exception handler to flink-connector-opensearch

    XMLWordPrintableJSON

Details

    Description

      Currently, when there is a failure coming from Opensearch, the FlinkRuntimeException is thrown from OpensearchWriter.java code (line 346). This makes the Flink pipeline fail. There is no way to handle the exception in the client code.

      I suggest to add an option to set a failure handler, similar to the way it is done in elasticsearch connector. This way the client code has a chance to examine the failure and handle it.

      Here is the use case example when it will be very useful. We are using streams on Opensearch side, and we are setting our own document IDs. Sometimes these IDs are duplicated; we need to ignore this situation and continue (this way it works for us with Elastisearch).
      However, with opensearch connector, the error comes back, saying that the batch failed (even though most of the documents were indexed, only the ones with duplicated IDs were rejected), and the whole flink job fails.

      Attachments

        Issue Links

          Activity

            People

              lilyevsky Leonid Ilyevsky
              lilyevsky Leonid Ilyevsky
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: