Uploaded image for project: 'REEF (Retired)'
  1. REEF (Retired)
  2. REEF-345 Complete implementation for YARN AM HA
  3. REEF-1660

Don't close streams in DFSEvaluatorLogOverwriteReaderWriter before flushing them

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 0.16
    • REEF-Runtime-YARN
    • None

    Description

      Currently DFSEvaluatorLogOverwriteReaderWriter has the following code in log writing:

      IOUtils.copyBytes(inputStream, outputStream, 4096, true);
      ...
      outputStream.hsync();
      if (inputStream != null) {
        inputStream.close();
      }
      

      The 4th parameter in copyBytes is boolean close, which directs IOUtils to close streams after copying is done. So when the code gets to hsync(), it attempts to sync closed stream. This fails on some systems. If we set close to false, closing will be done together with flushing, and writing succeeds.

      Attachments

        Activity

          People

            MariiaMykhailova Mariia Mykhailova
            MariiaMykhailova Mariia Mykhailova
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: