Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7113

OfflineSorter and BKD should verify checksums in their temp files

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 6.0, 6.1, 7.0
    • None
    • None
    • New

    Description

      I am trying to index all 3.2 B points from the latest OpenStreetMaps export.

      My SSDs were not up to this, so I added a spinning magnets disk to beast2.

      But then I was hitting scary bug-like exceptions (ArrayIndexOutOfBoundsException) when indexing the first 2B points, and I finally checked dmesg and saw that my hard drive is dying.

      I think it's important that our temp file usages also validate checksums (like we do for all our index files, either at reader open or at merge or CheckIndex), so we can hopefully easily differentiate a bit-flipping IO system from a possible Lucene bug, in the future.

      Attachments

        1. LUCENE-7113.patch
          53 kB
          Michael McCandless
        2. LUCENE-7113.patch
          53 kB
          Michael McCandless

        Activity

          People

            mikemccand Michael McCandless
            mikemccand Michael McCandless
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: