Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5038

Don't call MergePolicy / IndexWriter during DWPT Flush

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.3, 6.0
    • 4.4, 6.0
    • core/index
    • None
    • New, Patch Available

    Description

      We currently consult the indexwriter -> merge policy to decide if we need to write CFS or not which is bad in many ways.

      • we should call mergepolicy only during merges
      • we should never sync on IW during DWPT flush
      • we should be able to make the decision if we need to write CFS or not before flush, ie. we could write parts of the flush directly to CFS or even start writing stored fields directly.
      • in the NRT case it might make sense to write all flushes to CFS to minimize filedescriptors independent of the index size.

      I wonder if we can use a simple boolean for this in the IWC and get away with not consulting merge policy. This would simplify concurrency a lot here already.

      Attachments

        1. LUCENE-5038.patch
          13 kB
          Simon Willnauer
        2. LUCENE-5038.patch
          14 kB
          Simon Willnauer
        3. LUCENE-5038.patch
          16 kB
          Simon Willnauer
        4. LUCENE-5038.patch
          48 kB
          Simon Willnauer
        5. LUCENE-5038.patch
          70 kB
          Simon Willnauer
        6. LUCENE-5038.patch
          70 kB
          Simon Willnauer

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-4934 Prevent runtime failure if users use initargs "useCompoundFile" setting on LogMergePolicy or TieredMergePolicy

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SOLR-4941 useCompoundFile default has changed, simple config option no longer seems to work

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-4926 I am seeing RecoveryZkTest and ChaosMonkeySafeLeaderTest fail often on trunk.

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              simonw Simon Willnauer
              simonw Simon Willnauer
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: