Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7526

Improvements to UnifiedHighlighter OffsetStrategies

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 6.4
    • modules/highlighter
    • None
    • New

    Description

      This ticket improves several of the UnifiedHighlighter FieldOffsetStrategies by reducing reliance on creating or re-creating TokenStreams.

      The primary changes are as follows:

      • AnalysisOffsetStrategy - split into two offset strategies
        • MemoryIndexOffsetStrategy - the primary analysis mode that utilizes a MemoryIndex for producing Offsets
        • TokenStreamOffsetStrategy - an offset strategy that avoids creating a MemoryIndex. Can only be used if the query distills down to terms and automata.
      • TokenStream removal
        • MemoryIndexOffsetStrategy - previously a TokenStream was created to fill the memory index and then once consumed a new one was generated by uninverting the MemoryIndex back into a TokenStream if there were automata (wildcard/mtq queries) involved. Now this is avoided, which should save memory and avoid a second pass over the data.
        • TermVectorOffsetStrategy - this was refactored in a similar way to avoid generating a TokenStream if automata are involved.
        • PostingsWithTermVectorsOffsetStrategy - similar refactoring
      • CompositePostingsEnum - aggregates several underlying PostingsEnums for wildcard/mtq queries. This should improve relevancy by providing unified metrics for a wildcard across all it's term matches
      • Added a HighlightFlag for enabling the newly separated TokenStreamOffsetStrategy since it can adversely affect passage relevancy

      Attachments

        1. LUCENE-7526.patch
          94 kB
          David Smiley

        Issue Links

          Activity

            People

              dsmiley David Smiley
              Timothy055 Timothy M. Rodriguez
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: