Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2074

Use a separate JFlex generated Unicode 4 by Java 5 compatible StandardTokenizer

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0
    • 3.1, 4.0-ALPHA
    • None
    • None
    • New, Patch Available

    Description

      The current trunk version of StandardTokenizerImpl was generated by Java 1.4 (according to the warning). In Java 3.0 we switch to Java 1.5, so we should regenerate the file.

      After regeneration the Tokenizer behaves different for some characters. Because of that we should only use the new TokenizerImpl when Version.LUCENE_30 or LUCENE_31 is used as matchVersion.

      Attachments

        1. LUCENE-2074.patch
          40 kB
          Uwe Schindler
        2. jflexwarning.patch
          0.7 kB
          Mark Miller
        3. LUCENE-2074.patch
          40 kB
          Uwe Schindler
        4. LUCENE-2074-lucene30.patch
          3 kB
          Uwe Schindler
        5. LUCENE-2074.patch
          41 kB
          Uwe Schindler
        6. LUCENE-2074.patch
          40 kB
          Uwe Schindler
        7. jflex-1.4.1-vs-1.5-snapshot.diff
          5 kB
          Uwe Schindler
        8. LUCENE-2074.patch
          55 kB
          Uwe Schindler
        9. LUCENE-2074.patch
          65 kB
          Uwe Schindler
        10. LUCENE-2074.patch
          63 kB
          Uwe Schindler
        11. LUCENE-2074.patch
          65 kB
          Uwe Schindler
        12. LUCENE-2074.patch
          67 kB
          Uwe Schindler
        13. LUCENE-2074.patch
          69 kB
          Uwe Schindler

        Issue Links

          incorporates

          Sub-task - The sub-task of the issue LUCENE-2384 Reset zzBuffer in StandardTokenizerImpl* when lexer is reset.

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. LUCENE-2244 Improve StandardTokenizer's understanding of non ASCII punctuation and quotes

          • Major - Major loss of function.
          • Closed
          There are no Sub-Tasks for this issue.

          Activity

            People

              uschindler Uwe Schindler
              uschindler Uwe Schindler
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: