Uploaded image for project: 'Lucene.Net'
  1. Lucene.Net
  2. LUCENENET-640

Sequential IndexWriter performance in concurrent environments.

    XMLWordPrintableJSON

Details

    • Important

    Description

      When creating Lucene.Net indices in parallel, sequential-like performance is experienced. Profiling 8 concurrent IndexWriter instances writing in parallel shows that WeakIdentityMap::IdentityWeakReference::Equals spends most time garbage collecting (94.91%) and TokenStream::AssertFinal (87.09% garbage collecting) in my preliminary tests (see screenshots).

      The WeakIdentityMap implementation uses an IdentityWeakReference as key, which is implemented as a class. By inspection of this class, it is merely a System.Runtime.InteropServices.GCHandle wrapper as can be seen in the mono project, manually wrapping of this struct in a struct rather than a class - will eliminate some of the immense amounts of garbage collection.

      Attachments

        1. IdentityWeakReferenceBug.jpg
          18 kB
          Mathias Henriksen
        2. AssertFinalBug.jpg
          29 kB
          Mathias Henriksen
        3. overviewBug.jpg
          107 kB
          Mathias Henriksen
        4. Program.cs
          2 kB
          Mathias Henriksen

        Issue Links

          Is contained by

          Improvement - An improvement or enhancement to an existing feature or task. LUCENENET-630 Identify/Fix Bottlenecks

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. LUCENENET-616 Make Collections from Lucene.Net.Support into a 1st Class Feature

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LUCENENET-610 Reduce locking in FieldCacheImpl::Cache::Get

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. LUCENENET-636 Port ConditionalWeakTable from .NET Core 3.x to .NET Standard 2.0

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. LUCENENET-644 The design and implementation of a better ThreadLocal<T>

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. LUCENENET-642 Eliminate Runtime Reflection Code from TokenStream

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link GitHub Pull Request #236

          Activity

            People

              Unassigned Unassigned
              matt2843 Mathias Henriksen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m