Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-2808

Active deletion of 'deleted' Lucene index files from DataStore without relying on full scale Blob GC

    XMLWordPrintableJSON

Details

    Description

      With storing of Lucene index files within DataStore our usage pattern
      of DataStore has changed between JR2 and Oak.

      With JR2 the writes were mostly application based i.e. if application
      stores a pdf/image file then that would be stored in DataStore. JR2 by
      default would not write stuff to DataStore. Further in deployment
      where large number of binary content is present then systems tend to
      share the DataStore to avoid duplication of storage. In such cases
      running Blob GC is a non trivial task as it involves a manual step and
      coordination across multiple deployments. Due to this systems tend to
      delay frequency of GC

      Now with Oak apart from application the Oak system itself actively
      uses the DataStore to store the index files for Lucene and there the
      churn might be much higher i.e. frequency of creation and deletion of
      index file is lot higher. This would accelerate the rate of garbage
      generation and thus put lot more pressure on the DataStore storage
      requirements.

      Discussion thread http://markmail.org/thread/iybd3eq2bh372zrl

      Attachments

        1. OAK-2808-1.patch
          10 kB
          Thomas Mueller
        2. copyonread-stats.png
          27 kB
          Chetan Mehrotra

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. OAK-6314 ActiveDeletedBlobCollectorTest.multiThreadedCommits is failing intermittently for a few users

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OAK-6503 Active deletion of blobs tries to delete sub-16k inlined blobs in case of seg-tar

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OAK-6504 Active deletion of blobs needs to indicate information about purged blobs to mark-sweep collector

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. OAK-6950 Active deletion can delete blobs from a shared datastore when a clone setup is created

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. OAK-3140 DataStore / BlobStore: add a method to pass a "type" when writing

          • Major - Major loss of function.
          • Open

          Technical task - A technical task. OAK-4200 [BlobGC] Improve collection times of blobs available

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-5192 Reduce Lucene related growth of repository size

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-6269 Support non chunk storage in OakDirectory

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. OAK-6337 Decide major version bump of o.a.j.o.api.jmx

          • Major - Major loss of function.
          • Closed
          is required by

          Task - A task that needs to be done. OAK-6334 ActiveDeletedBlobCollectorFactory has lots of inner classes. Refactor it to be more modular and testable

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. OAK-3536 Indexing with Lucene and copy-on-read generate too much garbage in the BlobStore

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. OAK-6862 Active deletion of Lucene binaries: JMX bean, and ability to disable automatic

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-7968 Active deletion of Lucene binaries: configuration option to disable

          • Major - Major loss of function.
          • Closed

          Activity

            People

              catholicon Vikas Saurabh
              chetanm Chetan Mehrotra
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: