Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-550

Collocate rfile index entries within file

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.4.1, 1.5.0
    • tserver
    • None

    Description

      Before multi-level indexes were introduced, when an an rfile was written its entire index was held in memory and written out then the file was closed. With the introduction of multilevel index each index block is written when it fills up as the file is being written. This was done to handle the case where the index may not fit into memory. This leads to index blocks being sprinkled through the file. So any operation that iterates over the entire index can be slow because it turns into a lot of random accesses.

      One possible solution is to buffer lots of index blocks up to some some threshold and write out alot of index blocks at once. This would make a scan of the index much faster as it would turn into a set of sequential reads of large chunks of data.

      Could buffer all block at a particular level and write them out when the parent index block fills up.

      Attachments

        Activity

          People

            kturner Keith Turner
            kturner Keith Turner
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: