Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-3179

Add snapshot compression to reduce the disk IO

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.6.0
    • None

    Description

      When the snapshot becomes larger, the periodically snapshot after certain number of txns will be more expensive. Which will in turn affect the maximum throughput we can support within SLA, because of the disk contention between snapshot and txn when they're on the same drive.
       
      With compression like zstd/snappy/gzip, the actual snapshot size could be much smaller, the compress ratio depends on the actual data. It might make the recovery time (loading from disk) faster in some cases, but will take longer sometimes because of the extra time used to compress/decompress.
       
      Based on the production traffic, the performance various with different compress method as well, that's why we provided different implementations, we can select different compress method for different use cases.
       

      Attachments

        Issue Links

          Activity

            People

              yisong-yue Yisong Yue
              lvfangmin Fangmin Lv
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 40m
                  4h 40m