Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-7731 Phase III: Erasure Coding followup tasks
  3. HDDS-7195

EC: Additional checksum validations to make sure data integrity to avoid reconstruction time corruptions.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      For EC blocks, each chunk has a checksum as for usual writes. The user data is written as a stripe across the container group, and each chunk has its own standalone checksum.

      After calculating parity and writing the parity bytes, we know the checksums for all chunks in the stripe, including parity. This is the stripe checksum, and it is a simple concatenation of all the checksum bytes for each chunk in the stripe in replicaIndex order. This stripe checksum is written to a redundant number of replicas in the container group - namely replica index = 1 and all parity replicas. In the case of EC-6-3, this means 4 out of the 9 replicas hold the stripe checksum. For a container to be recovered at least 6 replicas must be available, and hence the stripe checksum must be available if the container is recoverable.

      To ensure the integrity of each recovered chunk in a stripe, needs strict validation by using checksums.

      Attachments

        Activity

          People

            nanda Nandakumar
            umamaheswararao Uma Maheswara Rao G
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: