Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-11261

Get key answer with "There are insufficient datanodes to read the EC block" even nodes amount is sufficient

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • 2.0.0
    • None

    Description

      How to reproduce:

      1. Create cluster with 9 datanodes
      2. On datanode create volume (ozone sh volume create /data)
      3. Create bucket with EC replication rs-6-3-1024k (ozone sh bucket create data/test-bucket --type EC --replication rs-6-3-1024k)
      4. Create file e.x. 50 MB (fallocate -l 10M small_file_1)
      5. Put file to bucket (ozone sh key put data/test-bucket/small_file_1 small_file_1 --type EC --replication rs-6-3-1024k)
      6. Disable 4 nodes
      7. Try to get file from bucket (ozone sh key get /data/test-bucket/small_file_1 /tmp/sm_1_1)
      8. You will get "There are insufficient datanodes to read the EC block". It's ok, nodes amount should be at least 6.
      9. Enable 1 node and as fast as possible try to get file.
      10. You will get "There are insufficient datanodes to read the EC block". It is not ok, nodes now 6.
      11. You can try get file from minute later and get this error again.

      I reproduced it via docker-compose. With fixed nodes ip addresses (it's important, because docker compose can change ip addresses if not fix).

      Why it happened? Command getKey in Ozone Manager has cache. And this cache in this case is not actual. When we try to get file again and again OM return for us list of 5 nodes, instead of 6.

      I solved it by recreate blockReader with blockReader from refreshFunction.

      Attachments

        Issue Links

          Activity

            People

              a.juncevich Alex Juncevich
              a.juncevich Alex Juncevich
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: