Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
Description
How to reproduce:
- Create cluster with 9 datanodes
- On datanode create volume (ozone sh volume create /data)
- Create bucket with EC replication rs-6-3-1024k (ozone sh bucket create data/test-bucket --type EC --replication rs-6-3-1024k)
- Create file e.x. 50 MB (fallocate -l 10M small_file_1)
- Put file to bucket (ozone sh key put data/test-bucket/small_file_1 small_file_1 --type EC --replication rs-6-3-1024k)
- Disable 4 nodes
- Try to get file from bucket (ozone sh key get /data/test-bucket/small_file_1 /tmp/sm_1_1)
- You will get "There are insufficient datanodes to read the EC block". It's ok, nodes amount should be at least 6.
- Enable 1 node and as fast as possible try to get file.
- You will get "There are insufficient datanodes to read the EC block". It is not ok, nodes now 6.
- You can try get file from minute later and get this error again.
I reproduced it via docker-compose. With fixed nodes ip addresses (it's important, because docker compose can change ip addresses if not fix).
Why it happened? Command getKey in Ozone Manager has cache. And this cache in this case is not actual. When we try to get file again and again OM return for us list of 5 nodes, instead of 6.
I solved it by recreate blockReader with blockReader from refreshFunction.
Attachments
Issue Links
- duplicates
-
HDDS-11209 Avoid insufficient EC pipelines in the container pipeline cache
- Resolved
- is required by
-
HDDS-7909 When DN is offline Read of EC data is failing [Failed to execute command GetBlock on the Pipeline]
- Open
- links to