Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
ITestS3ACommitterMRJob fails on FSO bucket.
S3 Gateway audit log:
2024-03-22 18:05:34,785 | INFO | OMAudit | user=s3acontract | ip=192.168.192.3 | op=CREATE_DIRECTORY {volume=s3v, bucket=fso-bucket, key=test/ITestS3ACommitterMRJob-execute-directory/, dataSize=0} | ret=SUCCESS |
2024-03-22 18:05:40,117 | INFO | OMAudit | user=s3acontract | ip=192.168.192.3 | op=DELETE_KEY {volume=s3v, bucket=fso-bucket, key=test/ITestS3ACommitterMRJob-execute-directory, dataSize=0} | ret=SUCCESS |
2024-03-22 18:05:40,160 | ERROR | OMAudit | user=s3acontract | ip=192.168.192.3 | op=COMPLETE_MULTIPART_UPLOAD {volume=s3v, bucket=fso-bucket, key=test/ITestS3ACommitterMRJob-execute-directory/part-m-00000, dataSize=0, multipartList=[partNumber: 1 partName: "f03627c876605452a26379320675064e" eTag: "f03627c876605452a26379320675064e" ]} | ret=FAILURE | DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to find parent directory of test/ITestS3ACommitterMRJob-execute-directory/part-m-00000
S3A uses multipart upload to commit job results in a batch. The staging committer's replace mode deletes the target directory before completing MPU. The problem is that FSO creates intermediate dirs during "initiate MPU" request. Thus the "complete MPU" request finds directory to be non-existent.
Attachments
Issue Links
- links to