Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-10521

ETag field should not be returned during GetObject if the key does not contain ETag field

    XMLWordPrintableJSON

Details

    Description

      A user encountered this error when it tries to download a key using the AWS S3 SDK (version 1.11.415).

      ERROR FileOperationServiceImpl - s3Client.getObject invoke error, objectKey:<redacted>. (FileOperationServiceImpl.java:438)
      java.lang.ArrayIndexOutOfBoundsException: 110
          at com.amazonaws.util.Base16Codec.pos(Base16Codec.java:96)
          at com.amazonaws.util.Base16Codec.decode(Base16Codec.java:87)
          at com.amazonaws.util.Base16Lower.decode(Base16Lower.java:53)
          at com.amazonaws.util.BinaryUtils.fromHex(BinaryUtils.java:48)
          at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1456)
          ...

      Although the key was able to be downloaded using AWS s3api

      aws s3api --endpoint <redacted> get-object --bucket <redacted> --key <redacted> download.txt
      {
          "AcceptRanges": "bytes",
          "LastModified": "Wed, 28 Feb 2024 11:06:30 GMT",
          "ContentLength": 1117055,
          "ETag": "\"null\"",
          "CacheControl": "no-cache",
          "ContentType": "application/octet-stream",
          "Expires": "Wed, 13 Mar 2024 08:10:05 GMT",
          "Metadata": {}
      } 

      The problem can be replicated even with the current version by uploading a file with ofs and downloading it using AWS S3 SDK.

      It seems object without ETag field was able to be downloaded using AWS CLI, but not AWS SDK. 

      After looking at the AWS SDK code it seems that AWS SDK will do a post-processing step that will validate the ETag field of the downloaded object to the object's content. If the ETag field is null, the post-processing step will skip the validation.

      Currently, S3G returns a string "null" for the ETag field if the ETag field does not exist, which should cause the AWS SDK to not be able to parse the string since it md5 string is longer than the "null" string. This is most probably why there is an ArrayIndexOutOfBoundsException

      One possible solution is to not return the ETag field at all if the key does not contain ETag to begin with. This way the post processing step in the AWS SDK will not validate the md5 hash.

      Future improvement: We might need to add some test coverage for AWS SDK (maybe in the integration tests) since the behavior might be different than AWS CLI.

      Attachments

        Issue Links

          Activity

            People

              ivanandika Ivan Andika
              ivanandika Ivan Andika
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: