OFS mkdir -p does not work as expected for bucket creation when volume exists due to volume create ACL check

We discovered this problem during the implementation of HttpFS Gateway. I did an acceptance test for the HttpFS with Robot framework. In the ozonesecure docker environment when I tried to make a volume with the testuser principal it didn't work, because it doesn't have permission to do it. So we decided to make a volume with an admin, set the testuser as the owner of it and then create buckets with the testuser. Even after the owner change happened successfully it gave the same error:
User testuser/httpfs@EXAMPLE.COM doesn't have CREATE permission to access volume vol01 null null
After debugging we found why this happened. As the bucket is not existing first it goes to the getBucket() method in the BasicRootedOzoneClientAdapterImpl class.

The createIfNotExist is true and both in the VOLUME_NOT_FOUND and BUCKET_NOT_FOUND cases tries to create volume first, which the testuser does not have permission. So we got the error from there, despite of the fact that testuser is the owner of that volume, so it should be able to create buckets inside.

We were able to recreate this in terminal in the scm container (in that because it has both testuser and testuser2 as principals).

bash-4.2$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: testuser/scm@EXAMPLE.COM

Valid starting     Expires            Service principal
10/18/21 11:23:39  10/19/21 11:23:39  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 10/25/21 11:23:39

In the scm testuser is an admin, testuser/scm@EXAMPLE.COM is added as an ozone administrator in the docker-config. I did the same with testuser/httpfs@EXAMPLE.COM but it is not an admin, as the username is mapped to short user principal name (with an auth-to-local rule), which is testuser. Because of this the equality check between testuser and testuser/httpfs@EXAMPLE.COM is false, so it is not taken as an admin user.

bash-4.2$ ozone sh volume update --user=testuser2 vol02
\{
  "metadata" : { },
  "name" : "vol02",
  "admin" : "testuser",
  "owner" : "testuser2",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "usedNamespace" : 0,
  "creationTime" : "2021-10-18T11:19:59.777Z",
  "modificationTime" : "2021-10-18T11:24:04.183Z",
  "acls" : [ \{
    "type" : "USER",
    "name" : "testuser",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  } ]
}
bash-4.2$ kinit -kt /opt/hadoop/compose/_keytabs/testuser2.keytab testuser2/scm@EXAMPLE.COM
bash-4.2$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: testuser2/scm@EXAMPLE.COM

Valid starting     Expires            Service principal
10/18/21 11:24:17  10/19/21 11:24:17  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 10/25/21 11:24:17
bash-4.2$ ozone fs -mkdir -p ofs://om/vol01/buck01
2021-10-18 11:24:47,729 [main] INFO rpc.RpcClient: Creating Volume: vol01, with testuser2 as owner and space quota set to -1 bytes, counts quota set to -1
mkdir: User testuser2/scm@EXAMPLE.COM doesn't have CREATE permission to access volume vol01 null null