Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-4

Implement security for Hadoop Distributed Storage Layer

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.4.0
    • Security
    • None
    • HDDS BadLands

    Description

      In HDFS-7240, we have created a scalable block layer that facilitates separation of namespace and block layer. Hadoop Distributed Storage Layer (HDSL) allows us to scale HDFS(HDFS-10419) and as well as create ozone (HDFS-13074).

      This JIRA is an umbrella JIRA that tracks the security-related work items for Hadoop Distributed Storage Layer.

      Attachments

        1. HadoopStorageLayerSecurity.pdf
          445 kB
          Anu Engineer

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-7240 Scaling HDFS

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. HDDS-7331 Ozone PKI improvements

          • Major - Major loss of function.
          • Open
          1.
          		 Enable OzoneManager kerberos auth Sub-task Resolved Ajay Kumar  
          2.
          		 Enable SCM kerberos auth Sub-task Resolved Ajay Kumar  
          3.
          		 Enable kerberos auth for Ozone client in hadoop rpc Sub-task Resolved Ajay Kumar  
          4.
          		 Fix config names for secure ksm and scm Sub-task Resolved Ajay Kumar  
          5.
          		 Add kdc docker image for secure ozone cluster Sub-task Resolved Ajay Kumar  
          6.
          		 Adding Ozone Manager Audit Log Sub-task Resolved Dinesh Chitlangia  
          7.
          		 Fix secure docker and configs Sub-task Resolved Xiaoyu Yao  
          8.
          		 Move OzoneSecure docker-compose after HDDS-447 Sub-task Resolved Xiaoyu Yao  
          9.
          		 Resolve bouncy castle dependency for hadoop-hdds-common Sub-task Resolved Ajay Kumar  
          10.
          		 SCM CA: generate public/private key pair for SCM/OM/DNs Sub-task Resolved Ajay Kumar  
          11.
          		 Create a Self-Signed Certificate Sub-task Resolved Anu Engineer  
          12.
          		 SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider Sub-task Resolved Xiaoyu Yao  
          13.
          		 Adding ASF license header to kadm5.acl Sub-task Resolved Ajay Kumar  
          14.
          		 Fix the Dependency convergence issue on HDDS-4 Sub-task Resolved Xiaoyu Yao  
          15.
          		 Fix HDDS-4 branch after HDDS-490 and HADOOP-15832 Sub-task Resolved Xiaoyu Yao  
          16.
          		 SCM CA: Add new security protocol for SCM to expose security related functions Sub-task Resolved Ajay Kumar  
          17.
          		 SCM CA: generate CSR for SCM CA clients Sub-task Resolved Xiaoyu Yao  
          18.
          		 Add asf license to TestCertificateSignRequest Sub-task Resolved Ajay Kumar  
          19.
          		 SCM security protocol server is not starting Sub-task Resolved Ajay Kumar  
          20.
          		 Fix ozone-secure.robot test Sub-task Resolved Ajay Kumar  
          21.
          		 Add an interface for CA and Clients for Certificate operations Sub-task Resolved Anu Engineer  
          22.
          		 Add TokenIdentifier Ozone for delegation token and block token Sub-task Resolved Ajay Kumar  
          23.
          		 SCM CA: Update DelegationKey to retrieve private/public key Sub-task Resolved Ajay Kumar  
          24.
          		 Bootstrap OM/SCM with private/public key pair Sub-task Resolved Ajay Kumar  
          25.
          		 Add OzoneManager Delegation Token support Sub-task Resolved Ajay Kumar  
          26.
          		 Add GRPC protocol interceptors for Ozone Block Token Sub-task Resolved Xiaoyu Yao  
          27.
          		 Fix TestSecureOzoneContainer NPE after HDDS-837 Sub-task Resolved Xiaoyu Yao  
          28.
          		 Bootstrap genesis SCM(CA) with self-signed certificate. Sub-task Resolved Anu Engineer  
          29.
          		 Block token: Add secret token manager Sub-task Resolved Ajay Kumar  
          30.
          		 Fix merge issue that causes NPE OzoneManager#httpServer Sub-task Resolved Xiaoyu Yao  
          31.
          		 GRPC: Support secure gRPC endpoint with mTLS Sub-task Resolved Xiaoyu Yao  
          32.
          		 Document ozone.max.key.len usage Sub-task Resolved Ajay Kumar  
          33.
          		 Remove ozone.max.key.len property Sub-task Resolved Ajay Kumar  
          34.
          		 Block token: Client api changes for block token Sub-task Resolved Ajay Kumar  
          35.
          		 Create an S3 Auth Table Sub-task Resolved Dinesh Chitlangia  
          36.
          		 SCM CA: SCM CA server signs certificate for approved CSR Sub-task Resolved Anu Engineer  
          37.
          		 SCM CA: Add CA to SCM. Sub-task Resolved Anu Engineer  
          38.
          		 Fix generics warnings in delegation token Sub-task Resolved Ajay Kumar  
          39.
          		 Add Client APIs for using S3 Auth interface Sub-task Resolved Dinesh Chitlangia  
          40.
          		 Fix failure in TestOzoneShell due to null check in SecurityConfig Sub-task Resolved Ajay Kumar  
          41.
          		 Fix test failure in TestOmMetrics Sub-task Resolved Ajay Kumar  
          42.
          		 Fix TestOzoneConfigrationFields Sub-task Resolved Anu Engineer  
          43.
          		 Fix classnotfound error for bouncy castle classes in OM,SCM init Sub-task Resolved Ajay Kumar  
          44.
          		 Fix failures in TestOzoneConfigurationFields Sub-task Resolved Ajay Kumar  
          45.
          		 Unblock certain SCM client APIs from SCM#checkAdminAccess Sub-task Resolved Xiaoyu Yao  
          46.
          		 Ratis: Support secure gRPC endpoint with mTLS for Ratis Sub-task Resolved Xiaoyu Yao  
          47.
          		 Add cli command option for getS3Secret Sub-task Resolved Dinesh Chitlangia  
          48.
          		 Fix TestOzoneManagerRatisServer.testIsReadOnlyCapturesAllCmdTypeEnums Sub-task Resolved Xiaoyu Yao  
          49.
          		 Add block token validation in HddsDispatcher/XceiverServer Sub-task Resolved Ajay Kumar  
          50.
          		 Manage ozone security tokens with ozone shell cli Sub-task Resolved Ajay Kumar  
          51.
          		 Adding getOMCertificate in SCMSecurityProtocol Sub-task Resolved Ajay Kumar  
          52.
          		 GRPC: DN changes to use cert issued by SCM for GRPC mTLS Sub-task Resolved Xiaoyu Yao  
          53.
          		 Add Default CertificateClient implementation Sub-task Resolved Ajay Kumar  
          54.
          		 Implement OM init in secure cluster Sub-task Resolved Ajay Kumar  
          55.
          		 Bootstrap DN with private/public key pair Sub-task Resolved Ajay Kumar  
          56.
          		 OzoneManager fails to connect with secure SCM Sub-task Resolved Ajay Kumar  
          57.
          		 Fix findbugs issues in DefaultCertificateClient#handleCase Sub-task Resolved Ajay Kumar  
          58.
          		 OzoneManager need to login during init when security is enabled. Sub-task Resolved Xiaoyu Yao  
          59.
          		 SCM CA: Write Certificate information to SCM Metadata Sub-task Resolved Anu Engineer  
          60.
          		 Add API to get OM certificate from SCM CA Sub-task Resolved Ajay Kumar  
          61.
          		 Support Service Level Authorization for Ozone Sub-task Resolved Xiaoyu Yao  
          62.
          		 Allow persisting X509CertImpl to SCM certificate table Sub-task Resolved Xiaoyu Yao  
          63.
          		 DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier Sub-task Resolved Ajay Kumar  
          64.
          		 Fix jdk 11 issue for ozonesecure base image and docker-compose Sub-task Resolved Xiaoyu Yao  
          65.
          		 Fix ClassNotFound issue with javax.xml.bind.DatatypeConverter used by DefaultProfile Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          66.
          		 SCM CA: OM sends CSR and uses certificate issued by SCM Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          67.
          		 Support getDelegationToken API for OzoneFileSystem Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          68.
          		 OzoneManager NPE reading private key file. Sub-task Resolved Xiaoyu Yao  
          69.
          		 Change name of ozoneManager service in docker compose files to om Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          70.
          		 SCM CA: DN sends CSR and uses certificate issued by SCM Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 5h 20m
          71.
          		 BaseHttpServer NPE is HTTP policy is HTTPS_ONLY Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          72.
          		 Add robot test for OM Block Token Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          73.
          		 Fix incorrect Ozone ClientProtocol KerberosInfo annotation Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          74.
          		 OM delegation expiration time should use Time.now instead of Time.monotonicNow Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          75.
          		 Fix checkstyle issue from Nightly run Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          76.
          		 Enable token based authentication for S3 api Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 29.5h
          77.
          		 Fix TestDefaultCertificateClient#testSignDataStream Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          78.
          		 Fix failure in TestOzoneManagerHttpServer & TestStorageContainerManagerHttpServer Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          79.
          		 Add robot test for OM Delegation Token Sub-task Resolved Ajay Kumar  
          80.
          		 Add ozone delegation token utility subcmd for Ozone CLI Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 40m
          81.
          		 Fix checkstyle issue from Nightly run Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          82.
          		 DN get OM certificate from SCM CA for block token validation Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10h 50m
          83.
          		 Refactor om token db as column family. Sub-task Resolved Anu Engineer  
          84.
          		 Enable service policy authorization for HDDS group Sub-task Resolved Ajay Kumar  
          85.
          		 OM get the certificate from SCM CA for token validation Sub-task Resolved Xiaoyu Yao  
          86.
          		 Change hadoop-runner and apache/hadoop base image to use Java8 Sub-task Resolved Marton Elek

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          87.
          		 Update MiniOzoneCluster to work with security protocol from SCM Sub-task Resolved Unassigned  
          88.
          		 Support TokenIssuer interface for running jobs with OzoneFileSystem Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 6.5h
          89.
          		 Set OmKeyArgs#refreshPipeline flag properly to avoid reading from stale pipeline Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          90.
          		 Refactor ozone acceptance test to allow run in secure mode Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 6h 40m
          91.
          		 Ozone serialization codec for Ozone S3 secret table Sub-task Resolved Zsolt Venczel  
          92.
          		 KeyOutputStream#write throws ArrayIndexOutOfBoundsException when running RandomWrite MR examples Sub-task Resolved Shashikant Banerjee

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          93.
          		 Fix MalformedTracerStateStringException on DN logs Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h
          94.
          		 Spark job fails to create ozone rpc client Sub-task Resolved Ajay Kumar  
          95.
          		 NPE if secure ozone if KMS uri is not defined. Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          96.
          		 ozone spark job failing with class not found error for hadoop 2 Sub-task Resolved Ajay Kumar  
          97.
          		 Inconsistent naming convention with Ozone Kerberos configuration Sub-task Resolved Xiaoyu Yao

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          98.
          		 Update ratis dependency to 0.3.0 Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          99.
          		 Add retry to kinit command in smoketests Sub-task Resolved Ajay Kumar

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h

          Activity

            People

              xyao Xiaoyu Yao
              aengineer Anu Engineer
              Votes:
              0 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 85h 40m
                  85h 40m