Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5959

Add Cloud KMS support to GCS copies

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.11.0
    • io-java-gcp, sdk-py-core
    • None

    Description

      Beam SDK currently uses the CopyTo GCS API call, which doesn't support copying objects that Customer Managed Encryption Keys (CMEK).
      CMEKs are managed in Cloud KMS.

      Items (for Java and Python SDKs):

      • Update clients to versions that support KMS keys.
      • Change copyTo API calls to use rewriteTo (Python - directly, Java - possibly convert copyTo API call to use client library)
      • Add unit tests.
      • Add basic tests (DirectRunner and GCS buckets with CMEK).

      Attachments

        Activity

          People

            udim Udi Meiri
            udim Udi Meiri
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 31.5h
                31.5h