Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-5447

Spark driver pod cannot create resource "configmaps" under k8s mode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.9.0
    • 0.9.1, 0.10.0
    • Kubernetes, spark
    • None

    Description

      Description:

      When zeppelin is running under k8s mode, it will create the interpreter pod through "k8s/interpreter/100-interpreter-spec.yaml". And if user wants to use the spark interpreter in a k8s cluster with RBAC enabled, the driver pod (i.e., the interpreter pod ) should have the appropriate permissions to list, create, edit and delete pods in the cluster.

       

      So in the "k8s/interpreter/100-interpreter-spec.yaml", it defines a role and a rolebinding for the pod to obtain the above permissions. 

       

      ---
      kind: Role
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: {{zeppelin.k8s.interpreter.pod.name}}
        namespace: {{zeppelin.k8s.namespace}}
        {% if zeppelin.k8s.server.uid is defined %}
        ownerReferences:
        - apiVersion: v1
          controller: false
          blockOwnerDeletion: false
          kind: Pod
          name: {{zeppelin.k8s.server.pod.name}}
          uid: {{zeppelin.k8s.server.uid}}
        {% endif %}
      rules:
      - apiGroups: [""]
        resources: ["pods", "services"]
        verbs: ["create", "get", "update", "list", "delete", "watch" ]
      

       

      However, when I tried to use the spark interpreter under k8s mode, I got the following error (The ip and port of my k8s cluster are replaced. For the complete log, please refer to the attachment log.txt):

       

      Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://<k8s-ip>:<k8s-port>/api/v1/namespaces/default/configmaps. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps is forbidden: User "system:serviceaccount:default:default" cannot create resource "configmaps" in API group "" in the namespace "default".
      

       

       

      This error seems to be that the service account used cannot correctly create the resource configmap.

      In the above defination of the Role in "k8s/interpreter/100-interpreter-spec.yaml", we can see that it only defines the permissions on resource: ["pods", "services"]

       

       

      Possible solution:

      When I add "configmaps" in the resource defination of Role in "k8s/interpreter/100-interpreter-spec.yaml": 

      resources: ["pods", "services", "configmaps"]
      

      the spark interpreter can be started correctly. 

       

      Reference:

      In the official spark documentation (https://spark.apache.org/docs/latest/running-on-kubernetes.html), there is such a description: The service account credentials used by the driver pods must be allowed to create pods, services and configmaps.

       

      Attachments

        1. log.txt
          4 kB
          rickcheng

        Activity

          People

            rickcheng rickcheng
            rickcheng rickcheng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m