Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.9.0
-
None
Description
Description:
When zeppelin is running under k8s mode, it will create the interpreter pod through "k8s/interpreter/100-interpreter-spec.yaml". And if user wants to use the spark interpreter in a k8s cluster with RBAC enabled, the driver pod (i.e., the interpreter pod ) should have the appropriate permissions to list, create, edit and delete pods in the cluster.
So in the "k8s/interpreter/100-interpreter-spec.yaml", it defines a role and a rolebinding for the pod to obtain the above permissions.
--- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: {{zeppelin.k8s.interpreter.pod.name}} namespace: {{zeppelin.k8s.namespace}} {% if zeppelin.k8s.server.uid is defined %} ownerReferences: - apiVersion: v1 controller: false blockOwnerDeletion: false kind: Pod name: {{zeppelin.k8s.server.pod.name}} uid: {{zeppelin.k8s.server.uid}} {% endif %} rules: - apiGroups: [""] resources: ["pods", "services"] verbs: ["create", "get", "update", "list", "delete", "watch" ]
However, when I tried to use the spark interpreter under k8s mode, I got the following error (The ip and port of my k8s cluster are replaced. For the complete log, please refer to the attachment log.txt):
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://<k8s-ip>:<k8s-port>/api/v1/namespaces/default/configmaps. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps is forbidden: User "system:serviceaccount:default:default" cannot create resource "configmaps" in API group "" in the namespace "default".
This error seems to be that the service account used cannot correctly create the resource configmap.
In the above defination of the Role in "k8s/interpreter/100-interpreter-spec.yaml", we can see that it only defines the permissions on resource: ["pods", "services"].
Possible solution:
When I add "configmaps" in the resource defination of Role in "k8s/interpreter/100-interpreter-spec.yaml":
resources: ["pods", "services", "configmaps"]
the spark interpreter can be started correctly.
Reference:
In the official spark documentation (https://spark.apache.org/docs/latest/running-on-kubernetes.html), there is such a description: The service account credentials used by the driver pods must be allowed to create pods, services and configmaps.