Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
kubernetes-operator-1.5.0, kubernetes-operator-1.6.0, kubernetes-operator-1.7.0
-
None
-
None
Description
Problem:
By reading the source code of the sessionjob's first reconcilition in the session mode of the flink kubernetes operator, a clear single point of bottleneck can be identified. When submitting a session job, the operator needs to first download the job jar from the jarURL to the local storage of kubernetes pod , then upload the jar to the job manager through the `/jars/upload` rest api, and finally call the `/jars/:jarid/run` rest api to launch the job.
In this process, the operator needs to first download the jar and then upload the jar. When multiple jobs are submitted to the session cluster simultaneously, the operator can become a single point of bottleneck, which may be limited by the network traffic or other resource constraints of the operator pod.
Solution:
We can modify the job submission process in the session mode. The jobmanager can provide a `/jars/run` rest api that supports self-downloading the job jar, and the operator only needs to send a rest request to submit the job, without download and upload the job jar. In this way, the submission pressure of the operator can be distributed to each job manager.