Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
In execution modes spark and hybrid_spark, the parfor optimizer uses a smaller memory budget (50% of normal memory budget) for deciding on the local parfor degree of parallelism in order to avoid unnecessary "guarded collect" over hdfs. However, "guarded collect" can only happen in case of existing spark instructions; otherwise this decision leads to unnecessary under-utilization if the memory requirement is limiting the degree of parallelism.
We should determine, if there are existing spark instructions in the body and accordingly adapt the memory budget the parfor optimizer works with. On this MSVM usecase, the performance differences are as follows:
#before the change MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 144 MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 156 #after the change MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 99 MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 117
Attachments
Issue Links
- Is contained by
-
SYSTEMDS-1010 Perftest 0.11 release and related improvements
- Resolved