Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-1019

Perftest: MSVM, L sparse, unncessary under-utilization (parfor dop)

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 0.11
    • None
    • None

    Description

      In execution modes spark and hybrid_spark, the parfor optimizer uses a smaller memory budget (50% of normal memory budget) for deciding on the local parfor degree of parallelism in order to avoid unnecessary "guarded collect" over hdfs. However, "guarded collect" can only happen in case of existing spark instructions; otherwise this decision leads to unnecessary under-utilization if the memory requirement is limiting the degree of parallelism.

      We should determine, if there are existing spark instructions in the body and accordingly adapt the memory budget the parfor optimizer works with. On this MSVM usecase, the performance differences are as follows:

      #before the change
      MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 144
      MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 156
      #after the change
      MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 99
      MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 117
      

      Attachments

        Issue Links

          Activity

            There are no comments yet on this issue.

            People

              mboehm7 Matthias Boehm
              mboehm7 Matthias Boehm
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: