Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
SystemDS 3.1
-
None
-
None
Description
Running with 20GB driver, the MLogreg 1M_1K_dense scenario (8GB matrix) spawns unnecessary Spark because because of the "recently" added isNaN call that potentially creates an output matrix as large as the input.
GENERIC (lines 50-62) [recompile=true] --------CP createvar _mVar16 scratch_space//_p626960_130.149.237.11//_t0/temp0 true MATRIX binary 1000000 1000 1000 -1 copy --------SPARK isnan X.MATRIX.FP64 _mVar16.MATRIX.FP64 --------CP createvar _mVar17 scratch_space//_p626960_130.149.237.11//_t0/temp1 true MATRIX binary 1 1 1000 -1 copy --------SPARK uak+ _mVar16.MATRIX.FP64 _mVar17.MATRIX.FP64 SINGLE_BLOCK --------CP rmvar _mVar16 --------CP castdts _mVar17.MATRIX.FP64.false _Var18.SCALAR.FP64 1 --------CP rmvar _mVar17 --------CP mvvar _Var18 numNaNs
resulting in below statistics, while a pure singlenode run would only take ~14s
Total elapsed time: 46.631 sec. Total compilation time: 1.309 sec. Total execution time: 45.322 sec. Number of compiled Spark inst: 89. Number of executed Spark inst: 3. Cache hits (Mem/Li/WB/FS/HDFS): 2023/0/0/0/2. Cache writes (Li/WB/FS/HDFS): 1/249/0/1. Cache times (ACQr/m, RLS, EXP): 3.787/0.003/0.022/0.441 sec. HOP DAGs recompiled (PRED, SB): 0/168. HOP DAGs recompile time: 0.263 sec. Functions recompiled: 1. Functions recompile time: 0.069 sec. Spark ctx create time (lazy): 25.694 sec. Spark trans counts (par,bc,col):0/0/0. Spark trans times (par,bc,col): 0.000/0.000/0.000 secs. Spark async. count (pf,bc,op): 0/0/0. Total JIT compile time: 26.818 sec. Total JVM GC count: 5. Total JVM GC time: 0.123 sec. Heavy hitter instructions: # Instruction Time(s) Count 1 sp_chkpoint 26.322 18 2 m_multiLogReg 18.556 1 3 sp_uak+ 5.346 1 4 uarsqk+ 4.038 1 5 mmchain 3.966 49 6 ba+* 2.263 146 7 - 0.443 164 8 write 0.441 1 9 uark+ 0.277 16 10 exp 0.272 16