Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-1274

Unnecessary rdd computation for nnz maintenance on write

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Done
    • None
    • SystemML 0.13
    • Runtime
    • None

    Description

      Our primitive for writing binary block RDDs to HDFS (as used in guarded collect), first computes the number of non-zeros (nnz) and subsequently writes out the data. This leads to redundant RDD computation, which can be expensive for large DAGs of RDD operations. Explicitly computing the nnz is unnecessary as we could simply piggyback this computation onto the write via an accumulator as done in multiple other places in SystemML.

      Attachments

        Activity

          There are no comments yet on this issue.

          People

            mboehm7 Matthias Boehm
            mboehm7 Matthias Boehm
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: