Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-2155

Recompute nnz on spark checkpoints for large matrices

    XMLWordPrintableJSON

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 1.1
    • None
    • None

    Description

      Operations over large matrices with dimensions > max integer, are always forced to spark which implies missing nnz information due to lazy evaluation in spark. This task aims to proactively recompute the nnz for such large matrices (which are often ultra-sparse) to improve size propagation during recompilation and repeated access.

      Attachments

        Activity

          People

            mboehm7 Matthias Boehm
            mboehm7 Matthias Boehm
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: