Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-1423

OOM on generating ultra-sparse rand data

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 0.14
    • None
    • None

    Description

      In order to ensure consistency across backends, we first determine the number of non-zeros per block and subsequently generate random data accordingly. However, in case of ultra-sparse data sets, this temporary array can be almost as large as the dataset. Since this memory consumption is unaccounted and even required for distributed operations, there are various possible scenarios where this would cause OOMs.

      This task aims to solve this issue for all backends, by determining the nnz per block in a streaming manner without materialization.

      Attachments

        Issue Links

          Activity

            People

              mboehm7 Matthias Boehm
              mboehm7 Matthias Boehm
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: