Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-2031

Perftest: Unnecessary compression of incompressible blocks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • SystemML 1.0.0
    • None
    • None

    Description

      By default, we apply compression for data sets that are known to exceed aggregate memory if all operations of the given script that touch the respective input are supported over compressed matrices.

      On the perftest 800GB dense scenario, this leads to a slight slowdown and increase in the matrix size due to incompressible data, where each block is represented as follows:

      --col groups sizes (OLE,RLE,DDC1,DDC2,UC): 0,0,0,0,1000
      --compression ratio: 0.999475777837746
      

      We should investigate the set of incompressible columns as well as final representation and simply return the uncompressed block in such such scenarios.

      Attachments

        Issue Links

          Activity

            People

              mboehm7 Matthias Boehm
              mboehm7 Matthias Boehm
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: