Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16201

Reduce amount of allocations during batch statement execution

    XMLWordPrintableJSON

Details

    Description

      In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, we see 4.0b2 going OOM from time to time. According to a heap dump, we have multiple NTR threads in a 3-digit MB range.

      This is likely related to object array pre-allocations at the size of BatchUpdatesCollector.updatedRows per BTree although there is always only 1 BTreeRow in the BTree.

      So it seems we have many, many 20K elemnts pre-allocated object arrays resulting in a shallow heap of 80K each, although there is only one element in the array.

      This sort of pre-allocation is causing a lot of memory pressure.

      Attachments

        1. screenshot-1.png
          167 kB
          Thomas Steinmaurer
        2. screenshot-2.png
          304 kB
          Thomas Steinmaurer
        3. screenshot-3.png
          301 kB
          Thomas Steinmaurer
        4. 16201_jfr_40b3_alloc.png
          2.52 MB
          Michael Semb Wever
        5. 16201_jfr_40b3_obj.png
          713 kB
          Michael Semb Wever
        6. 16201_jfr_3118_alloc.png
          2.49 MB
          Michael Semb Wever
        7. 16201_jfr_3118_obj.png
          730 kB
          Michael Semb Wever
        8. 16201_jfr_3023_alloc.png
          2.45 MB
          Michael Semb Wever
        9. 16201_jfr_3023_obj.png
          700 kB
          Michael Semb Wever
        10. screenshot-4.png
          153 kB
          Thomas Steinmaurer

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-13900 Massive GC suspension increase after updating to 3.0.14 from 2.1.18

          • Urgent - This issue should block release until it is resolved, and trigger immediate release once resolved.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-15430 Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap allocations compared to 2.1.18

          • Normal -
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-15430 Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap allocations compared to 2.1.18

          • Normal -
          • Resolved

          Activity

            People

              marcuse Marcus Eriksson
              tsteinmaurer Thomas Steinmaurer
              Marcus Eriksson
              Benjamin Lerer, Michael Semb Wever, Yifan Cai
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: