Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15013

Prevent client requests from blocking on executor task queue

    XMLWordPrintableJSON

Details

    Description

      This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue bounded, since, in the current state, items get added to the queue without any checks on queue size, nor with any checks on netty outbound buffer to check the isWritable state.
      We are seeing this issue hit our production 3.0 clusters quite often.

      Attachments

        1. heap dump showing each ImmediateFlusher taking upto 600MB.png
          321 kB
          Sumanth Pasupuleti
        2. BlockedEpollEventLoopFromHeapDump.png
          186 kB
          Sumanth Pasupuleti
        3. BlockedEpollEventLoopFromThreadDump.png
          658 kB
          Sumanth Pasupuleti
        4. RequestExecutorQueueFull.png
          150 kB
          Sumanth Pasupuleti
        5. perftest_writeops.png
          213 kB
          Sumanth Pasupuleti
        6. perftest_blockedthreads.png
          176 kB
          Sumanth Pasupuleti
        7. perftest_readops.png
          213 kB
          Sumanth Pasupuleti
        8. perftest_readlatency_99th.png
          246 kB
          Sumanth Pasupuleti
        9. perftest_readlatency_avg.png
          208 kB
          Sumanth Pasupuleti
        10. perftest_writelatency_99th.png
          280 kB
          Sumanth Pasupuleti
        11. perftest_writelatency_avg.png
          216 kB
          Sumanth Pasupuleti
        12. perftest_cpu_usage.png
          282 kB
          Sumanth Pasupuleti
        13. perftest_heap_usage.png
          513 kB
          Sumanth Pasupuleti
        14. perftest_connections_count.png
          259 kB
          Sumanth Pasupuleti
        15. perftest2_readops.png
          275 kB
          Sumanth Pasupuleti
        16. perftest2_read_latency_99th.png
          340 kB
          Sumanth Pasupuleti
        17. perftest2_read_latency_avg.png
          258 kB
          Sumanth Pasupuleti
        18. perftest2_writeops.png
          263 kB
          Sumanth Pasupuleti
        19. perftest2_write_latency_avg.png
          252 kB
          Sumanth Pasupuleti
        20. perftest2_write_latency_99th.png
          346 kB
          Sumanth Pasupuleti
        21. perftest2_blocked_threadpool.png
          89 kB
          Sumanth Pasupuleti
        22. perftest2_cpu_usage.png
          200 kB
          Sumanth Pasupuleti
        23. perftest2_heap.png
          1.20 MB
          Sumanth Pasupuleti
        24. perftest2_15013_patch_flamegraph.svg
          741 kB
          Sumanth Pasupuleti
        25. perftest2_15013_base_flamegraph.svg
          622 kB
          Sumanth Pasupuleti
        26. 15013-3.11.txt
          58 kB
          Sumanth Pasupuleti
        27. 15013-3.0.txt
          58 kB
          Sumanth Pasupuleti
        28. 15013-trunk.txt
          55 kB
          Sumanth Pasupuleti

        Issue Links

          is related to

          Task - A task that needs to be done. CASSANDRA-16100 Blog post on Improving Resiliency w/ "Prevent client requests from blocking on executor task queue"

          • Normal -
          • Resolved

          Activity

            People

              sumanth.pasupuleti Sumanth Pasupuleti
              sumanth.pasupuleti Sumanth Pasupuleti
              Sumanth Pasupuleti
              Benedict Elliott Smith
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: