Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9240

Performance issue after a restart

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Low
    • Resolution: Fixed
    • 2.2.0 rc2, 3.0 alpha 1
    • None
    • None
    • Low

    Description

      I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk:

      • 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency.
      • trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency.
      • trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5.

      EDIT: branch cassandra-2.1 is OK.

      I can help to bisect and/or profile on Monday if needed.

      Attachments

        1. trace_query.cql
          0.2 kB
          Alan Boudreault
        2. runs.log
          214 kB
          Alan Boudreault
        3. run_issue.sh
          0.6 kB
          Alan Boudreault
        4. issue.yaml
          0.7 kB
          Alan Boudreault
        5. Cassandra.snapshots.zip
          9.70 MB
          Alan Boudreault
        6. cassandra_trunk-clientrequest-read.log
          2 kB
          Alan Boudreault
        7. cassandra_trunk.log
          113 kB
          Alan Boudreault
        8. cassandra_trunk_no_restart-clientrequest-read.log
          0.9 kB
          Alan Boudreault
        9. cassandra_trunk_no_restart.log
          86 kB
          Alan Boudreault
        10. cassandra_2.1.5-clientrequest-read.log
          1 kB
          Alan Boudreault
        11. cassandra_2.1.5.log
          93 kB
          Alan Boudreault
        12. cassandra_2.1.4-clientrequest-read.log
          5 kB
          Alan Boudreault
        13. cassandra_2.1.4.log
          88 kB
          Alan Boudreault

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-9573 OOM when loading sstables (system.hints)

          • Urgent - This issue should block release until it is resolved, and trigger immediate release once resolved.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-11301 Non-obsoleting compaction operations over compressed files can impose rate limit on normal reads

          • Normal -
          • Resolved

          Activity

            People

              benedict Benedict Elliott Smith
              aboudreault Alan Boudreault
              Benedict Elliott Smith
              T Jake Luciani
              Alan Boudreault Alan Boudreault
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: