Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-11847

Cassandra dies on a specific node in a multi-DC environment

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Not A Problem
    • None
    • None
    • Cassandra 2.0.11, JDK build 1.7.0_79-b15

    • Normal

    Description

      We've a customer who runs a 16 node 2 DC (8 nodes each) environment where Cassandra pid dies randomly but on a specific node.

      Whenever Cassandra dies, admin has to manually restart Cassandra only on that node.

      I tried upgrading their environment from java 1.7 (patch 60) to java 1.7 (patch 79) but it still seems to be an issue.

      Is this a known hardware related bug or should is this issue fixed in later Cassandra versions?

      1. A fatal error has been detected by the Java Runtime Environment:
        #
      2. SIGSEGV (0xb) at pc=0x00007f4542d5a27f, pid=19030, tid=139933154096896
        #
      3. JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15)
      4. Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
      5. Problematic frame:
      6. C [libjava.so+0xe027f] _fini+0xbd5f7
        #
      7. Core dump written. Default location: /tmp/core or core.19030
        #
      8. If you would like to submit a bug report, please visit:
      9. http://bugreport.java.com/bugreport/crash.jsp
        #

      --------------- T H R E A D ---------------

      Current thread (0x00007f453c89f000): JavaThread "COMMIT-LOG-WRITER" [_thread_in_vm, id=19115, stack(0x00007f44b9ed3000,0x00007f44b9f14000)]

      siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x00007f4542d5a27f

      Registers:
      RAX=0x0000000000000000, RBX=0x00007f453c564ad0, RCX=0x0000000000000001, RDX=0x0000000000000020
      RSP=0x00007f44b9f125a0, RBP=0x00007f44b9f125b0, RSI=0x0000000000000000, RDI=0x0000000000000001
      R8 =0x00007f453c564ad8, R9 =0x0000000000004aab, R10=0x00007f453917a52c, R11=0x00000006fae57068
      R12=0x00007f453c564ad8, R13=0x00007f44b9f125d0, R14=0x0000000000000000, R15=0x00007f453c89f000
      RIP=0x00007f4542d5a27f, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
      TRAPNO=0x000000000000000e

      -------------------------------------

      #

      1. A fatal error has been detected by the Java Runtime Environment:
        #
      2. SIGSEGV (0xb) at pc=0x00007f28e08787a4, pid=2912, tid=139798767699712
        #
      3. JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15)
      4. Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
      5. Problematic frame:
      6. C 0x00007f28e08787a4
        #
      7. Core dump written. Default location: /tmp/core or core.2912
        #
      8. If you would like to submit a bug report, please visit:
      9. http://bugreport.java.com/bugreport/crash.jsp
        #

      --------------- T H R E A D ---------------

      Current thread (0x00007f2640008000): JavaThread "ValidationExecutor:15" daemon [_thread_in_Java, id=7393, stack(0x00007f256fdf8000,0x00007f256fe39000)]

      siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x00007f28e08787a4

      Registers:
      RAX=0x00000000ffffffff, RBX=0x000000003f8bb878, RCX=0x00000000c77040d6, RDX=0x00000000c770409a
      RSP=0x00007f256fe37430, RBP=0x000000063b820710, RSI=0x000000063b820530, RDI=0x00000000ffffffff
      R8 =0x000000003f8bb888, R9 =0x0000000000000000, R10=0x000000003f8bb888, R11=0x000000003f8bb878
      R12=0x0000000000000000, R13=0x000000063b820530, R14=0x000000000000000b, R15=0x00007f2640008000
      RIP=0x00007f28e08787a4, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000015
      TRAPNO=0x000000000000000e

      Attachments

        1. java_error9552.log
          233 kB
          Rajesh Babu
        2. java_error7539.log
          347 kB
          Rajesh Babu
        3. java_error4571.log
          247 kB
          Rajesh Babu
        4. java_error2912.log
          289 kB
          Rajesh Babu
        5. java_error19030.log
          211 kB
          Rajesh Babu

        Activity

          People

            Unassigned Unassigned
            rbabu Rajesh Babu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: