Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-4394

Learner.syncWithLeader got NullPointerException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.7.0
    • 3.10.0, 3.9.3
    • server
    • ZooKeeper 3.7.0

    Description

      ZooKeeper follower node encountered NullPointerException during syncWithLeader.

      Logs indicate that the follower has received NEWLEADER packet between a PROPOSAL packet and it's corresponding COMMIT packet. The NEWLEADER packet leads to packetsNotCommitted.clear(), yet the COMMIT packet still wants to do packetsNotCommitted.peekFirst() to get the former PROPOSAL packet, and the later if-statement raised NPE.

      case Leader.COMMIT:
      case Leader.COMMITANDACTIVATE:
          pif = packetsNotCommitted.peekFirst();
          if (pif.hdr.getZxid() == qp.getZxid() && qp.getType() == Leader.COMMITANDACTIVATE) {
              // ...
          }

      After look into the Leader side, I found:

      1. LearnerHandler.syncFollower queues packets with zxid <= maxCommittedLog (PROPOSAL/COMMIT pairs);
      2. Leader.startForwarding queues toBeApplied packets(PROPOSAL/COMMIT pairs);
      3. Leader.startForwarding queues outstandingProposals packets(PROSOAL only);
      4. LeanerHandler.run sends NEWLEADER message.

      Seams if the outstandingProposals is not empty at the certain moment, the follower could then receive PROPOSAL/NEWLEADER/COMMIT packets in order.

      The follower will retry from LOOKING again and is expected to be succeed at last, however, under heavy load it may be too many retries. Further more, I my case the follower has to sync data from leader's disk, and start over again after the NPE(prior sync not flushed?), which may harm the leader.

      I don't know if it is designed so or not, but consider the performance, can we at least avoid wasting of network/disk IO?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              littleorca Liu Haifeng
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 50m
                  2h 50m