Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-10595

[snapshot-LR] OM node is not shown in 'om roles' as follower node

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • OM, Snapshot

    Description

      OM node is not shown in 'om roles' as follower node

      OM roles - CLI -

      [root@node~]# ozone admin om roles -id=ozone1709211738
om233 : FOLLOWER (node1.domain.com)
om232 : LEADER (node2.domain.com)

      OM log output -

      2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress
2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: inconsistency entries. Reply:om232<-om363#14732148:FAIL-t54,INCONSISTENCY,nextIndex=0,followerCommit=-1,matchIndex=-1
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: receive installSnapshot: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: reply installSnapshot: om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastRequest: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastReply: null
2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress
2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: inconsistency entries. Reply:om232<-om363#14732149:FAIL-t54,INCONSISTENCY,nextIndex=0,followerCommit=-1,matchIndex=-1
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: receive installSnapshot: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: reply installSnapshot: om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastRequest: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastReply: null
2024-03-19 00:00:01,204 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress
2024-03-19 00:00:01,204 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: inconsistency entries. Reply:om232<-om363#14732150:FAIL-t54,INCONSISTENCY,nextIndex=0,followerCommit=-1,matchIndex=-1
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: receive installSnapshot: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: reply installSnapshot: om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastRequest: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastReply: null
2024-03-19 00:00:01,204 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress

       OM Leader log output pertaining to above OM Follower -

      2024-03-19 00:03:11,531 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,531 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,532 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,532 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,533 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,534 INFO [grpc-default-executor-26]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,534 INFO [grpc-default-executor-26]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,535 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS

      From logs the snapshot installation is failing in the the follower OM, this log is flooding in the follower OM and similar error is present in leader indicating the snapshot-installation is failing/in-progress.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            jyosin Jyotirmoy Sinha
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: