ZK ensemble lost quorum and lots of exceptions seen in all ZK pods

Running ZK cluster in kubernetes environment. ZK ensemble has 3 nodes

ZK ensemble is in a state where it looks like quorum is lost and see bunch of I/O exceptions . ZK cluster is not recovering from this state.

Log files have exceptions "Zookeeper is not running" but on exec to pod and running command it seems ok
echo ruok | nc 127.0.0.1 2181
imok

attaching the logs