Details
Description
As observed in an internal cluster:
mesos-default-executor: /pkg/src/mesos/3rdparty/stout/include/stout/option.hpp:112: T& Option<T>::get() & [with T = mesos::ContainerID]: Assertion `isSome()' failed. *** Aborted at 1527514147 (unix time) try "date -d @1527514147" if you are using GNU date *** PC: @ 0x7f9fe3b5c1f7 (unknown) *** SIGABRT (@0x6300000005) received by PID 5 (TID 0x7f9fdfe8e700) from PID 5; stack trace: *** @ 0x7f9fe3ef95e0 (unknown) @ 0x7f9fe3b5c1f7 (unknown) @ 0x7f9fe3b5d8e8 (unknown) @ 0x7f9fe3b55266 (unknown) @ 0x7f9fe3b55312 (unknown) @ 0x7f9fe581b9b0 _ZNR6OptionIN5mesos11ContainerIDEE3getEv.part.134 @ 0x7f9fe58a19f5 _ZZN5mesos8internal6checks14CheckerProcess18nestedCommandCheckEvENKUlRKN7process4http8ResponseEE0_clES7_ @ 0x7f9fe66a8edc process::ProcessManager::resume() @ 0x7f9fe66ae856 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv @ 0x7f9fe46d32b0 (unknown) @ 0x7f9fe3ef1e25 (unknown) @ 0x7f9fe3c1f34d (unknown)
The issue is caused by not this block in CheckerProcess not checking that previousCheckContainerId is still some after it had yielded control:
// checker_process.cpp:649 LOG(WARNING) << "Connection to remove the nested container '" << previousCheckContainerId.get() << "' used for the " << name << " for task '" << taskId << "' failed: " << failure;