Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
Description
11111 222222 333333 444444 11111 222222 333333 444444 I1129 23:24:45.469543 6592 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/MVgVC7/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/MVgVC7/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e/rootfs.overlay' I1129 23:24:45.473287 6592 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/MVgVC7/sha256:b56ae66c29370df48e7377c8f9baa744a3958058a766793f821dadcb144a4647 to rootfs '/tmp/mesos/store/docker/staging/MVgVC7/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3/rootfs.overlay' I1129 23:24:45.582002 6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6/rootfs.overlay' I1129 23:24:45.589404 6595 metadata_manager.cpp:167] Successfully cached image 'alpine' I1129 23:24:45.590204 6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e/rootfs.overlay' I1129 23:24:45.595190 6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f/rootfs.overlay' I1129 23:24:45.599500 6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721/rootfs.overlay' I1129 23:24:45.602047 6597 provisioner.cpp:506] Provisioning image rootfs '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/rootfses/b5d48445-848d-4274-a4f8-e909351ebc35' for container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 using overlay backend I1129 23:24:45.602751 6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:1db09adb5ddd7f1a07b6d585a7db747a51c7bd17418d47e91f901bdf420abd66 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16/rootfs.overlay' I1129 23:24:45.603054 6596 overlay.cpp:168] Created symlink '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/links' -> '/tmp/xAWQ8y' I1129 23:24:45.604398 6596 overlay.cpp:196] Provisioning image rootfs with overlayfs: 'lowerdir=/tmp/xAWQ8y/1:/tmp/xAWQ8y/0,upperdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/upperdir,workdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/workdir' I1129 23:24:45.607802 6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229/rootfs.overlay' I1129 23:24:45.612139 6594 registry_puller.cpp:395] Extracting layer tar ball '/tmp/mesos/store/docker/staging/6Zbc17/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to rootfs '/tmp/mesos/store/docker/staging/6Zbc17/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/rootfs.overlay' I1129 23:24:45.612253 6593 containerizer.cpp:1369] Checkpointed ContainerConfig at '/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/config' I1129 23:24:45.612298 6593 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from PROVISIONING to PREPARING I1129 23:24:45.625658 6596 containerizer.cpp:1838] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"clone_namespaces":[131072],"command":{"shell":true,"value":"sleep 1"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/mnt\/mesos\/sandbox"},{"name":"PATH","type":"VALUE","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"MESOS_CONTAINER_IP","type":"VALUE","value":"10.0.2.15"}]},"pre_exec_commands":[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/vagrant\/mesos\/build\/src\/mesos-containerizer"},{"arguments":["mount","-n","--rbind","\/tmp\/slaves\/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0\/frameworks\/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000\/executors\/default-executor\/runs\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5","\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5\/backends\/overlay\/rootfses\/b5d48445-848d-4274-a4f8-e909351ebc35\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}],"rootfs":"\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/01de09c5-d8e9-412e-8825-a592d2c875e5\/backends\/overlay\/rootfses\/b5d48445-848d-4274-a4f8-e909351ebc35","task_environment":{},"user":"root","working_directory":"\/mnt\/mesos\/sandbox"}" --pipe_read="12" --pipe_write="15" --runtime_directory="/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5" --unshare_namespace_mnt="false"' I1129 23:24:45.626317 6598 linux_launcher.cpp:438] Launching nested container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 and cloning with namespaces CLONE_NEWNS I1129 23:24:45.633211 6598 systemd.cpp:96] Assigned child process '6745' to 'mesos_executors.slice' I1129 23:24:45.636270 6596 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from PREPARING to ISOLATING I1129 23:24:45.691830 6597 metadata_manager.cpp:167] Successfully cached image 'mesosphere/inky' I1129 23:24:45.694399 6594 provisioner.cpp:506] Provisioning image rootfs '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/rootfses/1187cc83-a23a-4390-9c28-092a7b7690b5' for container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 using overlay backend I1129 23:24:45.694919 6596 overlay.cpp:168] Created symlink '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/links' -> '/tmp/GXhXiT' I1129 23:24:45.695103 6596 overlay.cpp:196] Provisioning image rootfs with overlayfs: 'lowerdir=/tmp/GXhXiT/6:/tmp/GXhXiT/5:/tmp/GXhXiT/4:/tmp/GXhXiT/3:/tmp/GXhXiT/2:/tmp/GXhXiT/1:/tmp/GXhXiT/0,upperdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/upperdir,workdir=/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/backends/overlay/scratch/1187cc83-a23a-4390-9c28-092a7b7690b5/workdir' I1129 23:24:45.696255 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:45.696349 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers I1129 23:24:45.696449 6593 containerizer.cpp:1369] Checkpointed ContainerConfig at '/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3/config' I1129 23:24:45.696506 6593 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from PROVISIONING to PREPARING I1129 23:24:45.697865 6595 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:45.697918 6595 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:45.697968 6595 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:45.697999 6595 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:45.698025 6595 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:45.698050 6595 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache I1129 23:24:45.698076 6595 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:45.698104 6595 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:45.698129 6595 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache I1129 23:24:45.698894 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:45.698966 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers I1129 23:24:45.700333 6596 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:45.700394 6596 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:45.700412 6596 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:45.700428 6596 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:45.700441 6596 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:45.700454 6596 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache I1129 23:24:45.700467 6596 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:45.700480 6596 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:45.700495 6596 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache I1129 23:24:45.702491 6594 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:45.702554 6594 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers I1129 23:24:45.703707 6592 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:45.703783 6592 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:45.703812 6592 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:45.703816 6592 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:45.703816 6592 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:45.704164 6592 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache I1129 23:24:45.704208 6592 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:45.704255 6592 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:45.704285 6592 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache I1129 23:24:45.704814 6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:45.704861 6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers I1129 23:24:45.708112 6592 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:45.708204 6592 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:45.708238 6592 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:45.708264 6592 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:45.708375 6592 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:45.708407 6592 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache I1129 23:24:45.708472 6592 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:45.708514 6592 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:45.708545 6592 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache I1129 23:24:45.709048 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:45.709161 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers I1129 23:24:45.710321 6594 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:45.710382 6594 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:45.710412 6594 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:45.710436 6594 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:45.710458 6594 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:45.710480 6594 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache I1129 23:24:45.710522 6594 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:45.710551 6594 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:45.710590 6594 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache I1129 23:24:45.713176 6594 containerizer.cpp:1838] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"clone_namespaces":[131072],"command":{"shell":true,"value":"sleep 100000"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/mnt\/mesos\/sandbox"},{"name":"HOME","type":"VALUE","value":"\/"},{"name":"PATH","type":"VALUE","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"MESOS_CONTAINER_IP","type":"VALUE","value":"10.0.2.15"}]},"pre_exec_commands":[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/vagrant\/mesos\/build\/src\/mesos-containerizer"},{"arguments":["mount","-n","--rbind","\/tmp\/slaves\/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0\/frameworks\/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000\/executors\/default-executor\/runs\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3","\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3\/backends\/overlay\/rootfses\/1187cc83-a23a-4390-9c28-092a7b7690b5\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}],"rootfs":"\/tmp\/provisioner\/containers\/3bbc3fd1-0138-43a9-94ba-d017d813daac\/containers\/7be37efd-7ec4-459a-a51a-97f6be99c3c3\/backends\/overlay\/rootfses\/1187cc83-a23a-4390-9c28-092a7b7690b5","task_environment":{},"user":"root","working_directory":"\/mnt\/mesos\/sandbox"}" --pipe_read="13" --pipe_write="14" --runtime_directory="/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3" --unshare_namespace_mnt="false"' I1129 23:24:45.713954 6597 linux_launcher.cpp:438] Launching nested container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 and cloning with namespaces CLONE_NEWNS I1129 23:24:45.721781 6597 systemd.cpp:96] Assigned child process '6775' to 'mesos_executors.slice' I1129 23:24:45.725494 6594 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from PREPARING to ISOLATING I1129 23:24:45.791635 6595 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from ISOLATING to FETCHING I1129 23:24:45.791880 6591 fetcher.cpp:379] Starting to fetch URIs for container: 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5, directory: /tmp/slaves/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0/frameworks/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000/executors/default-executor/runs/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5 I1129 23:24:45.792626 6591 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from FETCHING to RUNNING 11111 222222 333333 444444 I1129 23:24:45.807262 6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:45.807375 6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers I1129 23:24:45.808658 6591 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:45.808843 6591 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:45.808869 6591 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:45.808897 6591 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:45.808962 6591 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:45.808990 6591 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache I1129 23:24:45.809012 6591 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:45.809036 6591 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:45.809057 6591 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache I1129 23:24:45.893280 6596 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from ISOLATING to FETCHING I1129 23:24:45.893523 6596 fetcher.cpp:379] Starting to fetch URIs for container: 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3, directory: /tmp/slaves/65fa58c5-48c6-4998-b336-ceb9bcd2ec43-S0/frameworks/55b825d4-c922-46bb-ab8e-4e01abf1a756-0000/executors/default-executor/runs/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/7be37efd-7ec4-459a-a51a-97f6be99c3c3 I1129 23:24:45.894335 6594 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 from FETCHING to RUNNING I1129 23:24:45.902606 6598 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1/executor' I1129 23:24:45.903908 6597 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1' I1129 23:24:45.904618 6597 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1' I1129 23:24:45.908681 6597 http.cpp:1185] HTTP POST for /slave(1)/api/v1 from 10.0.2.15:57620 I1129 23:24:45.909113 6597 http.cpp:1185] HTTP POST for /slave(1)/api/v1 from 10.0.2.15:57622 I1129 23:24:45.909708 6597 http.cpp:2589] Processing WAIT_NESTED_CONTAINER call for container '3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5' I1129 23:24:45.910148 6597 http.cpp:2589] Processing WAIT_NESTED_CONTAINER call for container '3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3' I1129 23:24:45.938350 6596 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1/executor' I1129 23:24:45.938781 6596 http.cpp:1185] HTTP POST for /slave(1)/api/v1/executor from 10.0.2.15:57564 I1129 23:24:45.939141 6596 slave.cpp:4584] Handling status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.940809 6591 http.cpp:1185] HTTP POST for /slave(1)/api/v1/executor from 10.0.2.15:57564 I1129 23:24:45.941130 6591 slave.cpp:4584] Handling status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.942906 6591 task_status_update_manager.cpp:328] Received task status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.943076 6591 task_status_update_manager.cpp:383] Forwarding task status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent I1129 23:24:45.943322 6595 slave.cpp:5067] Forwarding the update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to master@127.0.0.1:5050 I1129 23:24:45.943332 6591 task_status_update_manager.cpp:328] Received task status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.943506 6591 task_status_update_manager.cpp:383] Forwarding task status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent I1129 23:24:45.943717 6595 slave.cpp:4960] Task status update manager successfully handled status update TASK_RUNNING (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.943905 6595 slave.cpp:5067] Forwarding the update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to master@127.0.0.1:5050 I1129 23:24:45.943905 6595 slave.cpp:4960] Task status update manager successfully handled status update TASK_RUNNING (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.992866 6591 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.993261 6595 slave.cpp:3868] Task status update manager successfully handled status update acknowledgement (UUID: 6ba2641b-ff6a-4ac2-9123-23579534147d) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.993968 6593 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:45.994295 6598 slave.cpp:3868] Task status update manager successfully handled status update acknowledgement (UUID: f22fe2c5-2a2e-43c9-9f22-a7da351bab08) for task 2 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 11111 222222 333333 444444 I1129 23:24:46.808684 6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:46.808876 6592 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers I1129 23:24:46.810683 6593 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:46.810751 6593 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:46.810781 6593 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:46.810808 6593 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:46.810834 6593 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:46.810860 6593 store.cpp:531] Layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' is retained by image store cache I1129 23:24:46.810885 6593 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:46.810911 6593 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:46.810937 6593 store.cpp:531] Layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' is retained by image store cache I1129 23:24:47.057590 6596 containerizer.cpp:2775] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has exited I1129 23:24:47.057667 6596 containerizer.cpp:2324] Destroying container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 in RUNNING state I1129 23:24:47.057695 6596 containerizer.cpp:2926] Transitioning the state of container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 from RUNNING to DESTROYING I1129 23:24:47.058082 6596 linux_launcher.cpp:514] Asked to destroy container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 I1129 23:24:47.059027 6596 linux_launcher.cpp:560] Using freezer to destroy cgroup mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5 I1129 23:24:47.060667 6596 cgroups.cpp:3058] Freezing cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5 I1129 23:24:47.062700 6597 cgroups.cpp:1413] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5 after 1.838336ms I1129 23:24:47.064627 6592 cgroups.cpp:3076] Thawing cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5 I1129 23:24:47.066498 6598 cgroups.cpp:1442] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/3bbc3fd1-0138-43a9-94ba-d017d813daac/mesos/01de09c5-d8e9-412e-8825-a592d2c875e5 after 1.642752ms I1129 23:24:47.071521 6592 provisioner.cpp:648] Destroying container rootfs at '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/rootfses/b5d48445-848d-4274-a4f8-e909351ebc35' for container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 I1129 23:24:47.098203 6596 overlay.cpp:296] Removed temporary directory '/tmp/xAWQ8y' pointed by '/tmp/provisioner/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/backends/overlay/scratch/b5d48445-848d-4274-a4f8-e909351ebc35/links' I1129 23:24:47.100265 6591 containerizer.cpp:2613] Checkpointing termination state to nested container's runtime directory '/var/run/mesos/containers/3bbc3fd1-0138-43a9-94ba-d017d813daac/containers/01de09c5-d8e9-412e-8825-a592d2c875e5/termination' I1129 23:24:47.107206 6594 process.cpp:3503] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/api/v1/executor' I1129 23:24:47.151911 6594 http.cpp:1185] HTTP POST for /slave(1)/api/v1/executor from 10.0.2.15:57564 I1129 23:24:47.152243 6594 slave.cpp:4584] Handling status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:47.154391 6594 task_status_update_manager.cpp:328] Received task status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:47.154578 6594 task_status_update_manager.cpp:383] Forwarding task status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to the agent I1129 23:24:47.154810 6593 slave.cpp:5067] Forwarding the update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 to master@127.0.0.1:5050 I1129 23:24:47.157249 6593 slave.cpp:4960] Task status update manager successfully handled status update TASK_FINISHED (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:47.385977 6592 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:47.386334 6592 task_status_update_manager.cpp:538] Cleaning up status update stream for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:47.387459 6592 slave.cpp:3868] Task status update manager successfully handled status update acknowledgement (UUID: 288dbdf5-3dde-48fa-b2d4-bfd6c098ea44) for task 1 of framework 55b825d4-c922-46bb-ab8e-4e01abf1a756-0000 I1129 23:24:47.387568 6592 slave.cpp:8457] Completing task 1 11111 222222 333333 444444 I1129 23:24:47.818768 6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:47.824591 6598 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:47.824724 6598 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:47.824753 6598 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:47.824767 6598 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:47.824782 6598 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:47.824861 6598 store.cpp:550] Marking layer '38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' to gc by renaming '/tmp/mesos/store/docker/layers/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e' to '/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784' I1129 23:24:47.824918 6598 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:47.824960 6598 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache I1129 23:24:47.825088 6598 store.cpp:550] Marking layer 'b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' to gc by renaming '/tmp/mesos/store/docker/layers/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3' to '/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040' I1129 23:24:47.825389 6598 store.cpp:577] Deleting path '/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040' I1129 23:24:47.829903 6598 store.cpp:584] Deleted '/tmp/mesos/store/docker/gc/b5815a31a59b66c909dbf6c670de78690d4b52649b8e283fc2bfd2594f61cca3.1511997887825015040' I1129 23:24:47.829980 6598 store.cpp:577] Deleting path '/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784' I1129 23:24:47.830047 6598 store.cpp:584] Deleted '/tmp/mesos/store/docker/gc/38135e3743e6dcb66bd1394b633053714333c00007b7cf930bfeebfda660c06e.1511997887824822784' 11111 222222 333333 444444 I1129 23:24:48.829519 6595 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:48.831161 6598 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:48.831225 6598 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:48.831248 6598 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:48.831266 6598 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:48.831284 6598 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:48.831302 6598 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:48.831321 6598 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache 11111 222222 333333 444444 I1129 23:24:49.830904 6597 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:49.832487 6597 store.cpp:531] Layer 'be4ce2753831b8952a5b797cf45b2230e1befead6f5db0630bcb24a5f554255e' is retained by image store cache I1129 23:24:49.832584 6597 store.cpp:531] Layer '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' is retained by image store cache I1129 23:24:49.833329 6597 store.cpp:531] Layer '53b5066c5a7dff5d6f6ef0c1945572d6578c083d550d2a3d575b4cdf7460306f' is retained by image store cache I1129 23:24:49.833367 6597 store.cpp:531] Layer 'e28617c6dd2169bfe2b10017dfaa04bd7183ff840c4f78ebe73fca2a89effeb6' is retained by image store cache I1129 23:24:49.833387 6597 store.cpp:531] Layer '120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16' is retained by image store cache I1129 23:24:49.833406 6597 store.cpp:531] Layer 'a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721' is retained by image store cache I1129 23:24:49.833425 6597 store.cpp:531] Layer '42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229' is retained by image store cache
Please neglect the debugging logs like '111111'. To reproduce this issue, just continuously trigger image gc. The log above was from a scenario that we launch two nested containers. One sleeps 1 second, another sleep forever.
This is related to this patch: https://github.com/apache/mesos/commit/e273efe6976434858edb85bbcf367a02e963a467#diff-a3593ed0ebd2b205775f7f04d9b5afe7
The root cause is that we did not set the 'layers' after we checkpoint the layer ids in provisioner. The log below is the prove:
I1129 23:24:45.698894 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.7be37efd-7ec4-459a-a51a-97f6be99c3c3 has no checkpointed layers I1129 23:24:45.698966 6596 provisioner.cpp:714] Container 3bbc3fd1-0138-43a9-94ba-d017d813daac.01de09c5-d8e9-412e-8825-a592d2c875e5 has no checkpointed layers