[MESOS-7587] Launching tasks with the Mesos Containerizer after a long time without launching new tasks fails - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Not A Problem
Affects Version/s: 1.2.0
Fix Version/s: None
Component/s: containerization
Labels:
- mesos-containerizer
- mesosphere

Description

After having a cluster running without launching new tasks for an extended period of time, ~1week. When launching a new task using the Mesos Containerizer, the task fails to launch with the error:

Failed to execute command: No such file or directory

The task is launched from Marathon with the app definition:

{
    "container": {
        "type": "MESOS",
        "docker": {
            "forcePullImage": true,
            "image": "private.repository.local/updated:fixed",
            "privileged": false
        }
    },
    "cpus": 0.1,
    "id": "/20150530/mesos9",
    "instances": 1,
    "minimumHealthCapacity": 1,
    "acceptedResourceRoles": ["*"],
    "constraints": [["hostname", "UNIQUE"]],
    "mem": 128
}

and Dockerfile

FROM private.repository.local/centos:stable
MAINTAINER Sebastian Gerlach "sge@boreus.de"

CMD python -m SimpleHTTPServer 80

The obtained stdout is:

Executing pre-exec command '{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/usr\/libexec\/mesos\/mesos-containerizer"}'
Executing pre-exec command '{"arguments":["mount","-n","--rbind","\/var\/lib\/mesos\/slaves\/562f1892-41c6-4c55-955e-662c63ab3ddf-S8\/frameworks\/562f1892-41c6-4c55-955e-662c63ab3ddf-0000\/executors\/20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0\/runs\/da1cb066-1354-42a6-82bc-eee688d61b94","\/var\/lib\/mesos\/provisioner\/containers\/da1cb066-1354-42a6-82bc-eee688d61b94\/backends\/overlay\/rootfses\/f74816b3-c115-4a5e-a51c-fbd502efd1fe\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}'
Received SUBSCRIBED event
Subscribed executor on bp-mesos8.private.local
Received LAUNCH event
Starting task 20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0
/usr/libexec/mesos/mesos-containerizer launch --help="false" --launch_info="{"command":{"arguments":["\/bin\/sh","-c","python -m SimpleHTTPServer 80"],"environment":{"variables":[{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"}]},"shell":false,"value":"\/bin\/sh"},"environment":{"variables":[{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"LIBPROCESS_IP","value":"0.0.0.0"},{"name":"LIBPROCESS_PORT","value":"0"},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"MESOS_AGENT_ENDPOINT","value":"10.107.19.26:5051"},{"name":"MESOS_CHECKPOINT","value":"1"},{"name":"MESOS_DIRECTORY","value":"\/var\/lib\/mesos\/slaves\/562f1892-41c6-4c55-955e-662c63ab3ddf-S8\/frameworks\/562f1892-41c6-4c55-955e-662c63ab3ddf-0000\/executors\/20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0\/runs\/da1cb066-1354-42a6-82bc-eee688d61b94"},{"name":"MESOS_EXECUTOR_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD","value":"5secs"},{"name":"MESOS_FRAMEWORK_ID","value":"562f1892-41c6-4c55-955e-662c63ab3ddf-0000"},{"name":"MESOS_HTTP_COMMAND_EXECUTOR","value":"0"},{"name":"MESOS_NATIVE_JAVA_LIBRARY","value":"\/usr\/lib\/libmesos-1.2.0.so"},{"name":"MESOS_NATIVE_LIBRARY","value":"\/usr\/lib\/libmesos-1.2.0.so"},{"name":"MESOS_RECOVERY_TIMEOUT","value":"15mins"},{"name":"MESOS_SANDBOX","value":"\/mnt\/mesos\/sandbox"},{"name":"MESOS_SLAVE_ID","value":"562f1892-41c6-4c55-955e-662c63ab3ddf-S8"},{"name":"MESOS_SLAVE_PID","value":"slave(1)@10.107.19.26:5051"},{"name":"MESOS_SUBSCRIPTION_BACKOFF_MAX","value":"2secs"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"PATH","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"PATH","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"ftp_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"http_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"https_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"no_proxy","value":"localhost,127.0.0.1,svc.boreus.de,private.local"},{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"}]},"rootfs":"\/var\/lib\/mesos\/provisioner\/containers\/da1cb066-1354-42a6-82bc-eee688d61b94\/backends\/overlay\/rootfses\/f74816b3-c115-4a5e-a51c-fbd502efd1fe","user":"root","working_directory":"\/mnt\/mesos\/sandbox"}" --unshare_namespace_mnt="true"
Forked command at 10438
Changing root to /var/lib/mesos/provisioner/containers/da1cb066-1354-42a6-82bc-eee688d61b94/backends/overlay/rootfses/f74816b3-c115-4a5e-a51c-fbd502efd1fe
Overwriting environment variable 'PATH', original: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', new: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
Overwriting environment variable 'MARATHON_APP_VERSION', original: '2017-05-30T09:06:59.666Z', new: '2017-05-30T09:06:59.666Z'
Overwriting environment variable 'HOST', original: 'bp-mesos8.private.local', new: 'bp-mesos8.private.local'
Overwriting environment variable 'MARATHON_APP_RESOURCE_CPUS', original: '0.1', new: '0.1'
Overwriting environment variable 'MARATHON_APP_RESOURCE_GPUS', original: '0', new: '0'
Overwriting environment variable 'MESOS_TASK_ID', original: '20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0', new: '20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0'
Overwriting environment variable 'MARATHON_APP_RESOURCE_MEM', original: '128.0', new: '128.0'
Overwriting environment variable 'MARATHON_APP_RESOURCE_DISK', original: '0.0', new: '0.0'
Overwriting environment variable 'MARATHON_APP_LABELS', original: '', new: ''
Overwriting environment variable 'MARATHON_APP_ID', original: '/20150530/mesos13', new: '/20150530/mesos13'
Command exited with status 1 (pid: 10438)

And a portion of the logs:

tail -f mesos-slave.ERROR mesos-slave.INFO mesos-slave.WARNING
==> mesos-slave.ERROR <==
Log file created at: 2017/05/30 10:48:17
Running on machine: bp-mesos7.private.local
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0530 10:48:17.653518  8876 shell.hpp:107] Command 'hadoop version 2>&1' failed; this is the output:
sh: hadoop: command not found

==> mesos-slave.INFO <==
I0530 10:48:19.953788  8906 slave.cpp:5558] Finished recovery
I0530 10:48:19.954376  8906 slave.cpp:931] New master detected at master@10.107.19.21:5050
I0530 10:48:19.954401  8906 slave.cpp:955] No credentials provided. Attempting to register without authentication
I0530 10:48:19.954416  8906 slave.cpp:966] Detecting new master
I0530 10:48:19.954444  8906 status_update_manager.cpp:177] Pausing sending status updates
I0530 10:48:20.208361  8907 slave.cpp:1242] Re-registered with master master@10.107.19.21:5050
I0530 10:48:20.208427  8907 slave.cpp:1279] Forwarding total oversubscribed resources {}
I0530 10:48:20.208452  8906 status_update_manager.cpp:184] Resuming sending status updates
I0530 10:48:20.208501  8907 slave.cpp:2827] Updating framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 pid to scheduler-afcaa9d7-6040-476d-8a86-d97138cc23c8@10.107.19.22:39431
I0530 10:48:20.208871  8907 status_update_manager.cpp:184] Resuming sending status updates

==> mesos-slave.WARNING <==
Log file created at: 2017/05/30 10:48:17
Running on machine: bp-mesos7.private.local
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0530 10:48:17.653518  8876 shell.hpp:107] Command 'hadoop version 2>&1' failed; this is the output:
sh: hadoop: command not found
W0530 10:48:17.694730  8903 state.cpp:477] Failed to find executor forked pid file '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.7c34f793-44fd-11e7-8f05-02420d613ec3/runs/0ac924bb-4ba5-41e9-b563-6768329f6f7d/pids/forked.pid'
W0530 10:48:17.695217  8903 state.cpp:477] Failed to find executor forked pid file '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.7041e28f-44fd-11e7-8f05-02420d613ec3/runs/d8afe43c-292c-4069-88d7-7edf4a730575/pids/forked.pid'
W0530 10:48:17.713217  8903 state.cpp:477] Failed to find executor forked pid file '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.0cad114c-44fe-11e7-8f05-02420d613ec3/runs/3adaddbb-5fd9-4ca6-8225-660daf91a7d9/pids/forked.pid'
W0530 10:48:17.741708  8904 linux_launcher.cpp:357] Couldn't find pid '14251' in 'mesos_executors.slice'. This can lead to lack of proper resource isolation
W0530 10:48:17.741747  8904 linux_launcher.cpp:357] Couldn't find pid '19408' in 'mesos_executors.slice'. This can lead to lack of proper resource isolation






==> mesos-slave.INFO <==
I0530 10:49:17.692493  8903 slave.cpp:5267] Current disk usage 0.88%. Max allowed age: 6.238522372392419days
I0530 10:49:32.356741  8905 slave.cpp:1625] Got assigned task '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.357070  8905 slave.cpp:1785] Launching task '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.357969  8905 paths.cpp:547] Trying to chown '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1' to user 'root'
I0530 10:49:32.358539  8905 slave.cpp:6480] Launching executor '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32 in work directory '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
I0530 10:49:32.359091  8903 docker.cpp:1104] Skipping non-docker container
I0530 10:49:32.359094  8905 slave.cpp:2118] Queued task '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for executor '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.359357  8903 containerizer.cpp:992] Starting container 424fd077-caf5-4b67-90b0-760a23775ec1 for executor '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.542762  8902 provisioner.cpp:453] Provisioning image rootfs '/var/lib/mesos/provisioner/containers/424fd077-caf5-4b67-90b0-760a23775ec1/backends/overlay/rootfses/ea0f760c-5079-4893-a5ec-469af9715abd' for container 424fd077-caf5-4b67-90b0-760a23775ec1 using overlay backend
I0530 10:49:32.555275  8901 memory.cpp:451] Started listening for OOM events for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.555717  8901 memory.cpp:562] Started listening on 'low' memory pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.556246  8901 memory.cpp:562] Started listening on 'medium' memory pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.556879  8901 memory.cpp:562] Started listening on 'critical' memory pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.559020  8901 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 160MB for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.559835  8901 memory.cpp:251] Updated 'memory.limit_in_bytes' to 160MB for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.560269  8901 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2) for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.566457  8902 linux_launcher.cpp:429] Launching container 424fd077-caf5-4b67-90b0-760a23775ec1 and cloning with namespaces CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWNET
I0530 10:49:32.568379  8902 systemd.cpp:96] Assigned child process '9019' to 'mesos_executors.slice'
I0530 10:49:32.570240  8901 containerizer.cpp:1637] Checkpointing container's forked pid 9019 to '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1/pids/forked.pid'
I0530 10:49:32.570843  8901 cni.cpp:872] Bind mounted '/proc/9019/ns/net' to '/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1/ns' for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.541935  8908 cni.cpp:1290] Got assigned IPv4 address '192.168.235.73/32' from CNI network 'calico-net-1' for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.542150  8908 cni.cpp:999] Unable to find DNS nameservers for container 424fd077-caf5-4b67-90b0-760a23775ec1. Using host '/etc/resolv.conf'
I0530 10:49:35.703557  8906 slave.cpp:3385] Got registration for executor '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.73:38212
I0530 10:49:35.704555  8908 disk.cpp:207] Updating the disk resources for container 424fd077-caf5-4b67-90b0-760a23775ec1 to cpus(*)(allocated: *):0.2; mem(*)(allocated: *):160
I0530 10:49:35.705009  8906 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 160MB for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.705909  8906 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2) for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.706162  8906 slave.cpp:2331] Sending queued task '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' to executor '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.73:38212
I0530 10:49:35.713090  8905 slave.cpp:3816] Handling status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.73:38212
I0530 10:49:35.713526  8905 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.713686  8905 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.713820  8905 slave.cpp:4256] Forwarding the update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:35.713904  8905 slave.cpp:4166] Sending acknowledgement for status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.73:38212
I0530 10:49:35.757575  8905 status_update_manager.cpp:395] Received status update acknowledgement (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.757642  8905 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.801589  8905 slave.cpp:3816] Handling status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.73:38212
I0530 10:49:35.802206  8906 disk.cpp:207] Updating the disk resources for container 424fd077-caf5-4b67-90b0-760a23775ec1 to cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32
I0530 10:49:35.802685  8908 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 32MB for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.803598  8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1) for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.803773  8908 status_update_manager.cpp:323] Received status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.803802  8908 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.803984  8903 slave.cpp:4256] Forwarding the update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:35.804077  8903 slave.cpp:4166] Sending acknowledgement for status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.73:38212
I0530 10:49:35.817703  8906 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.817754  8906 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:36.804152  8906 slave.cpp:4388] Got exited event for executor(1)@192.168.235.73:38212
I0530 10:49:36.847506  8906 containerizer.cpp:2479] Container 424fd077-caf5-4b67-90b0-760a23775ec1 has exited
I0530 10:49:36.847568  8906 containerizer.cpp:2116] Destroying container 424fd077-caf5-4b67-90b0-760a23775ec1 in RUNNING state
I0530 10:49:36.847664  8906 linux_launcher.cpp:505] Asked to destroy container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.848474  8906 linux_launcher.cpp:548] Using freezer to destroy cgroup mesos/424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.849936  8906 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.851379  8907 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1 after 1.411072ms
I0530 10:49:36.853444  8906 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.855401  8906 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1 after 1.910016ms
I0530 10:49:37.375128  8902 slave.cpp:1625] Got assigned task '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.375424  8902 slave.cpp:1785] Launching task '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.375635  8902 paths.cpp:547] Trying to chown '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000' to user 'root'
I0530 10:49:37.376144  8902 slave.cpp:6480] Launching executor '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32 in work directory '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
I0530 10:49:37.376639  8902 slave.cpp:2118] Queued task '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for executor '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.376938  8902 docker.cpp:1104] Skipping non-docker container
I0530 10:49:37.377231  8902 containerizer.cpp:992] Starting container 1248ce41-4143-4adb-98d0-04c6cf8ff000 for executor '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.565088  8901 provisioner.cpp:453] Provisioning image rootfs '/var/lib/mesos/provisioner/containers/1248ce41-4143-4adb-98d0-04c6cf8ff000/backends/overlay/rootfses/585e6e95-cd8b-4ed9-8341-49360770675b' for container 1248ce41-4143-4adb-98d0-04c6cf8ff000 using overlay backend
I0530 10:49:37.578210  8901 memory.cpp:451] Started listening for OOM events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.578721  8901 memory.cpp:562] Started listening on 'low' memory pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.579267  8901 memory.cpp:562] Started listening on 'medium' memory pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.579759  8901 memory.cpp:562] Started listening on 'critical' memory pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.581714  8901 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 160MB for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.582581  8901 memory.cpp:251] Updated 'memory.limit_in_bytes' to 160MB for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.583048  8901 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2) for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.585544  8905 linux_launcher.cpp:429] Launching container 1248ce41-4143-4adb-98d0-04c6cf8ff000 and cloning with namespaces CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWNET
I0530 10:49:37.587668  8905 systemd.cpp:96] Assigned child process '9101' to 'mesos_executors.slice'
I0530 10:49:37.589622  8903 containerizer.cpp:1637] Checkpointing container's forked pid 9101 to '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000/pids/forked.pid'
I0530 10:49:37.590232  8904 cni.cpp:872] Bind mounted '/proc/9101/ns/net' to '/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000/ns' for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:38.826182  8907 http.cpp:307] HTTP GET for /slave(1)/state from 10.114.99.73:54369 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4'
I0530 10:49:39.358474  8905 cni.cpp:1468] Unmounted the network namespace handle '/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1/ns' for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:39.358549  8905 cni.cpp:1479] Removed the container directory '/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1'
I0530 10:49:39.378610  8901 provisioner.cpp:570] Destroying container rootfs at '/var/lib/mesos/provisioner/containers/424fd077-caf5-4b67-90b0-760a23775ec1/backends/overlay/rootfses/ea0f760c-5079-4893-a5ec-469af9715abd' for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:39.380601  8907 slave.cpp:4769] Executor '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
I0530 10:49:39.380643  8907 slave.cpp:4869] Cleaning up executor '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.73:38212
I0530 10:49:39.380894  8906 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1' for gc 6.99999559317037days in the future
I0530 10:49:39.380941  8906 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for gc 6.99999559270815days in the future
I0530 10:49:39.380957  8906 gc.cpp:55] Scheduling '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1' for gc 6.99999559248889days in the future
I0530 10:49:39.380970  8906 gc.cpp:55] Scheduling '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for gc 6.99999559230222days in the future
I0530 10:49:39.961740  8904 cni.cpp:1290] Got assigned IPv4 address '192.168.235.74/32' from CNI network 'calico-net-1' for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:39.961976  8904 cni.cpp:999] Unable to find DNS nameservers for container 1248ce41-4143-4adb-98d0-04c6cf8ff000. Using host '/etc/resolv.conf'
I0530 10:49:40.125231  8902 slave.cpp:3385] Got registration for executor '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.74:43999
I0530 10:49:40.126565  8902 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 160MB for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.127490  8902 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2) for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.127542  8902 disk.cpp:207] Updating the disk resources for container 1248ce41-4143-4adb-98d0-04c6cf8ff000 to cpus(*)(allocated: *):0.2; mem(*)(allocated: *):160
I0530 10:49:40.127712  8902 slave.cpp:2331] Sending queued task '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' to executor '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.74:43999
I0530 10:49:40.134186  8902 slave.cpp:3816] Handling status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.74:43999
I0530 10:49:40.134565  8902 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.134727  8902 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.134851  8902 slave.cpp:4256] Forwarding the update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:40.134941  8902 slave.cpp:4166] Sending acknowledgement for status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.74:43999
I0530 10:49:40.143187  8908 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.143262  8908 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.223562  8908 slave.cpp:3816] Handling status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.74:43999
I0530 10:49:40.224624  8908 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 32MB for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.225495  8903 disk.cpp:207] Updating the disk resources for container 1248ce41-4143-4adb-98d0-04c6cf8ff000 to cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32
I0530 10:49:40.225520  8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1) for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.225795  8908 status_update_manager.cpp:323] Received status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.225888  8908 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.226018  8902 slave.cpp:4256] Forwarding the update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:40.226104  8902 slave.cpp:4166] Sending acknowledgement for status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.74:43999
I0530 10:49:40.233405  8902 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.233520  8902 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:41.225358  8903 slave.cpp:4388] Got exited event for executor(1)@192.168.235.74:43999
I0530 10:49:41.266679  8903 containerizer.cpp:2479] Container 1248ce41-4143-4adb-98d0-04c6cf8ff000 has exited
I0530 10:49:41.266719  8903 containerizer.cpp:2116] Destroying container 1248ce41-4143-4adb-98d0-04c6cf8ff000 in RUNNING state
I0530 10:49:41.266774  8903 linux_launcher.cpp:505] Asked to destroy container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.267336  8903 linux_launcher.cpp:548] Using freezer to destroy cgroup mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.268402  8903 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.269942  8903 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000 after 1.48992ms
I0530 10:49:41.271653  8902 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.273180  8903 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000 after 1.490944ms
I0530 10:49:42.395020  8908 slave.cpp:1625] Got assigned task '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.395308  8908 slave.cpp:1785] Launching task '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.396659  8908 paths.cpp:547] Trying to chown '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8' to user 'root'
I0530 10:49:42.406666  8908 slave.cpp:6480] Launching executor '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32 in work directory '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
I0530 10:49:42.407295  8908 slave.cpp:2118] Queued task '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for executor '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.407347  8908 docker.cpp:1104] Skipping non-docker container
I0530 10:49:42.407714  8908 containerizer.cpp:992] Starting container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 for executor '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.576038  8903 provisioner.cpp:453] Provisioning image rootfs '/var/lib/mesos/provisioner/containers/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/backends/overlay/rootfses/843753d1-28f7-4bf1-bbe5-84463f30fa2b' for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 using overlay backend
I0530 10:49:42.588337  8907 memory.cpp:451] Started listening for OOM events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.588832  8907 memory.cpp:562] Started listening on 'low' memory pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.589339  8907 memory.cpp:562] Started listening on 'medium' memory pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.589799  8907 memory.cpp:562] Started listening on 'critical' memory pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.591732  8907 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 160MB for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.593526  8907 memory.cpp:251] Updated 'memory.limit_in_bytes' to 160MB for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.593991  8907 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2) for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.596405  8907 linux_launcher.cpp:429] Launching container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 and cloning with namespaces CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWNET
I0530 10:49:42.598201  8907 systemd.cpp:96] Assigned child process '9206' to 'mesos_executors.slice'
I0530 10:49:42.600131  8903 containerizer.cpp:1637] Checkpointing container's forked pid 9206 to '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/pids/forked.pid'
I0530 10:49:42.600713  8901 cni.cpp:872] Bind mounted '/proc/9206/ns/net' to '/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/ns' for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.974720  8905 cni.cpp:1468] Unmounted the network namespace handle '/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000/ns' for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:42.974813  8905 cni.cpp:1479] Removed the container directory '/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000'
I0530 10:49:42.987810  8902 provisioner.cpp:570] Destroying container rootfs at '/var/lib/mesos/provisioner/containers/1248ce41-4143-4adb-98d0-04c6cf8ff000/backends/overlay/rootfses/585e6e95-cd8b-4ed9-8341-49360770675b' for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:42.989212  8908 slave.cpp:4769] Executor '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
I0530 10:49:42.989243  8908 slave.cpp:4869] Cleaning up executor '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.74:43999
I0530 10:49:42.989409  8908 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000' for gc 6.99998854937482days in the future
I0530 10:49:42.989439  8908 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for gc 6.9999885490163days in the future
I0530 10:49:42.989454  8908 gc.cpp:55] Scheduling '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000' for gc 6.99998854880889days in the future
I0530 10:49:42.989467  8908 gc.cpp:55] Scheduling '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for gc 6.99998854864593days in the future
I0530 10:49:45.888653  8906 cni.cpp:1290] Got assigned IPv4 address '192.168.235.75/32' from CNI network 'calico-net-1' for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:45.888829  8906 cni.cpp:999] Unable to find DNS nameservers for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8. Using host '/etc/resolv.conf'
I0530 10:49:46.050271  8902 slave.cpp:3385] Got registration for executor '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.75:40717
I0530 10:49:46.051580  8902 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 160MB for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.052515  8902 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2) for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.052567  8902 disk.cpp:207] Updating the disk resources for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 to cpus(*)(allocated: *):0.2; mem(*)(allocated: *):160
I0530 10:49:46.052742  8902 slave.cpp:2331] Sending queued task '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' to executor '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.75:40717
I0530 10:49:46.056938  8902 slave.cpp:3816] Handling status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.75:40717
I0530 10:49:46.057296  8902 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.057456  8902 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.057593  8902 slave.cpp:4256] Forwarding the update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:46.057663  8902 slave.cpp:4166] Sending acknowledgement for status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.75:40717
I0530 10:49:46.064831  8902 status_update_manager.cpp:395] Received status update acknowledgement (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.064899  8902 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.148407  8902 slave.cpp:3816] Handling status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.75:40717
I0530 10:49:46.149554  8908 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 32MB for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.150315  8901 disk.cpp:207] Updating the disk resources for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 to cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32
I0530 10:49:46.150490  8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1) for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.150804  8908 status_update_manager.cpp:323] Received status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.150835  8908 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.151010  8903 slave.cpp:4256] Forwarding the update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:46.151093  8903 slave.cpp:4166] Sending acknowledgement for status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.75:40717
I0530 10:49:46.158910  8904 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.158972  8904 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:47.151319  8903 slave.cpp:4388] Got exited event for executor(1)@192.168.235.75:40717
I0530 10:49:47.193552  8903 containerizer.cpp:2479] Container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 has exited
I0530 10:49:47.193584  8903 containerizer.cpp:2116] Destroying container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 in RUNNING state
I0530 10:49:47.193639  8903 linux_launcher.cpp:505] Asked to destroy container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.194249  8903 linux_launcher.cpp:548] Using freezer to destroy cgroup mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.195314  8903 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.196699  8904 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 after 1.355008ms
I0530 10:49:47.198119  8901 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.199431  8904 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 after 1.277952ms
I0530 10:49:48.698365  8906 cni.cpp:1468] Unmounted the network namespace handle '/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/ns' for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:48.698437  8906 cni.cpp:1479] Removed the container directory '/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
I0530 10:49:48.714375  8906 provisioner.cpp:570] Destroying container rootfs at '/var/lib/mesos/provisioner/containers/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/backends/overlay/rootfses/843753d1-28f7-4bf1-bbe5-84463f30fa2b' for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:48.715656  8906 slave.cpp:4769] Executor '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
I0530 10:49:48.715685  8906 slave.cpp:4869] Cleaning up executor '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.75:40717
I0530 10:49:48.715879  8906 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8' for gc 6.99999171553185days in the future
I0530 10:49:48.715922  8906 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for gc 6.99999171524444days in the future
I0530 10:49:48.715948  8906 gc.cpp:55] Scheduling '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8' for gc 6.99999171495407days in the future
I0530 10:49:48.715963  8906 gc.cpp:55] Scheduling '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for gc 6.99999171468741days in the future

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Alexander Rojas

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 30/May/17 13:35

Updated:: 05/Oct/17 13:37

Resolved:: 05/Oct/17 13:37