Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-7183

Always get coredump by add a health check on docker container app

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Resolved
    • 1.1.1, 1.2.0
    • None
    • None
    • None

    Description

      the key message is : Failed to enter the net namespace of task (pid: '22392'): Pid 22392 does not exist

      see the sandbox's stderr log:

      I0227 09:20:02.624827 22345 exec.cpp:162] Version: 1.1.0
      I0227 09:20:02.651790 22347 exec.cpp:237] Executor registered on agent f2aeab4d-b224-479c-869d-121daa0c12cb-S0
      I0227 09:20:02.656651 22347 docker.cpp:811] Running docker -H unix:///var/run/docker.sock run --privileged --cpu-shares 2048 --memory 33554432 -e WORDPRESS_DB_HOST=192.168.1.210 -e WORDPRESS_DB_PASSWORD=root -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-f2aeab4d-b224-479c-869d-121daa0c12cb-S0.8f6de3ab-0e85-434a-a099-d16f9654a10c -v /home:/data:rw -v /var/lib/mesos/slaves/f2aeab4d-b224-479c-869d-121daa0c12cb-S0/frameworks/67b3106e-fe2b-4eaa-8dcc-51653d027738-0001/executors/0-wordpress4-nmg-nmgtest-55ba456bf6eb4e979610f5ec1fb23980/runs/8f6de3ab-0e85-434a-a099-d16f9654a10c:/mnt/mesos/sandbox --net bridge --label=APP_ID=wordpress --label=USER=nmg --label=CLUSTER=nmgtest --label=SLOT=0 --label=APP=wordpress4 -p 31000:8080/tcp --name mesos-f2aeab4d-b224-479c-869d-121daa0c12cb-S0.8f6de3ab-0e85-434a-a099-d16f9654a10c wordpress
      WordPress not found in /var/www/html - copying now...
      Complete! WordPress has been successfully copied to /var/www/html
      
      MySQL Connection Error: (2002) Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      W0227 09:20:18.425110 22353 health_checker.cpp:205] Health check failed 1 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:19.535784 22347 health_checker.cpp:205] Health check failed 2 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      W0227 09:20:20.646812 22350 health_checker.cpp:205] Health check failed 3 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:21.758222 22353 health_checker.cpp:205] Health check failed 4 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:22.773813 22349 health_checker.cpp:205] Health check failed 5 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      W0227 09:20:23.883586 22352 health_checker.cpp:205] Health check failed 6 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:24.994628 22350 health_checker.cpp:205] Health check failed 7 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:26.106149 22352 health_checker.cpp:205] Health check failed 8 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      W0227 09:20:27.218143 22351 health_checker.cpp:205] Health check failed 9 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:28.329988 22350 health_checker.cpp:205] Health check failed 10 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:29.440842 22348 health_checker.cpp:205] Health check failed 11 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      W0227 09:20:30.554321 22348 health_checker.cpp:205] Health check failed 12 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      W0227 09:20:31.664559 22347 health_checker.cpp:205] Health check failed 13 times consecutively: HTTP health check failed: curl returned exited with status 7: curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
      
      MySQL Connection Error: (2002) Connection refused
      F0227 09:20:32.666734 22601 health_checker.cpp:94] Failed to enter the net namespace of task (pid: '22392'): Pid 22392 does not exist
      *** Check failure stack trace: ***
          @     0x7f9b33f07862  google::LogMessage::Fail()
          @     0x7f9b33f077c1  google::LogMessage::SendToLog()
          @     0x7f9b33f071d2  google::LogMessage::Flush()
          @     0x7f9b33f09f06  google::LogMessageFatal::~LogMessageFatal()
          @     0x7f9b32c98b28  _ZZN5mesos8internal6health14cloneWithSetnsERKSt8functionIFivEE6OptionIiERKSt6vectorISsSaISsEEENKUlvE_clEv
          @     0x7f9b32c9fbc3  _ZNSt17_Function_handlerIFivEZN5mesos8internal6health14cloneWithSetnsERKSt8functionIS0_E6OptionIiERKSt6vectorISsSaISsEEEUlvE_E9_M_invokeERKSt9_Any_data
          @           0x4e44c0  std::function<>::operator()()
          @     0x7f9b32ca1652  process::defaultClone()
          @     0x7f9b32c98e4c  mesos::internal::health::cloneWithSetns()
          @     0x7f9b32caf2bc  _ZNSt5_BindIFPFiRKSt8functionIFivEE6OptionIiERKSt6vectorISsSaISsEEESt12_PlaceholderILi1EES6_S9_EE6__callIiIS4_EILm0ELm1ELm2EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE
          @     0x7f9b32cac55d  _ZNSt5_BindIFPFiRKSt8functionIFivEE6OptionIiERKSt6vectorISsSaISsEEESt12_PlaceholderILi1EES6_S9_EEclIJS4_EiEET0_DpOT_
          @     0x7f9b32ca8ad1  std::_Function_handler<>::_M_invoke()
          @     0x7f9b33ec4b8b  std::function<>::operator()()
          @     0x7f9b33ec3174  process::internal::cloneChild()
          @     0x7f9b33ec113e  process::subprocess()
          @     0x7f9b32c9c44c  mesos::internal::health::HealthCheckerProcess::_httpHealthCheck()
          @     0x7f9b32c9a269  mesos::internal::health::HealthCheckerProcess::_healthCheck()
          @     0x7f9b32ca6344  _ZZN7process8dispatchIN5mesos8internal6health20HealthCheckerProcessEEEvRKNS_3PIDIT_EEMS6_FvvEENKUlPNS_11ProcessBaseEE_clESD_
          @     0x7f9b32cac6fe  _ZNSt17_Function_handlerIFvPN7process11ProcessBaseEEZNS0_8dispatchIN5mesos8internal6health20HealthCheckerProcessEEEvRKNS0_3PIDIT_EEMSA_FvvEEUlS2_E_E9_M_invokeERKSt9_Any_dataS2_
          @     0x7f9b33e76211  std::function<>::operator()()
          @     0x7f9b33e58a81  process::ProcessBase::visit()
          @     0x7f9b33e60c64  process::DispatchEvent::visit()
          @           0x4d499a  process::ProcessBase::serve()
          @     0x7f9b33e54e7b  process::ProcessManager::resume()
          @     0x7f9b33e51b69  _ZZN7process14ProcessManager12init_threadsEvENKUt_clEv
          @     0x7f9b33e6040c  _ZNSt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEE9_M_invokeIIEEEvSt12_Index_tupleIIXspT_EEE
          @     0x7f9b33e60363  _ZNSt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEclEv
          @     0x7f9b33e602fc  _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
          @     0x7f9b2e7bf220  (unknown)
          @     0x7f9b2ea19dc5  start_thread
          @     0x7f9b2df2873d  __clone
      W0227 09:20:33.082443 22351 health_checker.cpp:205] Health check failed 14 times consecutively: HTTP health check failed: curl returned terminated with signal Aborted: 
      

      Attachments

        1. stderr
          7 kB
          Deshi Xiao

        Issue Links

          Activity

            People

              xds2000 Deshi Xiao
              xds2000 Deshi Xiao
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: