Uploaded image for project: 'Airavata'
  1. Airavata
  2. AIRAVATA-2624

Stampede2 cluster SSH connectivity issue

    XMLWordPrintableJSON

Details

    Description

      Job submission fails at env creation due to JSch initialization error.
      Error messages
      2018-01-09 09:46:10,786 [pool-7-thread-15] ERROR o.a.a.gfac.core.GFacException process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - JSch initialization error
      com.jcraft.jsch.JSchException: Auth fail
      at com.jcraft.jsch.Session.connect(Session.java:512)
      at com.jcraft.jsch.Session.connect(Session.java:183)
      at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:138)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:315)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.makeDirectory(HPCRemoteCluster.java:242)
      at org.apache.airavata.gfac.impl.task.EnvironmentSetupTask.execute(EnvironmentSetupTask.java:51)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.configureWorkspace(GFacEngineImpl.java:553)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:324)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
      at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
      at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
      at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      2018-01-09 09:46:10,786 [pool-7-thread-15] ERROR o.a.a.g.i.t.EnvironmentSetupTask process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - Error while environment setup
      org.apache.airavata.gfac.core.GFacException: JSch initialization error
      at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:545)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:138)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:315)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.makeDirectory(HPCRemoteCluster.java:242)
      at org.apache.airavata.gfac.impl.task.EnvironmentSetupTask.execute(EnvironmentSetupTask.java:51)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.configureWorkspace(GFacEngineImpl.java:553)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:324)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
      at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
      at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
      at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: com.jcraft.jsch.JSchException: Auth fail
      at com.jcraft.jsch.Session.connect(Session.java:512)
      at com.jcraft.jsch.Session.connect(Session.java:183)
      at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
      ... 14 common frames omitted
      2018-01-09 09:46:10,786 [pool-7-thread-15] INFO o.a.a.g.c.context.TaskContext process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - expId: Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, processId: PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, taskId: TASK_03ea62e9-55c0-4d9f-bfc9-af26c9c45aa0, type: ENV_SETUP : Task status changed EXECUTING -> FAILED
      2018-01-09 09:46:10,793 [pool-7-thread-15] ERROR o.a.a.gfac.impl.GFacEngineImpl process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - expId: Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, processId: PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, taskId: TASK_03ea62e9-55c0-4d9f-bfc9-af26c9c45aa0 type: ENV_SETUP,:- Input staging failed, reason: Error while environment setup
      2018-01-09 09:46:10,803 [pool-7-thread-15] ERROR o.a.a.gfac.impl.GFacWorker process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - GFac Worker throws an exception
      org.apache.airavata.gfac.core.GFacException: Error while environment setup
      at org.apache.airavata.gfac.impl.GFacEngineImpl.configureWorkspace(GFacEngineImpl.java:572)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:324)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
      at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
      at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
      at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      2018-01-09 09:46:10,803 [pool-7-thread-15] INFO o.a.a.g.c.c.ProcessContext process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - expId: Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, processId: PROCESS_650014f6-fcb6-4680-90ea-898bee373f37 :- Process status changed CONFIGURING_WORKSPACE -> FAILE

      Attachments

        Activity

          People

            marcuschristie Marcus Christie
            eroma_a Eroma
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: