Uploaded image for project: 'Apache Whirr (retired)'
  1. Apache Whirr (retired)
  2. WHIRR-576

Starting more instances than specified (and more).

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 0.7.1
    • None
    • core, service/hbase
    • None

    Description

      When starting an hbase cluster with whirr, I encountered the following problem:
      This is what instance templates looks like:
      whirr.instance-templates=1 zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,3 hadoop-datanode+hadoop-tasktracker+hbase-regionserver

      Despite the failure to start the cluster, I noticed that whirr tried to start more instances than I specified.

      I took a loot at aws console, indeed 6(1+3+2) instances were started.

      Moreover, whirr wrongly report that the cluster failed to start and destroyed the cluster.

      whirr-0.7.1 ren$ bin/whirr launch-cluster --config recipes/exalt-hbase-ec2.properties
      Bootstrapping cluster
      Configuring template
      Configuring template
      Starting 1 node(s) with roles [zookeeper, hadoop-namenode, hadoop-jobtracker, hbase-master]
      Starting 3 node(s) with roles [hadoop-datanode, hadoop-tasktracker, hbase-regionserver]
      Starting 2 node(s) with roles [hadoop-datanode, hadoop-tasktracker, hbase-regionserver]
      Nodes started: [[id=us-east-1/i-a4efefc0, providerId=i-a4efefc0, group=exalt-hbase, name=exalt-hbase-a4efefc0, location=[id=us-east-1c, scope=ZONE, description=us-east-1c, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-ab36fbc2, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=false, description=099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20110930], state=RUNNING, loginPort=22, hostname=domU-12-31-39-0B-F6-07, privateAddresses=[10.214.249.241], publicAddresses=[174.129.105.39], hardware=[id=m1.small, providerId=m1.small, name=null, processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null, type=LOCAL, size=150.0, device=/dev/sda2, durable=false, isBootDevice=false], [id=vol-63cbc00f, type=SAN, size=null, device=/dev/sda1, durable=true, isBootDevice=true]], supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,Not(is64Bit())), tags=[]], loginUser=ubuntu, userMetadata=

      {Name=exalt-hbase-a4efefc0}

      , tags=[]]]
      Destroying failed nodes [us-east-1/i-daefefbe, us-east-1/i-c4f5f5a0, us-east-1/i-a6efefc2]
      Destroyed failed nodes [us-east-1/i-daefefbe, us-east-1/i-c4f5f5a0, us-east-1/i-a6efefc2]
      Unable to start the cluster. Terminating all nodes.
      java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
      at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:129)
      at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
      at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
      at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
      at org.apache.whirr.cli.Main.run(Main.java:64)
      at org.apache.whirr.cli.Main.main(Main.java:97)
      Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
      at java.util.concurrent.FutureTask.get(FutureTask.java:83)
      at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:124)
      ... 5 more
      Caused by: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
      at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:92)
      at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:40)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:680)
      Unable to load cluster state, assuming it has no running nodes.
      java.io.FileNotFoundException: /Users/ren/.whirr/exalt-hbase/instances (No such file or directory)
      at java.io.FileInputStream.open(Native Method)
      at java.io.FileInputStream.<init>(FileInputStream.java:120)
      at com.google.common.io.Files$1.getInput(Files.java:100)
      at com.google.common.io.Files$1.getInput(Files.java:97)
      at com.google.common.io.CharStreams$2.getInput(CharStreams.java:91)
      at com.google.common.io.CharStreams$2.getInput(CharStreams.java:88)
      at com.google.common.io.CharStreams.readLines(CharStreams.java:306)
      at com.google.common.io.Files.readLines(Files.java:580)
      at org.apache.whirr.state.FileClusterStateStore.load(FileClusterStateStore.java:54)
      at org.apache.whirr.state.ClusterStateStore.tryLoadOrEmpty(ClusterStateStore.java:58)
      at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:143)
      at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:118)
      at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
      at org.apache.whirr.cli.Main.run(Main.java:64)
      at org.apache.whirr.cli.Main.main(Main.java:97)
      Starting to run scripts on cluster for phase destroyinstances:
      Starting to run scripts on cluster for phase destroyinstances:
      Finished running destroy phase scripts on all cluster instances
      Destroying exalt-hbase cluster
      Cluster exalt-hbase destroyed
      Exception in thread "main" java.lang.RuntimeException: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
      at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:125)
      at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
      at org.apache.whirr.cli.Main.run(Main.java:64)
      at org.apache.whirr.cli.Main.main(Main.java:97)
      Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
      at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:129)
      at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
      at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
      ... 3 more
      Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
      at java.util.concurrent.FutureTask.get(FutureTask.java:83)
      at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:124)
      ... 5 more
      Caused by: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
      at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:92)
      at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:40)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:680)

      Attachments

        1. exalt-hbase-ec2.properties
          3 kB
          Zuocheng Ren

        Activity

          People

            Unassigned Unassigned
            awakenrz Zuocheng Ren
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: