Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
0.7.1
-
None
-
None
Description
When starting an hbase cluster with whirr, I encountered the following problem:
This is what instance templates looks like:
whirr.instance-templates=1 zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,3 hadoop-datanode+hadoop-tasktracker+hbase-regionserver
Despite the failure to start the cluster, I noticed that whirr tried to start more instances than I specified.
I took a loot at aws console, indeed 6(1+3+2) instances were started.
Moreover, whirr wrongly report that the cluster failed to start and destroyed the cluster.
whirr-0.7.1 ren$ bin/whirr launch-cluster --config recipes/exalt-hbase-ec2.properties
Bootstrapping cluster
Configuring template
Configuring template
Starting 1 node(s) with roles [zookeeper, hadoop-namenode, hadoop-jobtracker, hbase-master]
Starting 3 node(s) with roles [hadoop-datanode, hadoop-tasktracker, hbase-regionserver]
Starting 2 node(s) with roles [hadoop-datanode, hadoop-tasktracker, hbase-regionserver]
Nodes started: [[id=us-east-1/i-a4efefc0, providerId=i-a4efefc0, group=exalt-hbase, name=exalt-hbase-a4efefc0, location=[id=us-east-1c, scope=ZONE, description=us-east-1c, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-ab36fbc2, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=false, description=099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20110930], state=RUNNING, loginPort=22, hostname=domU-12-31-39-0B-F6-07, privateAddresses=[10.214.249.241], publicAddresses=[174.129.105.39], hardware=[id=m1.small, providerId=m1.small, name=null, processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null, type=LOCAL, size=150.0, device=/dev/sda2, durable=false, isBootDevice=false], [id=vol-63cbc00f, type=SAN, size=null, device=/dev/sda1, durable=true, isBootDevice=true]], supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,Not(is64Bit())), tags=[]], loginUser=ubuntu, userMetadata=
, tags=[]]]
Destroying failed nodes [us-east-1/i-daefefbe, us-east-1/i-c4f5f5a0, us-east-1/i-a6efefc2]
Destroyed failed nodes [us-east-1/i-daefefbe, us-east-1/i-c4f5f5a0, us-east-1/i-a6efefc2]
Unable to start the cluster. Terminating all nodes.
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:129)
at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
at org.apache.whirr.cli.Main.run(Main.java:64)
at org.apache.whirr.cli.Main.main(Main.java:97)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:124)
... 5 more
Caused by: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:92)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:40)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Unable to load cluster state, assuming it has no running nodes.
java.io.FileNotFoundException: /Users/ren/.whirr/exalt-hbase/instances (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at com.google.common.io.Files$1.getInput(Files.java:100)
at com.google.common.io.Files$1.getInput(Files.java:97)
at com.google.common.io.CharStreams$2.getInput(CharStreams.java:91)
at com.google.common.io.CharStreams$2.getInput(CharStreams.java:88)
at com.google.common.io.CharStreams.readLines(CharStreams.java:306)
at com.google.common.io.Files.readLines(Files.java:580)
at org.apache.whirr.state.FileClusterStateStore.load(FileClusterStateStore.java:54)
at org.apache.whirr.state.ClusterStateStore.tryLoadOrEmpty(ClusterStateStore.java:58)
at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:143)
at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:118)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
at org.apache.whirr.cli.Main.run(Main.java:64)
at org.apache.whirr.cli.Main.main(Main.java:97)
Starting to run scripts on cluster for phase destroyinstances:
Starting to run scripts on cluster for phase destroyinstances:
Finished running destroy phase scripts on all cluster instances
Destroying exalt-hbase cluster
Cluster exalt-hbase destroyed
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:125)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
at org.apache.whirr.cli.Main.run(Main.java:64)
at org.apache.whirr.cli.Main.main(Main.java:97)
Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:129)
at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
... 3 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:124)
... 5 more
Caused by: java.io.IOException: Too many instance failed while bootstrapping! 2 successfully started instances while 3 instances failed
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:92)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:40)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)