This still needs an integration test. Also, the configuration needs some work - I haven't looked at it.
Thomas White
added a comment - Here's a first attempt at this. I can start a flume master and node using the following config:
whirr.cluster-name=flume
whirr.instance-templates=1 flume-master,1 flume-node
whirr.provider=ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.client-cidrs=0.0.0.0/0
This still needs an integration test. Also, the configuration needs some work - I haven't looked at it.
Andrei Savu
added a comment - I think this could be really useful if we can also deploy a hdfs / s3 sink.
http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/
Here's a first attempt at this. I can start a flume master and node using the following config:
This still needs an integration test. Also, the configuration needs some work - I haven't looked at it.