Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • new service
    • None

    Description

      Add a new service for Flume.

      Attachments

        1. WHIRR-218.patch
          20 kB
          Thomas White

        Issue Links

          Activity

            tomwhite Thomas White added a comment -

            Here's a first attempt at this. I can start a flume master and node using the following config:

            whirr.cluster-name=flume
            whirr.instance-templates=1 flume-master,1 flume-node
            whirr.provider=ec2
            whirr.identity=${env:AWS_ACCESS_KEY_ID}
            whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
            whirr.client-cidrs=0.0.0.0/0
            

            This still needs an integration test. Also, the configuration needs some work - I haven't looked at it.

            tomwhite Thomas White added a comment - Here's a first attempt at this. I can start a flume master and node using the following config: whirr.cluster-name=flume whirr.instance-templates=1 flume-master,1 flume-node whirr.provider=ec2 whirr.identity=${env:AWS_ACCESS_KEY_ID} whirr.credential=${env:AWS_SECRET_ACCESS_KEY} whirr.client-cidrs=0.0.0.0/0 This still needs an integration test. Also, the configuration needs some work - I haven't looked at it.
            savu.andrei Andrei Savu added a comment -

            I think this could be really useful if we can also deploy a hdfs / s3 sink.

            http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/

            savu.andrei Andrei Savu added a comment - I think this could be really useful if we can also deploy a hdfs / s3 sink. http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/
            russell.jurney Russell Jurney added a comment -

            I agree with Andrei. A whirr to use flume to log events to s3 in avro would be useful to many people.

            russell.jurney Russell Jurney added a comment - I agree with Andrei. A whirr to use flume to log events to s3 in avro would be useful to many people.

            People

              Unassigned Unassigned
              tomwhite Thomas White
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: