Details
-
Sub-task
-
Status: Closed
-
Minor
-
Resolution: Not A Problem
-
None
-
None
-
None
-
None
Description
Modify the scripts to install monitoring and auto-scaling rules to launch new slaves on demand according to CPU, disk, or I/O (over)use.
http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/
http://docs.amazonwebservices.com/AutoScaling/latest/DeveloperGuide/
CloudWatch metrics are provided as statistics ['Minimum', 'Maximum', 'Sum', 'Average', 'Samples'] in units of ['Seconds', 'Percent', 'Bytes', 'Bits', 'Count', 'Bytes/Second', 'Bits/Second', 'Count/Second']. Some useful metrics:
- NetworkIn
- NetworkOut
- CPUUtilization
- DiskWriteBytes
- DiskWriteOps
- DiskReadBytes
- DiskReadOps
We can be agnostic about statistics and scaling triggers and just pass them through to EC2.
This would be especially easy if we port our scripts to Python like Hadoop has on trunk and use boto: http://code.google.com/p/boto/.