Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.4, 1.11
-
None
-
None
Description
I would like to initiate something I've provisionally called BUTCH wit the aim of providing a continuous benchmarking evaluation for Nutch.
I wrote a utility script called [nipt](https://github.com/lewismc/nipt/blob/master/bootstrap.sh) which essentially pulls the top 1M URL's from Alexa, does some simple reformatting using sed and provides us with a flat file containing the top 1M URLs.
Loads of these are obviously porn (and god knows whatever else) related so I would not advise injecting this garbage into any crawldb that you own or administer.
I want to augment the [Benchmark tool](https://github.com/apache/nutch/blob/trunk/src/java/org/apache/nutch/tools/Benchmark.java) to imitate injecting the script and fetching the URLs. Essentially this could run continuously with us sending results to the dev@ list or making them available via some GUI.
The first step is for me to code this up. The second stage is for me to get Apache Infra to provide us with some nice machines (courtesy of Rackspace) which can host this for us.