Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.2.1
-
None
-
None
Description
The FileDumper tool is a tool that reads the crawled data from Nutch and dumps this data into its raw files. This tool currently dumps every single file irrespective of status, duplicates etc. This cause files that are fetched in error or files that have not been fetched because they were made unavailable by the server to also be dumped.
The fix should be to fetch only those files that were fetched with status db_fetched by Nutch.