Major Description
When certain documents have the same signature de deduplication script will elect one as duplicate. The urls are stored url encoded in the crawldb. When two urls are compared by url length, the urls are not first decoded. This could lead to misleading url length.