Link Parsing in http is flawed

The link parsing in wagon http has a few issues.

a) not all links detected.
b) the various ways that page content is identified via url string manipulation isn't working in many example cases.
c) the use of jtidy introduces a large dependency and high memory usage.