I have written a crawler in Perl awhile back and it was super simple giving the high-level capability of libwww-perl. It is so straight forward in fact, it can take the raw HTML response of one request, and create the next HTTP request for you from the FORMs on that page (as in it will parse the HTML for you). Does anyone know any library like this on .NET or Java? Selenium is out of question because it requires the browser to be open, which we cannot accommodate in our implementation)