Screen-scraping of a secure page of any site on https:// with asp.net in C#

Posted by Ajit on Stack Overflow See other posts from Stack Overflow or by Ajit
Published on 2010-03-19T09:15:35Z Indexed on 2010/03/19 9:21 UTC
Read the original article Hit count: 541

Filed under:

screen-scraping

|

https

|

ssl

I've done site scraping of secure page of any site on http:// but when I am trying to scrap any site on https:// then i always scrape the login page not secure page.

Please advice what should i do for scraping a secure page of any site on https://.

© Stack Overflow or respective owner

Related posts about screen-scraping

PHP Screen Scraping Class

as seen on Bradino - Search for 'Bradino'
After some positive feedback I have decided to continue to develop the PHP Screen Scraping class. This post will server as the permanent home for the class. Download PHP Screen Scraping Class Updates 20009-07-30 Added setHeader() function >>> More
Screen scraping over SSL with .NET

as seen on Stack Overflow - Search for 'Stack Overflow'
What solutions exist for screen scraping a site over SSL for use with .NET? My use case is that I need to login to a partner website (https), navigate through a dynamic hierarchy, and download a zipped file of reports. I certainly could use other screen scrapers if there are no good viable options… >>> More
looking for alternative to Webzinc .NET , screen scraping, web automation library for .net

as seen on Stack Overflow - Search for 'Stack Overflow'
i came across this .net library http://www.webzinc.com/online/faq.aspx however, i was wondering if there was a free alternative out there ? >>> More
Screen-scraping of a secure page of any site on https:// with asp.net in C#

as seen on Stack Overflow - Search for 'Stack Overflow'
I've done site scraping of secure page of any site on http:// but when I am trying to scrap any site on https:// then i always scrape the login page not secure page. Please advice what should i do for scraping a secure page of any site on https://. >>> More
How different is mashup from screenscraping and consuming webservices

as seen on Stack Overflow - Search for 'Stack Overflow'
From what I understand, Mashup is aggregating data from separate sources and providing a single view. How different is mashup when compared to screenscraping or using webservices to get data from external sources? >>> More

Related posts about https

Need Java https proxy which can be enhanced to emulate production https proxy behaviour

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a production environment which require access through a proxy server. Occasionally said server returns blank responses badly confusing the Metro web service library causing all kinds of interesting RuntimeExceptions. I believe the proxy is Squid. In order to handle these better, I would… >>> More
How To Find Out If You are Using HTTPS Without $_SERVER['HTTPS']

as seen on Stack Overflow - Search for 'Stack Overflow'
I've seen many tutorials online that says you need to check $_SERVER['HTTPS'] if the server is connection is secured with HTTPS. My problem is that on some of the servers I use, $_SERVER['HTTPS'] is an undefined variable that results in an error. Is there another variable I can check that should always… >>> More
Redirect request from https domain to https subdomain with only one certificate

as seen on Server Fault - Search for 'Server Fault'
I'm trying to redirect users to a subdomain in server2 if they make an https request to server1. I only have one certificate, and that's installed on server2. So for instance, from (server1) https://www.example.com to (server2) https://ssl.example.com My best guess is that I will need a certificate… >>> More
Apache: https to https redirect

as seen on Super User - Search for 'Super User'
I'm trying to get Apache to redirect all http and https traffic to a single endpoint www.example.org. The http part is easy: <VirtualHost *:80> ServerName example.org Redirect permanent / https://www.example.org/ </VirtualHost> # long list of other domains, all redirecting to… >>> More
SSL / HTTP / No Response to Curl

as seen on Server Fault - Search for 'Server Fault'
I am trying to send commands to a SOAP service, and getting nothing in reply. The SOAP service is at a completely separate site from either server I am testing with. I have written a dummy script with the SOAP XML embedded. When I run it at my local site, on any of three machines -- OSX, Ubuntu,… >>> More