Screen-scraping a site with a asp.net form login in C#?

Posted by Ajit on Stack Overflow See other posts from Stack Overflow or by Ajit
Published on 2010-03-18T13:35:55Z Indexed on 2010/03/18 13:41 UTC
Read the original article Hit count: 377

Filed under:

Site

|

scraping

|

simulate

|

click

Hi Friends,

i've created a web application in asp.net so far. where i've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but i haven't accessed the secure page yet. Every time the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.

Please advice what should i do ?

© Stack Overflow or respective owner

Related posts about Site

Outside VPN traffic not able to ping site-to-site VPN remote site

as seen on Server Fault - Search for 'Server Fault'
we have two ASA 5510s running 8.4 in a site-to-site VPN setup. All internal traffic is working smoothly. Site/Subnet A: 192.100.0.0 - local Site/Subnet B: 192.200.0.0 - remote VPN Users: 192.100.40.0 - assigned by ASA When you VPN into the network, all traffic hits Site A, and everything on subnet… >>> More
Site-to-site VPN using MD5 instead of SHA and getting regular disconnection

as seen on Server Fault - Search for 'Server Fault'
We are experiencing some strange behavior with a site-to-site IPsec VPN that goes down about every week for 30 minutes (Iam told 30 minutes exactly). I don't have access to the logs, so it's difficult to troubleshoot. What is also strange is that the two VPN devices are set to use SHA hash algorithm… >>> More
I have a very long and repetitive python path, where do I look to correct this?

as seen on Stack Overflow - Search for 'Stack Overflow'
I know it is probably not necessary to paste the whole path, but just for the record I have done so below. Whenever I run a python command, it takes a long time to load this path I suppose. I have checked in .bash_profile and only have these two lines: export PATH=/Users/username/bin:/opt/local/Library/Frameworks/Python… >>> More
WSS 3.0 Backup/Restore Root Site Collection to Sub-Site of New Site Collection

as seen on Server Fault - Search for 'Server Fault'
Our intranet was originally setup to be at the root of its site collection. We are trying to change this so that our new internet site will live in the root and the intranet will be a sub-site. At this point I have created a new web application and site collection to house the internet and intranet… >>> More
Creating a Membership Site - 4 Reasons to Build a Membership Site

as seen on Ezine Articles - Search for 'Ezine Articles'
If you really want to cash in on one of the hottest trends online, you might want to think about creating a membership site. Membership sites are an excellent way to leverage your time, resources and clients while generating monthly cash deposits into your bank account on autopilot. >>> More

Related posts about scraping

Screen-scraping of a secure page of any site on https:// with asp.net in C#

as seen on Stack Overflow - Search for 'Stack Overflow'
I've done site scraping of secure page of any site on http:// but when I am trying to scrap any site on https:// then i always scrape the login page not secure page. Please advice what should i do for scraping a secure page of any site on https://. >>> More
looking for alternative to Webzinc .NET , screen scraping, web automation library for .net

as seen on Stack Overflow - Search for 'Stack Overflow'
i came across this .net library http://www.webzinc.com/online/faq.aspx however, i was wondering if there was a free alternative out there ? >>> More
PHP Screen Scraping Class

as seen on Bradino - Search for 'Bradino'
After some positive feedback I have decided to continue to develop the PHP Screen Scraping class. This post will server as the permanent home for the class. Download PHP Screen Scraping Class Updates 20009-07-30 Added setHeader() function >>> More
Alert Log Scraping with Oracle&#146;s ADRCI Utility

as seen on Internet.com - Search for 'Internet.com'
Oracles new ADR with command interface shows promise for database administrators who like to script their own solution for quickly scraping the alert log and automatically looking for errors. >>> More
Web scraping etiquette

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm considering writing a simple web scraping application to extract information from a website that does not seem to specifically prohibit this. I've checked for other alternatives (eg RSS, web service) to get this information, but there are none available at this stage. Despite this I've also… >>> More