WGet a Page that Requires Logging in

Posted by Synetech inc. on Super User See other posts from Super User or by Synetech inc.
Published on 2011-02-21T14:41:08Z Indexed on 2011/02/21 15:27 UTC
Read the original article Hit count: 187

Filed under:

Windows

|

wget

|

cookies

I’m trying to figure out a way to use WGET or a similar tool so that I can schedule a web page to be downloaded regularly as a sort of updating log. The problem is that the page requires that I be logged in otherwise I get a different page, generic. Further, the page does not take login information as GET parameters in the URL, it uses POST to log in on the login page and cookies to save the login information that’s read by the regular page.

I’m currently using GNU Wget 1.10.2 for Windows. I’ve tried using WGET’s cookie functionality but have had mixed results, usually skewing towards it not working.

Can anyone please advise on a way to accomplish this? Thanks a lot.

© Super User or respective owner

Related posts about Windows

Server 2012 DFS New Member Issue

as seen on Server Fault - Search for 'Server Fault'
I am trying to add a new member to our DFS topology. We have 3 DCs (VMs - VMware) running Windows server 2012, two servers are located in or Primary site and the third at our DR site. Currently the two servers at our primary site are currently replicating DFS (full mesh) and are working fine. I have… >>> More
error in IIS7 but not on IIS6

as seen on Server Fault - Search for 'Server Fault'
I have a website that is we are now deploying to windows 2008 servers that has worked in the past on IIS6 without a problem. It is using .net 2 framework. Most of the website works. Just when we create a screen report over a certain size on the server we get this error. Event code: 3005 Event… >>> More
error in IIS7 but not on IIS6

as seen on Server Fault - Search for 'Server Fault'
I have a website that is we are now deploying to windows 2008 servers that has worked in the past on IIS6 without a problem. It is using .net 2 framework. Most of the website works. Just when we create a screen report over a certain size on the server we get this error. Event code: 3005 Event… >>> More
Microsoft Management Console stops working when I add snap-in to it

as seen on Super User - Search for 'Super User'
I have Windows 7 Ultimate OS. I'm opening mmc.exe as administrator and trying add Certificates or any other snap-in, then while loading that snap-in MMC breaks and displays following message and after that it closes automatically once I click on close button on that message. What could be the problem… >>> More
Handling keyboard and mouse input (Win API)

as seen on Game Development - Search for 'Game Development'
There is a number of ways to catch mouse or keyboard under Windows. So I tried some of them, but every of them has some advantages and drawbacks. I want to ask you: Which method do use? I've tried these: WM_KEYDOWN/WM_KEYUP - Main disadvantage is that, I can't distinguish between left and right-handed… >>> More

Related posts about wget

Make wget not download files larger than X size

as seen on Super User - Search for 'Super User'
Okay, I give up. How do I size limit which files are downloaded, like say I don't want any files bigger than 2 MB? >>> More
How to start using Wget?

as seen on Super User - Search for 'Super User'
Please, forgive me for asking this question. Usually I would try to learn thisngs myself first before bothering others, but my situation is urgent - if I don't act now and don't download all my family pictures from this website, it will be closed in about two weeks from now and I will loose all of… >>> More
wget mirroring, subdomains and directories and cookies

as seen on Server Fault - Search for 'Server Fault'
Hi all, I have an account on a web page that is now "full" (ie I have used up all my allocated space) and I would like to make a mirror of that site. wget seems like the thing to use. The problem is that I would only like to mirror the sites the lie within this directory http://user.domain.com/room/2324343/transcript/… >>> More
How can I install things in Linux with *no yum* and *no wget*?

as seen on Super User - Search for 'Super User'
I'm a newbie to Linux (that mainly uses Windows and Mac OS X) needing some advice. I was trying to install git on a Linux machine today, and encountered some problems: Not knowing the version of the installed OS, I've opened the /proc/version file which said: Linux version 2.6.9-42.0.2.ELsmp (bhcompile@ls20-bc1-13… >>> More
Getting wget to dowload only files with specific name patterns

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to use wget to DL some files. I want to DL only files whose name that fit a certain pattern, e.g. ???.txt and not any other *.txt files. Can this be done with wget? I could only find a way to --accept/--reject files based on the extension. Thanks! >>> More