Getting all pdf files from a domain (for example *.adomain.com)

Posted by Zack on Stack Overflow See other posts from Stack Overflow or by Zack
Published on 2010-06-05T09:02:04Z Indexed on 2010/06/05 9:12 UTC
Read the original article Hit count: 208

Filed under:

pdf

|

google

|

multiple

|

download

|

scraper

I need to download all pdf files from a certain domain. There are about 6000 pdf on that domain and most of them don't have an html link (either they have removed the link or they never put one in the first place).

I know there are about 6000 files because I'm googling: filetype:pdf site:*.adomain.com

However, Google lists only the first 1000 results. I believe there are two ways to achieve this:

a) Use Google. However, how I can get all 6000 results from Google? Maybe a scraper? (tried scroogle, no luck) b) Skip Google and search directly on domain for pdf files. How do I do that when most them are not linked?

© Stack Overflow or respective owner

Related posts about pdf

PDF Converter wanted: Convert 8.5*11 PDF images into 600*800px PDF images for the Nook

as seen on Super User - Search for 'Super User'
I have PDF files that are maritime charts, For example this one from the Delaware Bay http://ocsdata.ncd.noaa.gov/BookletChart/12304_BookletChart_HomeEd.pdf There is a lot of detailed information in the image. When I show them on a monitor the details are shown. When I put them on the Nook they… >>> More
Loop through values and display in a pdf file

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all! i have written the following code: As you can see there is a for loop to go through some values and display them in the generated pdf. The problem is that all the values are being written at the same place. I have tried to insert a new line but it does not seem to work. Can anyone suggest… >>> More
convert scanned images pdf file to searchable pdf file

as seen on Super User - Search for 'Super User'
I have a pdf of a scanned book. I'm looking for a free software that will perform ocr and then provide an option to save it as pdf/doc. Is there one? Thanks. >>> More
Integrate Nitro PDF Reader with Windows 7

as seen on How to geek - Search for 'How to geek'
Would you like a lightweight PDF reader that integrates nicely with Office and Windows 7? Here we look at the new Nitro PDF Reader, a nice PDF viewer that also lets you create and markup PDF files. Adobe Reader is the de-facto PDF viewer, but it only lets you view PDFs and not much else. … >>> More
foxpro to pdf and pdf to foxpro

as seen on Stack Overflow - Search for 'Stack Overflow'
how will i convert pdf database converted from foxpro back to foxpro >>> More

Related posts about google

Removing malware of a particular kind

as seen on Super User - Search for 'Super User'
I need to remove some malware from my computer. It is a trojan, and very annoying. It blocks access to Google and search sites. The trojan, with its name spelled out on each line cause it seems to block sites when i reference it in a url, is a r t (some text to mess it up) e m (more text i s First… >>> More
Trouble installing Matlab

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
So whenever I try to install Matlab, the first image appears, but then goes away and that's the end of it. Here's what I'm doing- user@host$> cd ~/mathworks_downloads user@host$> unzip matlab_R2012a_student_glnx86_installer.zip user@host$> ./install The Matlab logo and image pop up here… >>> More
Google chrome is always searching in local google domain instead of Google.com

as seen on Super User - Search for 'Super User'
I have changed in the searched preferences to google.com but still when I do search from the address bar (instant or non-instant) it will go to google.co.kr. Even though I change "Google.com in English", still same... The only way is to open google.com website first, then do search in it. So the… >>> More
Google I/O 2010: Google TV Keynote - Introducing Google TV

as seen on Google Code - Search for 'Google Code'
Google I/O 2010: Google TV Keynote - Introducing Google TV Due to licensing and permissions issues, we are unable to show the full Google TV demonstration from the Day 2 keynote at Google I/O. Until we are able to get these permissions, please check out these clips. For Google I/O session videos… >>> More
Google I/O 2010: Google TV Keynote - Android Apps On Google TV

as seen on Google Code - Search for 'Google Code'
Google I/O 2010: Google TV Keynote - Android Apps On Google TV Due to licensing and permissions issues, we are unable to show the full Google TV demonstration from the Day 2 keynote at Google I/O. Until we are able to get these permissions, please check out these clips. For Google I/O session videos… >>> More