How much HDD space would I need to cache the web while respecting robot.txts?

Posted by Koning Baard XIV on Server Fault See other posts from Server Fault or by Koning Baard XIV
Published on 2010-06-05T12:56:09Z Indexed on 2010/06/05 13:02 UTC
Read the original article Hit count: 289

Filed under:

web

|

caching

|

robots.txt

|

crawler

I want to experiment with creating a web crawler. I'll start with indexing a few medium sized website like Stack Overflow or Smashing Magazine. If it works, I'd like to start crawling the entire web. I'll respect robot.txts. I save all html, pdf, word, excel, powerpoint, keynote, etc... documents (not exes, dmgs etc, just documents) in a MySQL DB. Next to that, I'll have a second table containing all restults and descriptions, and a table with words and on what page to find those words (aka an index).

How much HDD space do you think I need to save all the pages? Is it as low as 1 TB or is it about 10 TB, 20? Maybe 30? 1000?

Thanks

© Server Fault or respective owner

Related posts about web

Why is Java EE 6 better than Spring ?

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Java EE 6 was released over 2 years ago and now there are 14 compliant application servers. In all my talks around the world, a question that is frequently asked is Why should I use Java EE 6 instead of Spring ? There are already several blogs covering that topic: Java EE… >>> More
Hosting a website on Heroku.... I know how to, but im running into problems!

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I'm starting to learn more on the back-end scale of programing. Recently I started up Heroku for the second or third time. This time I actually installed the Git update to my Mac and installed Heroku in the terminal. I wanted to upload a static html site with the sinatra gem. Everything worked out… >>> More
Microsoft .NET Web Programming: Web Sites versus Web Applications

as seen on Samir ASP.NET with C# Technology - Search for 'Samir ASP.NET with C# Technology'
In .NET 2.0, Microsoft introduced the Web Site. This was the default way to create a web Project in Visual Studio 2005. In Visual Studio 2008, the Web Application has been restored as the default web Project in Visual Studio/.NET 3.x The Web Site is a file/folder based Project structure. It… >>> More
VS2008 - Unable to Add Web Reference to Web Application Project (The web services enumeration compon

as seen on Stack Overflow - Search for 'Stack Overflow'
I've run into a situation where I was unable to add a Web Reference in Visual Studio 2008 to a Web Application Project. The error I couldn't resolve was "The web services enumeration components are not available. You need to reinstall Visual Studio to add web references to your application." How… >>> More
Outlook Web Access: "Outlook Web Access has encountered a Web browsing error"

as seen on Super User - Search for 'Super User'
When one of my colleagues is accessing Outlook Web Access from IE, he frequently gets an error reported: "Outlook Web Access has encountered a Web browsing error". The error report includes the following: Client Information User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4… >>> More

Related posts about caching

Windows Azure Evolution – Caching (Preview)

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Caching is a popular topic when we are building a high performance and high scalable system not only on top of the cloud platform but the on-premise environment as well. On March 2011 the Windows Azure AppFabric Caching had been production launched. It provides an in-memory, distributed caching service… >>> More
Disable eclipselink caching and query caching - not working?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am using eclipselink JPA with a database which is also being updated externally to my application. For that reason there are tables I want to query every few seconds. I can't get this to work even when I try to disable the cache and query cache. For example: EntityManagerFactory entityManagerFactory… >>> More
apache's caching (mod_cache) vs squid reverse-proxy caching?

as seen on Server Fault - Search for 'Server Fault'
simple question about an area im not familiar with. which one is best for caching? are there other options? cause i want to be sure which one to use before i learn it. so i dont have to realize afterwards that it was a bad choice. so would be great if someone could shed a light on this topic. >>> More
.htaccess for compression, browser caching, proxy caching, etc.

as seen on Stack Overflow - Search for 'Stack Overflow'
Can someone provide me with an optimize .htaccess configuration that handles compression, browser caching, proxy caching, etc. for a typical website? Aside from my visitors, I'm also trying to make Google PageSpeed happy. >>> More
Caching And Queing while caching is being populated?

as seen on Stack Overflow - Search for 'Stack Overflow'
whats the correct way to cache a object(DataSet) and then get it and when it expires repopulate it with out a major hiccup in the application... the application relies on this data to be there for read only purposes. >>> More