Harvesting Dynamic HTTP Content to produce Replicating HTTP Static Content

Posted by Neil Pitman on Stack Overflow See other posts from Stack Overflow or by Neil Pitman
Published on 2009-07-02T17:47:42Z Indexed on 2010/06/12 17:02 UTC
Read the original article Hit count: 261

Filed under:
|
|
|
|

I have a slowly evolving dynamic website served from J2EE. The response time and load capacity of the server are inadequate for client needs. Moreover, ad hoc requests can unexpectedly affect other services running on the same application server/database. I know the reasons and can't address them in the short term. I understand HTTP caching hints (expiry, etags....) and for the purpose of this question, please assume that I have maxed out the opportunities to reduce load.

I am thinking of doing a brute force traversal of all URLs in the system to prime a cache and then copying the cache contents to geodispersed cache servers near the clients. I'm thinking of Squid or Apache HTTPD mod_disk_cache. I want to prime one copy and (manually) replicate the cache contents. I don't need a federation or intelligence amongst the slaves. When the data changes, invalidating the cache, I will refresh my master cache and update the slave versions, probably once a night.

Has anyone done this? Is it a good idea? Are there other technologies that I should investigate? I can program this, but I would prefer a configuration of open source technologies solution

Thanks

© Stack Overflow or respective owner

Related posts about http

Related posts about dynamic