How to spread XML Sitemaps over several webservers behind AWS loadbalancer?
- by Jurik
We have a web portal with almost a million products and way more other urls.
I wrote a script that checks database. If there is a new url needed or an old one update, this script will update/create the XML Sitemaps.
But we have several servers behind the load balancer at our rented AWS space.
Further this script checks database for each url if there was an update so that it updates the appropriate xml file too.
My question is how to spread those XML Sitemaps over all webservers behind this AWS load balancer?
Our approaches/ideas:
we could just generate them on one server with a cron job and copy them to the other servers, but this could be difficult because of automatic raising numbers of servers and so on.
we put them on our S3 - but this one is not avaible thru our domain, so I guess google will have a problem with it
I let my script run on every webserver but change it in a way that it will generate each time all xml files if they do not exist. But then I would have conflicts with updated URLs in my database, where I saved timestamp of last changed value of every url
Is there another - better - solution that I do not know? Are there any special services by amazon for such cases?
