How to identify the client is a search robot?
- by Yau Leung
I have built my entire site using AJAX (indeed it's GWT). I have also implemented AJAX crawling proposed by Google. However, after the implementation, I found that neither Yahoo , Bing, nor Baidu implemented that scheme!
I'm wondering if there is a way to identify the web client is a search robot. If they are, they will be shown the HTML snapshot I created.
It will be best if I can identify them in APACHE level, then I can just do a mod_rewrite. But it's still ok if I can do that in PHP or GWT.