Website content crawling

Posted by klork on Stack Overflow See other posts from Stack Overflow or by klork
Published on 2010-05-13T19:18:57Z Indexed on 2010/05/16 9:50 UTC
Read the original article Hit count: 273

Filed under:

website

|

content

|

crawling

|

bot

We have a Business Listings directory hosted on IIS 6 Windows 2003. Our competitors crawl and steal our content and customers.

We have tried IP blocking using honeypot URLs and log parsing without much success. Is anyone aware of a network device or a proxy server that I can run in front of my web server to minimize this issue?

All suggestions are highly appreciated.

© Stack Overflow or respective owner

Related posts about website

Need help redirecting http://website.com/ to http://www.website.com/

as seen on Stack Overflow - Search for 'Stack Overflow'
What I'm trying to do is to redirect my website visitors who enter "website.com" to "www.website.com". I would do this with a standard redirect, but I don't know how to make a site specific to WWW or non-WWW addresses. I see that Firefox thinks my site is clearly different at the WWW version, because… >>> More
Official BETA release of Developer IT

as seen on Developer IT - Search for 'Developer IT'
We finally did it It’s been a week since our first online publish and our indexer robot is going as well as the website. We already have reach more than 20,000 articles and it’s only the begining. Stay tune on http://www.developerit.com/ >>> More
Get Visitors to Your Website Using Website Building Tips

as seen on Ezine Articles - Search for 'Ezine Articles'
Before building a website there are many things that you need to take in mind. For example, how are you going to design your website, what is it going to cost you, how long will it take you to build, etc. All of these things mentioned are important aspects to consider when designing a website, but… >>> More
Website Optimization - Requirement of Every Website

as seen on Ezine Articles - Search for 'Ezine Articles'
Website optimization is now days a must for each and every website to be on the top at search engines result pages. Also it has become a vital method to drive traffic to the website. >>> More
Website Basics - Planning Your Website

as seen on Ezine Articles - Search for 'Ezine Articles'
A website begins with an idea. You probably have an idea for a site and that is the reason that you are exploring your options more. Or you may be looking to getting into Internet Marketing and need to know what is involved in getting a site up and running. >>> More

Related posts about content

Help with Collapse and Expand Accordion at same time using JQuery (demo)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm stuck on an Expand/Collapse accordion using JQuery. After the initial headline is clicked and it expands, if you click to another headline it will collapse the former headline completely FIRST then it will expand the headline you clicked. This collapse first then expand second technique is very… >>> More
Splitting up content with PHP

as seen on Stack Overflow - Search for 'Stack Overflow'
I have content that is given by an XML feed that does not include line breaks. Is there away using PHP that I could include line breaks to show paragraphs? Current: content content content content content content content content content content content content content content content content content… >>> More
Reset crawled content for one content source in MOSS

as seen on Server Fault - Search for 'Server Fault'
I'm using MOSS 2007 and I want to wipe out the search index and do a full re-crawl of one of my content sources. I see the link to 'Reset all crawled content' but I don't want to have to re-crawl the other content sources. Is there any way to do this? >>> More
The Minimalist Approach to Content Governance - Retire Phase

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Originally posted by John Brunswick. Good news - the Retire Phase is actually more fun than the Manage Phase. During the Retire Phase our content management team should not have to track down content creators if the Request Phase of this process was completed successfully… >>> More
jax-ws change Content-type to Content-Type because server is hyper sensitive

as seen on Stack Overflow - Search for 'Stack Overflow'
I have to connect to a poorly implemented server that only understands Content-Type(capital-T) and not Content-type. How can I ask my jax-ws client to send Content-Type? I've tried Map<String, List<String>> headers = (Map<String, List<String>>) ((BindingProvider)port).getRequestContext()… >>> More