How to reduce duplicate web pages when rolling my own search engine?

Posted by user198729 on Stack Overflow See other posts from Stack Overflow or by user198729
Published on 2010-04-15T14:15:58Z Indexed on 2010/04/15 14:33 UTC
Read the original article Hit count: 184

  1. One articles can be posted on many sites
  2. the same article on a specific website can be visited by several different urls.

How do I reduce duplicated stuff in the above occasions?

© Stack Overflow or respective owner

Related posts about search-engine

Related posts about reduce-duplicate