How to reduce duplicate web pages when rolling my own search engine?
- by user198729
One articles can be posted on many sites
the same article on a specific website can be visited by several different urls.
How do I reduce duplicated stuff in the above occasions?