I'm trying to detect if an article or forum post is a duplicate entry within the database. I've given this some thought, coming to the conclusion that someone who duplicate content will do so using one of the three (in descending difficult to detect):

1. simple copy paste the whole text
2. copy and paste parts of text merging it with their own
3. copy an article from an external site and masquerade as their own

Prepping Text For Analysis

Basically any anomalies; the goal is to make the text as "pure" as possible. For more accurate results, the text is "standardized" by:

1. Stripping duplicate white spaces and trimming leading and trailing.
2. Newlines are standardized to \n.
3. HTML tags are removed.
4. Using a RegEx called Daring Fireball URLs are stripped.
5. I use BB code in my application so that goes to.
6. (ä)ccented and foreign (besides Enlgish) are converted to their non foreign form.

I store information about each article in (1) statistics table and in (2) keywords table.

(1) Statistics Table The following statistics are stored about the textual content (much like this post)

1. text length
2. letter count
3. word count
4. sentence count
5. average words per sentence
6. automated readability index
7. gunning fog score

For European languages Coleman-Liau and Automated Readability Index should be used as they do not use syllable counting, so should produce a reasonably accurate score.

(2) Keywords Table

The keywords are generated by excluding a huge list of stop words (common words), e.g., 'the', 'a', 'of', 'to', etc, etc.

Sample Data

• text_length, 3963
• letter_count, 3052
• word_count, 684
• sentence_count, 33
• word_per_sentence, 21
• gunning_fog, 11.5
• auto_read_index, 9.9
• keyword 1, killed
• keyword 2, officers
• keyword 3, police

It should be noted that once an article gets updated all of the above statistics are regenerated and could be completely different values.

How could I use the above information to detect if an article that's being published for the first time, is already existing within the database?

I'm aware anything I'll design will not be perfect, the biggest risk being (1) Content that is not a duplicate will be flagged as duplicate (2) The system allows the duplicate content through.

So the algorithm should generate a risk assessment number from 0 being no duplicate risk 5 being possible duplicate and 10 being duplicate. Anything above 5 then there's a good possibility that the content is duplicate. In this case the content could be flagged and linked to the article's that are possible duplicates and a human could decide whether to delete or allow.

As I said before I'm storing keywords for the whole article, however I wonder if I could do the same on paragraph basis; this would also mean further separating my data in the DB but it would also make it easier for detecting (2) in my initial post.

I'm thinking weighted average between the statistics, but in what order and what would be the consequences...