Finding duplicate files by content across multiple directories

Posted by gagneet on Stack Overflow See other posts from Stack Overflow or by gagneet
Published on 2010-03-08T03:52:24Z Indexed on 2010/03/08 4:21 UTC
Read the original article Hit count: 370

Filed under:

duplicate-data

|

regex

I have downloaded some files from the internet related to a particular topic. Now I wish to check if the files have any duplicates. The issue is that the names of the files would be different, but the content may match.

Is there any way to implement some code, which will iterate through the multiple folders and inform which of the files are duplicates?

© Stack Overflow or respective owner

Related posts about duplicate-data

Avoiding Duplicate Data in DB (for use with Rails)

as seen on Stack Overflow - Search for 'Stack Overflow'
I have five tables that I am trying to get to work nicely together but may need some help. I have three main tables: accounts members and roles. With two join tables account_members and account_member_roles. The accounts and members table are joined by account_members (fk account_id… >>> More
Preventing duplicate Data with ASP.NET AJAX

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Some times you need to prevent User names ,E-mail ID's or other values from being duplicated by a new user during Registration or any other cases,So I will add a simple approach to make the page more user-friendly. Instead the user filled all the Registration fields then press submit… >>> More
Preventing duplicate Data with ASP.NET AJAX

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Some times you need to prevent User names ,E-mail ID's or other values from being duplicated by a new user during Registration or any other cases,So I will add a simple approach to make the page more user-friendly. Instead the user filled all the Registration fields then press submit… >>> More
duplicate data from another sheet in Excel

as seen on Super User - Search for 'Super User'
I have a rather large Excel document with a lot of separate sheets in it. There is some info (email, last name, first name) that has to be the first three columns on each sheet. In order to be sure that no mistakes are made, I created a "Person" sheet that only contains those three columns. On the… >>> More
SQL query to return all rows that contain duplicate data from one field

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a table with card numbers and names. Some of those names are the same, meaning they have multiple cards. How can I select all the rows where the name occurs more than once? >>> More

Related posts about regex

Find multiple regex in each line and skip result if one of the regex doesn't match

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a list of variables: variables = ['VariableA', 'VariableB','VariableC'] which I'm going to search for, line by line ifile = open("temp.txt",'r') d = {} match = zeros(len(variables)) for line in ifile: emptyCells=0 for i in range(len(variables)): regex = r'('+variables[i]+r')[:|=|\(](-… >>> More
OWASP Regex Repository: Is this regex correct?

as seen on Stack Overflow - Search for 'Stack Overflow'
I was looking at the regular expression for validating various data types from the (OWASP Regex Repository). One of the regular expressions in there is called safetext and looks like: ^[a-zA-Z0-9\s.\-]+$ My first question is: Is this regular expression correct? complementary question If this… >>> More
Make a Perl-style regex interpreter behave like a basic or extended regex interpreter

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a tool to help students learn regular expressions. I will probably be writing it in Java. The idea is this: the student types in a regular expression and the tool shows which parts of a text will get matched by the regex. Simple enough. But I want to support several different regex… >>> More
JS regex isn't matching, even thought it works with a regex tester

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm writing a piece of client-side javascript code that takes a function and finds the derivative of it, however, the regex that's supposed to match with the power rule fails to work in the context of the javascript program, even though it sucessfully matches when it's used with an independent regex… >>> More
c# RegEx with "|"

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to be able to check for a pattern with | in them. For example an expression like d*|*t should return true for a string like "dtest|test". I'm no regex hero so I just tried a couple of things, like: Regex Pattern = new Regex("s*\|*d"); //unable to build because of single backslash Regex Pattern… >>> More