What is the fastest way to check if files are identical?

Posted by ojblass on Stack Overflow See other posts from Stack Overflow or by ojblass
Published on 2009-04-24T05:02:57Z Indexed on 2010/05/07 10:28 UTC
Read the original article Hit count: 202

Filed under:

comparison

|

files

|

language-agnostic

If you have 1,000,0000 source files, you suspect they are all the same, and you want to compare them what is the current fasted method to compare those files? Assume they are Java files and platform where the comparison is done is not important. cksum is making me cry. When I mean identical I mean ALL identical.

Update: I know about generating checksums. diff is laughable ... I want speed.

Update: Don't get stuck on the fact they are source files. Pretend for example you took a million runs of a program with very regulated output. You want to prove all 1,000,000 versions of the output are the same.

Update: read the number of blocks rather than bytes? Immediatly throw out those? Is that faster than finding the number of bytes?

Update: Is this ANY different than the fastest way to compare two files?

© Stack Overflow or respective owner

Related posts about comparison

Store comparison in variable (or execute comparison when it's given as an string)

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello everyone. I'd like to know if the super-powerful python allows to store a comparison in a variable or, if not, if it's possible calling/executing a comparison when given as an string ("==" or "!=") I want to allow the users of my program the chance of giving a comparison in an string. For… >>> More
On StringComparison Values

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
When you use the .NET Framework’s String.Equals and String.Compare methods do you use an overloStringComparison enumeration value? If not, you should be because the value provided for that StringComparison argument can have a big impact on the results of your string comparison. The StringComparison… >>> More
Use Comparison Operators in SQL to Remove/Return Columns with Null Values

as seen on Internet.com - Search for 'Internet.com'
When you use the comparison operators in SQL with the Where clause, you can remove the columns with null values. >>> More
Use Comparison Operators in SQL to Remove/Return Columns with Null Values

as seen on Internet.com - Search for 'Internet.com'
When you use the comparison operators in SQL with the Where clause, you can remove the columns with null values. >>> More
Building a Web Form in ASP.NET and PHP: a Comparison

as seen on ASP Free - Search for 'ASP Free'
While there are important differences between PHP and ASP.NET both are used to build websites. Because of this both need to enable developers to build web forms among other tasks. This article compares building a web form in PHP with building the same form in ASP.NET to help those familiar with… >>> More

Related posts about files

Unzip individual files from multiple zip files and extract those individual files to home directory

as seen on Super User - Search for 'Super User'
I would like to unzip individual files. These files have a .txt extension. These files also live within multiple zipped files. Here is the command I'm trying to use. unzip -jn /path/to/zipped/files/zipArchiveFile2011\*.zip /path/to/specific/individual/files/myfiles2011*.txt -d /path/to/home/directory/for/extract/ From… >>> More
Storing image files, psd files, ai files, flash in subversion

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, Can I store large amounts of image files in subversion. My designers usually create these designs and store them anywhere on their pc and there's no system. Can I store the files in an svn repository. That way I can also protect my data against unauthorized access and its also easier to archive… >>> More
Copy files in Linux, avoid the copy if files do exist in destination

as seen on Super User - Search for 'Super User'
Hi, I need to copy a /home/user folder from one hardisk to another one. It has 100000 files and around 10G size. I use cp -r /origin /destination sometines I get some errors due to broken links, permissions and so on. So I fix the error, and need to start again the copy. I wonder how could… >>> More
Best way to convert pdf files to tiff files

as seen on Stack Overflow - Search for 'Stack Overflow'
I have around 1000 pdf filesand I need to convert them to 300 dpi tiff files. What is the best way to do this? If there is an SDK or something or a tool that can be scripted that would be ideal. >>> More
what is the difference between ELF files and bin files

as seen on Stack Overflow - Search for 'Stack Overflow'
The final images produced by compliers contain both bin file and extended loader format ELf file ,what is the difference between too , especially the utility of ELF file. >>> More