Comments on: Find duplicate files by content not name

By: T

Tue, 05 Aug 2008 01:59:36 +0000

it has to be the coolest command ever !
Why didn’t i think of it.

Two similar files will have same md5, isn’t it ?

Jakub Szypulka — Fri, 04 Jul 2008 15:58:35 +0000

Would be cool to have a GUI for that, or even integrated in the file browser.

IAnjo — Thu, 03 Jul 2008 19:44:08 +0000

Yeah fdupes works very well too!

Ted Hanney — Thu, 03 Jul 2008 17:26:22 +0000

Sam Merrell — Wed, 02 Jul 2008 04:25:35 +0000

If I remember right, xargs is easier on memory usage as well.

John — Tue, 01 Jul 2008 12:38:57 +0000

we use duplicate finder from ashisoft to find and remove duplicate files.

You can find the free trial version at : http://www.ashisoft.com

Andreas Schneider — Tue, 01 Jul 2008 08:56:48 +0000

Normally you want to use xargs instead of -exec. The -exec option calls md5sum for every file found. With xargs, md5sum is only called once.

Tested with cold caches on ~200 files:

time find . -type f -exec md5sum ‘{}’ \; | sort | awk ‘dup[$1]++{print $2}’
…

real 0m6.567s
user 0m0.636s
sys 0m0.604s

—

time find . -type f -print0 | xargs -0 md5sum | sort | awk ‘dup[$1]++{print $2}’
…

real 0m5.454s
user 0m0.620s
sys 0m0.272s