Remove duplicate files based on md5 hash

OUTF=rem-duplicates.sh; echo “#! /bin/sh” > $OUTF; find “$@” -type f -exec md5sum {} \; | sort –key=1,32 | uniq -w 32 -d –all-repeated=separate | sed -r ‘s/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/’ >> $OUTF; chmod a+x $OUTF; ls -l $OUTF

http://elonen.iki.fi/code/misc-notes/remove-duplicate-files/

  1. #1 by MD5 Hash Generator on July 27, 2010 - 5:00 pm

    Now that’s the handiest script I’ve seen in a long time. Well done.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: