Bits of this will get grabbed for safenetwork-community/ dedup-tester
Thanks @Aragorn for doing my bash-scripting homework for me 
Both scripts will be similar, mine however will use “standard candles” so we can get some clues about de-duplication which forms part of the discussion over here