Over time, you may have ended up with a load of duplicated photos that you need to clean up. Merging catalogs can introduce duplicates, but so can importing photos with Don’t import suspected duplicates unchecked, importing exported photos or using the catalog on more than one device can also clutter your catalog with duplicate photos. Whatever the cause, it’s time to clean up…
You could scroll through the photos in Capture Time order, manually looking for each duplicate photo, but if you have more than a few thousand photos, this could take a LONG time. Fortunately, there are a couple of plug-ins that can help you identify the duplicates – Duplicate Finder (£9.95 GBP/$13.99 USD approx.) and Teekesselchen (open source donationware).
Both plug-ins come with documentation, but to get an overview of how the plug-ins work and learn a few of the pitfalls and tricks, we’ll take Teekesselchen for a test run.
Install the Plug-in
- Download the latest ZIP file and double-click to access the plug-in inside.
- Move the teekesselchen.lrplugin file somewhere safe.
- I keep mine in a Plug-ins folder with my presets and templates. To find this folder, go to Lightroom’s Preferences dialog > Presets tab and click the Show Lightroom Presets Folder button, then create a Plug-ins folder, and copy the plug-in inside.
- You might prefer to keep the plug-in next to your catalog, or another location you can easily find.
- In Lightroom, go to File menu > Plug-in Manager.
- Click the Add button, navigate to the teekesselchen.lrplugin folder/file (not the files inside it) and click Done.
A Little Bit of Prep
A little bit of prep work can save time sorting through the suspected duplicates later.
- Back up your catalog first. (We all make mistakes!)
- Fix any missing photos, if it’s possible to do so.
- Delete photos already marked with a reject flag. (If you don’t want to delete your rejected photos, you’ll need to tweak the plug-in settings to prevent it changing your flag status.) To do so:
- With the All Photographs collection still selected, go to Photo menu > Delete Rejected Photos to delete your rejected photos from the hard drive.
- Mark your externally edited photos (e.g. Photoshop PSD/TIFF files) with an Ignore keyword, so that they’re not identified as duplicates for deletion. To do so:
- Select All Photographs collection in the Catalog panel.
- Go to Photo menu > Stacking > Expand All Stacks to show all of the photos.
- Select Text Filters at the top of the Grid view. Set it to Filename contains -Edit (or whatever word you use to identify your externally edited files).
- Select all of the resulting photos using Ctrl-A (Windows) / Cmd-A (Mac).
- In the Keywording panel, add a new keyword. I used “Teekesselchen Ignore”.
Search for Duplicates
Next, you’re ready to run the plug-in and search for suspected duplicates.
- Decide which photos to search.
- The plug-in can search just the selected photos, selected folders/collections, or the whole catalog.
- In most cases, you’ll want to search the entire catalog, so select the All Photographs collection in the Catalog panel, and click in the border surrounding a thumbnail to ensure that only 1 photo is selected.
- Open the plug-in by going to Library menu > Plug-in Extras > Find Duplicates (not File > Plug-in Extras).
- Decide on plug-in settings. The default settings shown on the Summary tab are pretty good, but you can save some work by fine-tuning the settings for your own needs. I’ll leave you to read the documentation, but these are a few settings to look out for:
- On the Marks tab, you decide how the suspected duplicates are recorded in the catalog:
- If you use reject flags in your own workflow, and you decided not to delete the rejects from the hard drive, go to the Marks tab and uncheck Mark duplicates as rejected and Reset rejected flag to prevent the plug-in overriding your reject flags.
- If you use color labels in your own workflow, go to the Marks tab and uncheck Abuse color labels for sorting so that the plug-in doesn’t change your color labels.
- On the Rules tab, you can decide which metadata is used to identify duplicates. These determine how identical photos must be in order to be recognised as duplicates.
- To save time sorting through the duplicates, you can run multiple passes of the plug-in, initially using very strict criteria to find and delete photos that are almost certainly duplicates, and gradually expanding the criteria to find other possible duplicates.
- Lightroom only stores capture times to the nearest second, so photos shot in burst mode may be marked as duplicates. The plug-in can use ExifTool to access sub-second data (if it was recorded by the camera) however the process is slow, so if you don’t frequently shoot in burst mode, it’s usually quicker to exclude these manually later.
- For your initial pass, you’ll probably want to check Ignore virtual copies, but this option can come in handy when tidying up virtual copies created by the merge process.
- At the bottom, check Ignore Keywords and enter the keyword you added to your externally edited photos when you were preparing your catalog (e.g. Teekesselchen Ignore) so that these photos are skipped.
- On the Marks tab, you decide how the suspected duplicates are recorded in the catalog:
- Click Find Duplicates to start the process. This takes some time, but you can walk away from the computer while it works. My catalog of 105k photos took around 1 hour to complete.
Check the Results
The plug-in doesn’t delete any files. It just identifies suspected duplicates so you can decide which versions of the photos to keep.
Once the plug-in finishes, go to the Collections panel and select the Duplicates smart collection to view the suspected duplicates. If you left the settings on their defaults, the plug-in will have marked most photos as rejected, just keeping one copy of each photo unflagged.
Now it’s time to start making some decisions…
- When you find a photo you want to delete, select it and hit the X key to mark it as rejected.
- If you find a rejected photo you want to keep, hit the U key to unflag it.
As you’re looking through the suspected duplicates, there are a few things to look out for:
- File Types/Sizes – You’re generally going to want to keep the original raw file or JPEG, rather than an exported copy. The plug-in is fairly intelligent about selecting the best file, but it’s still worth double checking as you go through the photos. You can see this information in the Library module’s Metadata panel, with the pop-up set to EXIF and IPTC.
- Missing Photos – If you were unable to fix all of the missing photos before running the plug-in, take extra care when sorting through the photos. If some of the unflagged photos are marked as missing (with a rectangle or exclamation mark in a rectangle in the corner), you’ll probably want to mark it as a reject and keep one of the duplicates that can still find its original. In some cases, the “missing” photo will have the edits, in which case, keep both the missing photo and a duplicate original, and copy the metadata between photos using instructions in the next post.
- Burst Shots – The plug-in can’t usually tell which photos are burst shots, shot within the same second, so it may incorrectly mark them as rejects. As you scroll through, look out for these photos. When you find them, select them and hit the U key to remove the reject flag, as they’re not really duplicates and you won’t want to delete them. While they’re still selected, you could also uncheck the Duplicate keyword from the Keyword List panel, so that they’re removed from the smart collection and disappear from the current view.
- Edited Photos – As you’re scrolling through the photos, look out for versions of the photos that you’ve edited in some way – perhaps adding star ratings, title, caption, keywords, collection membership or Develop edits – and you’ll probably want to keep the edited version or copy the metadata onto the copy of the photo you’re keeping. (There’s more on copying metadata in the next post.)
The main places to check for edits are:
- Thumbnail
- Have you added flags, stars or color labels?
- Have you edited the photo, adding keywords, GPS location, collection membership, or Develop settings?
- Is the original file missing or damaged?
- Is the photo a virtual copy, shown by a triangle in the lower left corner?
- Metadata panel
- Have you added a Title or Caption?
- What’s the file type?
- Keywording panel
- Have you added keywords to the photo?
If the copy of the photo you want to keep is a virtual copy, it’s simple. First, promote the VC to Master status using Photo menu > Set Copy As Master and then delete the other copy.
But at this point, you may still have some photos where one copy of the photo has your Develop edits and another has your metadata. In another post, we’ll discuss how to copy metadata and settings from one photo to another, so you can then go back and delete the duplicate.
Delete the Rejects
Once you’ve finished sorting through the photos, it’s time to delete the duplicates. Select the All Photographs collection in the Catalog panel, then go to Photo menu > Delete Rejected Photos to delete them. They’re deleted from the hard drive, as well as being removed from the catalog.
For extensive information on Lightroom Classic, see Adobe Lightroom Classic – The Missing FAQ.
Note: purchase of these books includes the first year’s Classic or cloud-based Premium Membership (depending on the book purchased), giving access to download the latest eBook (each time Adobe updates the software), email assistance for the applicable Lightroom version if you hit a problem, and other bonuses.
We also have a special bundle offer for the two books. This includes Premium Membership for the first year as described above for the whole Lightroom family!
Originally posted 22nd May, 2017, updated 18th March 2019 with Teekesselchen verified on Lightroom Classic 8.2.
Will this work for Lightroom 4 and if not what will – Thanks
Lightroom 4… as in the version released in 2012? Or the cloud version released a couple of years ago? The cloud version, no. The really old one… it might as it was first developed in 2013, but I no longer have that version running to be able to check. No harm in trying it.
Here it is, 2024, and this post is still accurate and relevant. Thank you, Lightroom Queen, for taking the time to share such clear directions. No jargon, just facts. Yay!
Thanks Suzanne, glad we could help!
Suzanne,
Thank you for posting an August 2024 comment. The original date of the Lightroom Queen article gave me concern that Teekesselchen might not work with LrC’s recent versions (I’m using Lightroom Classic 13.5).
Thank you, too, Victoria, for a genuinely helpful review, including the time to scan 105K photos. My catalog is 174K and I was worried about processing time.
Big thanks to both of you!
We’re pleased to continue to recommend it, and glad you’ve found it helpful.
What an excellent guide! Worked first time and picked up 24000 duplicates out of 40,000 photos. Could never have worked this out on my own. Thanks for that.
Thanks for the feedback! I’m really pleased it worked well for you.
Is the Duplicate Finder plugin still being supported? I downloaded it recently, but I am getting various error messages. The most recent information I find on their website appears to be from 2018. I am not certain if they are continuing the upgrade their product to reflect more recent OS and versions of Lightroom.
As far as I know, nothing’s changed that should require updates, but you could email the developer about the error messages, or try the Teekesselchen plug-in instead.
Hi – is there a way of searching for duplicates in Lightroom (the online product)? All the articles I see seem to relate to Lightroom CC. There is no help from Adobe on the subject that I’ve found.
No, only in Lightroom Classic so far.
Nice tutorial, but this works better for Windows: https://www.duplicate-photo-finder.com/lightroom-duplicate-finder-plugin.html
Does it show the photos in Lightroom, so you can make sure you don’t delete your edited versions? Otherwise you could create quite a mess.
If you’re on a Mac, try Photosweeper. It understands Lightroom Classic catalogs.
Fast, reliable and very configurable. You can set filters based on metadata. Do fuzzy search based on histograms or bitmaps. Mix and match with files on disk, Apple photos, etc for consolidation. I’ve used it on catalogs of hundreds of gigabytes with 100k+ photos.
Doesn’t work on Lightroom CC (cloud). Flies on the M1-based Macs. I bought it years ago and it’s been consistently maintained with free updates. I’m a happy customer.
Since Photosweeper is a program outside of Lightroom Classic, and it is “really important” to make ALL changes to Lightroom Classic folders while IN Lightroom Classic, does anything get messed up using Photosweeper to remove duplicates? The image locations are not moving, so I am assuming that the answer is no, but I have so many thousands of images, I really wanted to ask someone who knows for sure. Thank you.
Neither of us have used it, but the author states it doesn’t actually remove anything (good) but marks suspected duplicates as rejects in Lightroom and puts them in a special collection.
That’s the same idea as the one we discussed above (Teekesselchen) which is quite safe as you’re in command of deciding what to do with the results.
Thank you, Paul. Much appreciated. Seems like I should use the plugin to be safe.
I used Teekesselchen using the recommendations in this post and they worked great! I also use Gavin Gough’s “The Photographer’s Workflow”, with uses color labels for sorting and reject flags, so your instructions on looking through the settings were immensely helpful and also helped prevent disasters for me.
I do have two questions, please:
(1) I have a total of 40,777 photos and “T” found 25,253 duplicates — which will be quite a few to go through one-by-one. How do you know if the ones it found are the ones you really want to delete vs. the one that is (well, will be) left as the “original”? Is there a straight-forward way to do determine?
(2) Rather than actually delete the duplicates, have you considered moving them to a separate drive that you then delete from Lightroom “just in case” — or, is that more or less just continuing the problem being solved in the first place?
Thank you, again, for the clear explanations, advice, and graphics in this post — absolutely excellent!
Knowing which to delete may depend on which ones you’ve keyworded or edited or used in a collection. If they’re identical, it probably doesn’t matter.
Yes, you could move them to another drive and then remove from Lightroom’s catalog if you wanted an additional safety net for a while.
Thank you very much!
Now that my operation is completed (thank you for your additional responses!):
(1) The real problem *was* that I had not checked the “set to mark duplicates as rejected” box marked and, therefore, what I had was other ones that I had marked to delete but had not, yet. Completely de-dup’d, from 40,777 to 25,465 — all unique (as far as I can tell!).
(2) I decided that moving them to another disk, in the end, would just perpetuate the problem that got me to this point. So, once I am all “finished-up” consolidating 9 drives to 1 and re-doing my catalog structure do dates (as you suggest) I will probably just create a legitimate backup to store offline of, space permitting, put in the cloud as a compressed backup.
Now, on to the next step using “How do I consolidate photos from multiple drives”, to be followed by “Pros and cons of different folder structures” and “How do I rearrange existing photos into a new dated folder structure”.
Thank you again for these great articles and tutorials!
I posted this on the teekesselchen site but not been lucky with replies so far:
I need help with the ExifTool option please, the more I Google it, the more confused I get.
Problem: bursts of photos taken within a 1 second slot are marked as duplicates.
Solution: tick the “Use ExifTool” box, but I can’t get it to work.
I downloaded the ExifTool.exe and placed its location in the Windows “Path” environment variable – it runs from a command prompt OK.
BUT, how does Lightroom (I use the full ver6 paid-for version), and the plugin know about it?
Do I need to do something with ExifTool before I go looking for duplicates?
Is there an ExifTool plugin I should find or install into Lightroom?
That’s a good question, and one I don’t know the answer to, I’m afraid. That’ll have to be one for the developer to answer, if your camera does record the sub second data (some don’t).
Is there a Duplicate File Detector/Deleter for Lightroom (cloud based) at all?
Very satisfied with Teekesselchen! Thanks for the developer and thank you for the tips.
If someone wants to find and delete duplicated files, I ‘d recommend you to look at some software like Duplicate Files Deleter, I am very satisfied from it.
Thanks for your comment, Garry. A word of warning to everyone though – that’s an Operating System level duplicate files program, this won’t help you at all with Lightroom duplicate files (if you delete files from the OS then Lightroom will show them as missing, this doesn’t help clean-up especially if you’ve deleted the copy that Lightroom has all the amendments applied to)
teekesselchen.lrplugin seesm to be updated “V1.8 released on 25th of January 2014” so it’s getting really old.
Do you still think that it should work ok and should be used with newest Lightroom or do you think that the Duplicate Finder would be better?
The Lightroom SDK hasn’t changed for years, so I think it should still work the same as before. No harm in trying it.
Now I have used it few days and it seems to work. I was going to buy that Duplicate Finder again but because of your site I found this teekesselchen and it is everything what I needed =) So thank you about that tip especially even that all your lr tips have been really good and useful =)
Glad to hear it Petri!
Hi, Im really struggling to install teekesselchen – it says it’s installed but when I go to plug-in manager in LR and click “add”, teekesselchen is greyed out and I can’t select it – any ideas?
If it’s already installed, you shouldn’t need to add it in the plug-in manager. Check if it’s already showing up in the menus.
I have the same issue a d tried using both listed as. Plugin manager never finds the apo. I installed both on my HD and they clearly are there but LRC Plugin Manager doesn’t seem to see it. Ido a browse and it doesn’t show them at all to add.
Getting very frustrated with the many bugs and illogical coding of LRC. Why would anyone set the default to add duplicates? That box should always be checked automatically and you have to manually uncheck if you want dups
Make sure the location you put the plugins in are ones Lightroom has permissions to access.
The ‘Don’t Import Suspected Duplicates’ box should remember the option you set it to previously.
Hi, Victoria! If you were choosing again today (from one day shy of a year ago when you wrote the article),
(1) Would you choose the same two plug-ins to do this? And,
(2) Would you prefer one over the other?
I am about to do this and would like to make the best-advised choice since I have just over 39,000 photos in my library and *many* duplicates (unfortunately) due to issues too numerous to describe. 🙂
Many thanks!
Sincerely,
Karl.
I haven’t come across any other good Lightroom duplicate detection plug-ins, and I’d try the free one first because… well, it’s free!
Thank you very much — I appreciate it!
Thanks Victoria. That did the trick. You are a GREAT help. Best Kurt
Hi and thanks for some great work on making LR easier for us. ?
I can go through the whole Tesselkelchen proces, but when I have chosen to reject a picture (or many) I get the message: “You can’t delete photos directly from a smart collection.” So I cannot delete the duplicates. What am I doing wrong?
best Kurt
Once they’re marked as rejected, you can swap to the All Photographs view in the Catalog panel and delete them from there, or you can use the splat-delete in the smart collection (that’s Ctrl-Alt-Shift-Delete / Cmd-Opt-Shift-Delete) but be careful as it doesn’t ask for confirmation.
Hi Victoria, I am having similar problems to most of the above, but I am using Duplicate Finder. So I am unable to find “Mark or Rules” tab. Please advise.
Which problem specifically Peter? I haven’t tested the Duplicate Finder plug-in for a long time, so you’d need to refer to the developer’s documentation.
Hi Victoria! Thanks for reading this in the first instance. My problem is, each time I export a final picture to any folder on my hard drive, Lightroom (CC Classic) detects the file and auto-import the JPG exported file to my library, so I have at least 2 pictures of EACH photo I export, and I don’t like to use that often Teekesselchen (duplicate finder), why is this happening? Thanks a lot!
In the Export dialog, make sure you don’t have Add to Catalog checked. That’s the most likely suspect. Lightroom doesn’t monitor your hard drive, except for a single watched folder you may set up, so it must be something you’ve set.
they are not the same program (duplicate finder 2 and teekesselchan, but they are very similar
I have the same problem–shows installed and working in LR Manager and also shows up under Library Plug-in Extras but it is grayed out and thus not a choice. Any ideas?
Just found that Find Duplicates just below and indented from Teekesselchan does sort for duplicates. Also noticed that the interface is very similar to previous plug-in FindDuplicates2. Are they the same program? I don’t need to have both is there is no difference.
Yes, it’s the indented one you want. I’m not sure which FindDuplicates2 plug-in you’re thinking of, but it certainly sounds like they do a similar job, if they’re not the same one.
I am using the OS X version of Lightroom Classic CC. My OS and Lr Classic CC are up to date.
Yesterday (2018.02.03) I purchased and downloaded the latest OS X version of Duplicate Finder. I installed the plug-in in my Lightroom Plug-In folder and added it to Lightroom Plug-in Manager.
The plug-in shows up with a green dot next to it in the Lightroom Plug-in Manager menu. I also added EXIFTool as well and I think I have the path indicated correctly (but maybe not, this is the first time I have tried indicating a path to something like EXIFTool in Lightroom . )
Still, when I go to the Plug-in Extras menu , Duplicate Finder does not show up.
I have quit and restarted Lightoom twice since installing it and still it doesn’t show up.
To follow up, there are other plug-ins that are green dotted do not show up in the Plug-In Extras menu as well. I have made screen shots of both the Plug-In Manager and Plug-In Extras menus. How can I share those so we can determine whether this is user error , plug-in author error, or an Adobe problem?
I re-read your comments above about ‘You’re definitely looking under the Library menu > Plug-in Extras and not the File menu one?” and no I was not looking at Library menu > Plug-in Extras.
Problem solved!
Oh good, well done Ellis! That one trips up loads of people.
For duplicate files I recommend you to try DuplicateFilesDeleter.
The downside of an external duplicate detection app is it doesn’t know which of the duplicates you’ve edited in Lightroom, so you can end up deleting the wrong ones and creating a long job restoring broken links.
I am looking for the absolutely easiest method to put everything into the same catalog. Can I do this without any steps like cleaning up or identifying duplicates, etc. Recognizing this is less elegant, can it be done this way without harm being done?
If photos are duplicated at multiple locations, it creates more of a mess to clean up later, but yes, give it a try and if it looks messy, you can start over. https://www.lightroomqueen.com/merging-catalogs-3-import-catalog/
Maybe try posting on the developer’s website?
Same thing with me, plugin is active but instead of showing the program is shows nothing.
If the duplicate finder plug-in has already been run in the past, yet more photos have been added to LR since, can the plug-in be run again? Will it add any newly found duplicates to the duplicate collection that already exists? Or will it create a whole new duplicate collection? (Not sure if that’s a silly question or not) Thanks!
Hi Brenna
You can run the plug-in as many times as you like, it puts duplicates found into the collection you specify. It doesn’t take any actions on the actual photos themselves, just reports on them
I have the same problem
Plug installed and shows in the manager with the green circle next to it but in the “Plug in extra” tab I cannot see “Find Duplicates” instead it says “none defined”.
any idea why?
Thanks
You’re the second person to say that this week, so it’s certainly possible there’s a bug, although it’s showing up ok on mine. You’re definitely looking under the Library menu > Plug-in Extras and not the File menu one?
I know that I need help on lightroom, but first what is teekesselchen sending to someone else, not me? What is teekesselchen doing, other than finding duplicates, for me?
Hi George
It’s just finding the duplicates and recording them in your catalog – it doesn’t send any information anywhere else
Paul
Plug installed and shows in manager but does not show in plugin
Does it have a green circle next to it in the plug-in manager, showing that it’s active? Where exactly is it not showing?
I have the same problem
Plug installed and shows in the manager with the green circle next to it but in the “Plug in extra” tab I cannot see “Find Duplicates” instead it says “none defined”.
any idea why?
Thanks
I had the same problem on my Mac. The solution was to get into Lightroom-> Files->Manage Plugins and add a new one (plus symbol). I found the plugin in /private/tmp. After adding the plugin that way (make sure you installed it before) it worked in Lightroom