Duplicates !

[david.gb]

Lightroom Version Number

15.4.1

Operating System

1. macOS 26 Tahoe
2. iOS

Don't know how I manage to import so many duplicates but I have.
Now it seems that Adobe have introduced a duplicates filter in Lightroom Classic.
Sounds a great idea but it seems to be offering two means of attack.
Finding duplicates and deleting them permanently from the disc OR finding duplicates and removing them from the catalog.
(I hope I'm correct about this but I'm sure someone will pipe up and set me straight if I have this wrong.)
My "Personal Catalog" has about 180,00 images and I know that many are duplicates but going through the pix one by one is a task that would take longer than my projected lifespan.
So, a duplicate finder is a super idea.
But what is the criteria that is being used to assess which is a duplicate and which is an original.?
Is it a MetaData search or something involving Ai recognition ? What about resolution - obviously I want to keep the HiRes and lose the LoRes copies ?
Sometimes my duplicates have an entirely different filename to the original too - gets complicated !
Basically I'm happy to dump JPEG versions of RAW files too, providing I have't done something worthwhile in the JPEG - for me RAW files are the Holy Grail.
But sometimes I even have duplicates of them too.
At the moment I would like to clean up my hard drive and save some space and cull all the dupes but NOT delete them entirely BUT instead move them to another location on another disc just in case there is something valuable that I need to go back to in the future.
Another thought ... is this catalog wide search for dupes or can I specify a folder or a year in which to search ?

I have lost valuable images in the past down to my own stupidity and sometimes through tech failures and I am reluctant to simply put my life in the hands of software without guaranteeing some sort of fail safe .

Thoughts /Suggestions ??

 

[david.gb]

I'm getting the hang of the Duplicates tool now and it is very useful.
For one single day in May I managed to get 5 copies of the same image for 300 pictures.
Still can't figure out how I did that but glad I've managed to clear that one up.
Now let's see what happened on the next day !!

 

[david.gb]

Actually I think I may have a clue as to how I imported so many dupes ... not sure yet but whilst I was away on an assignment in europe I didn't format my camera cards at all (They always serve as backup if my portable hard drive went missing) - so every time I did an import I used a new folder on my hard drive and that is where the dupes came in - duplicates being silly beasts didn't realise I'd done a previous download of them because they didn't go into the same folder. It was only when I got back and used my master catalog that they all got grouped together on the correct dates , regardless of the fact they came from different folders.

 

[Gnits]

That is a very common scenario.. and sometimes might be hard to avoid if little time between consecutive shoots... to do all the backup stuff.

I now have a strict nightly routine when travelling.. 1. Ingest card to my main disk (Samsung T9 Black). 2. Synch Main T9 to my backup Red Samsung T7.

I use Beyond Compare for Step 2.. which is a super GUI to check all images from source are on the destination drive and tally with my card count.

Step 3. Format the card. This is the most important step as it stops previous session becoming the next batch of duplicates.

All 3 steps happen immediately after each other.. and is a well worn routine.

My biggest source of duplicate images was repeat ingest of previous session images, which I had ingested to Disk (and Lightroom) but had not finished the backup to a second disk.

I know some people keep their cards as a safety backup.. but on multiple trips with lots of other photographers I hear loads of horror stories of cards getting mixed up, the wrong cards formatted, cards shared because people run out of space, etc.

There is no perfect solution.. but keeping to a consistent routine is a big step in the right direction.

 

[Swingman]

That is a very common scenario.. and sometimes might be hard to avoid if little time between consecutive shoots... to do all the backup stuff.

I now have a strict nightly routine when travelling.. 1. Ingest card to my main disk (Samsung T9 Black). 2. Synch Main T9 to my backup Red Samsung T7.

I use Beyond Compare for Step 2.. which is a super GUI to check all images from source are on the destination drive and tally with my card count.

Step 3. Format the card. This is the most important step as it stops previous session becoming the next batch of duplicates.

All 3 steps happen immediately after each other.. and is a well worn routine.

My biggest source of duplicate images was repeat ingest of previous session images, which I had ingested to Disk (and Lightroom) but had not finished the backup to a second disk.

I know some people keep their cards as a safety backup.. but on multiple trips with lots of other photographers I hear loads of horror stories of cards getting mixed up, the wrong cards formatted, cards shared because people run out of space, etc.

There is no perfect solution.. but keeping to a consistent routine is a big step in the right direction.

Lot of backup softwares and the LrM have date selection for copying to avoid repopulating the drives/LrM. Few years back I used to delete/reformat the cards before next secession. It was pointed out that frequent deleting/reformatting the cards degrades them and develop errors and bad sectors. Now we all use larger capacity cards, so I only periodically delete/reformat them. Quite often it also act as additional backup during travelling.

 

[Gnits]

Now we all use larger capacity cards, so I only periodically delete/reformat them. Quite often it also act as additional backup during travelling.

LR Classic does not have an import option to select / filter by date or date range. I have campaigned for this since LRC was launched... to the extent I have given up on this ever becoming a feature and built my own app to ingest my images from card to disk (which has this option) and then use the standard LrC Add mode to import to LrC.

So, if using LrC .. how do you avoid duplicate images if you keep previous imported images on cards.

I totally accept it is more important to have your images backed up (especially when travelling) than to risk duplicate images..

Also, I no longer need to bring a laptop with me when travelling but can use my iPhone to back up my cards to the iPhone itself or to an external SSD ( eg in my case a Samsung T9 or the perfectly formed smaller Samsung T7). I use an IOS App called SynchFoldersPro.

 

[Swingman]

LR Classic does not have an import option to select / filter by date or date range. I have campaigned for this since LRC was launched... to the extent I have given up on this ever becoming a feature and built my own app to ingest my images from card to disk (which has this option) and then use the standard LrC Add mode to import to LrC.

So, if using LrC .. how do you avoid duplicate images if you keep previous imported images on cards.

I totally accept it is more important to have your images backed up (especially when travelling) than to risk duplicate images..

Also, I no longer need to bring a laptop with me when travelling but can use my iPhone to back up my cards to the iPhone itself or to an external SSD ( eg in my case a Samsung T9 or the perfectly formed smaller Samsung T7). I use an IOS App called SynchFoldersPro.

I have been using LrM on ipad for imports from the start of using Lightroom, so I haven’t looked at others. Did you check your iPhone import into LrM? I take my iPad on travel. I also use iPad with uploading at home.
I also create the final destination Album or if continuation of a different one, I import in to it, this avoid dealing with moving photos in Classic as they go straight to right collections.

 

[Gnits]

Thanks. Good to know that Date is a filter option for importing on Mobile Apps. I must check it out as it is a key desireable feature for me.

However, I do not use Lr Mobile Apps to import images for a few reasons.. the main reasons are..

1. I wish to have a workflow which will work for low and high volume imports, ie from 20-50 to tens of thousands. I do not wish to have my images do a round trip to the cloud.
2. I use a lot more apps than Lightroom and active in large projects which involve both technology and construction. It is critical for me to have a formal folder structure to meet these document and image hungry needs. So, it works best for me to use LrClassic where I have optimum control of both my image and project folder structures.
3. I do not plan to touch any of the LrCloud apps to import images until the LrCloud / LrClassic Synch is proven to be rock solid and free of the persistent problems which clog these forums.

I cannot believe that the Lightroom EcoSystem has such a flawed synch system. Synch systems have been working for at least 50 years that cover 99.9999 % of transactions in the ATM and CreditCard business. I am sure there are lots of other real world example. It should not be bleeding edge tech to solve this permanently. I might be temped to use the Cloud Apps for certain scenarios but not until the synch aspect is proven to be stable.

 

[Swingman]

Thanks. Good to know that Date is a filter option for importing on Mobile Apps. I must check it out as it is a key desireable feature for me.

However, I do not use Lr Mobile Apps to import images for a few reasons.. the main reasons are..

1. I wish to have a workflow which will work for low and high volume imports, ie from 20-50 to tens of thousands. I do not wish to have my images do a round trip to the cloud.
2. I use a lot more apps than Lightroom and active in large projects which involve both technology and construction. It is critical for me to have a formal folder structure to meet these document and image hungry needs. So, it works best for me to use LrClassic where I have optimum control of both my image and project folder structures.
3. I do not plan to touch any of the LrCloud apps to import images until the LrCloud / LrClassic Synch is proven to be rock solid and free of the persistent problems which clog these forums.

I cannot believe that the Lightroom EcoSystem has such a flawed synch system. Synch systems have been working for at least 50 years that cover 99.9999 % of transactions in the ATM and CreditCard business. I am sure there are lots of other real world example. It should not be bleeding edge tech to solve this permanently. I might be temped to use the Cloud Apps for certain scenarios but not until the synch aspect is proven to be stable.

Matt, I don’t think any of the Lightroom varieties would handle ten of thousands of images in one go, they were not designed to handle like ATM terminal. ATM machines use package switching to get the data across to the recipient (individual bank), which is more efficient than sync used by Lightroom. You may have to wait until version 99.9 to get similar speedy version.

 

[Gnits]

Matt, I don’t think any of the Lightroom varieties would handle ten of thousands of images in one go, they were not designed to handle like ATM terminal. ATM machines use package switching to get the data across to the recipient (individual bank), which is more efficient than sync used by Lightroom. You may have to wait until version 99.9 to get similar speedy version.

I have regularly handled 5-10 k on cards .. Either helping others while on a trip or assisting people to process their high volume images when back from a trip. Typically using CfExpress cards, Thunderbolt connections and saving to a Samsung T9 SSD.

My own needs are spectacularly more modest as I mostly do landscape and travel. However, I had the opportunity to join a wildlife photography group in Costa Rica who were hoping to capture the fabulous and spectacular Resplendent Quetzal bird. For this special occasion I had my camera set for a fast (not max) frame rate and was extremely cautious about tapping the shutter button... but in that early morning, say 2 hour outing, I was gobsmacked..to have shot more than 3k frames. It was a learning experience for me .. and worth it ..because I captured the image I was looking for..


In the very early days of Lightroom ... when disks were smaller and computers a lot less powerful, I criticised Adobe for not providing an option to import images based on file type (ie jpg or raw). The reason was simple... when travelling, I would only import jpgs into a travel catalog on my laptop, because handling raw was just not an option. I would import the raw files when I got home, to my main system.

I am comfortable with LrClassic, have it tuned and optimised for my needs, but have developed my own apps to streamline getting images into the catalog and getting well formed Jpgs out, in a very simple and efficient manner. I do produce very high quality fine art prints from time to time .. and in those situations time is not a factor... I am happy to work as long as it takes on an image to get the max quality print.

The Atm and Credit cards reference were just examples... and in most cases .. these are really slow transactions... see hwo long it takes an individual to use an Atm. Instead, say, think of paying for a parking spot using App ... so many systems need to coexist.. but the end user experience is vg.. and the parking bookings do not get lost.

 

[david.gb]

Many people use date and time as part of their image filenames. This works ok for most scenarios, but once you start using bursts of images then multiple images may want to use the same filename, as most software often did not cater well for sub second filename tokens.

Yes, I often wondered about that , but not being a sports photographer I very rarely shoot that fast. But I can see the problem emerging - anyone lost images from a burst capture as LRc could not differentiate the metadata timings ?

 

[Swingman]

Yes, I often wondered about that , but not being a sports photographer I very rarely shoot that fast. But I can see the problem emerging - anyone lost images from a burst capture as LRc could not differentiate the metadata timings ?

To improve burst shoots metadata, we also need cameras able to record capture time to 2 decimal places as some are capable of shooting 120fps, and Lr to cope with additional decimal places.

 

[Gnits]

To improve burst shoots metadata, we also need cameras able to record capture time to 2 decimal places as some are capable of shooting 120fps, and Lr to cope with additional decimal places.

For a long time (maybe since beta, not 100% sure) LrC catered for sub-second capture times. What it was missing was to better handle the presentation of this info... and also supply tokens for where people may wish to use this info.

This was improved with updates in Version 15... (and there may be other refinements over time )

https://helpx.adobe.com/ie/lightroom-classic/help/whats-new/release-notes.html

Extract From October 2025 release (version 15.0)​

• Library:
  • 
    
  • Lightroom Classic preserves capture time details with millisecond precision, displayed consistently across time-related fields in the Metadata panel. You can modify capture time details in the Edit Capture Time dialog box.

Here is a sample from a recent shot of mine .. which shows 3 decimal places for the second timestamp segment.

It should be noted ..
1. The original Exif standard only catered for full seconds. They had to add a second field subsequently to store the sub second element.
2. Different camera makers stored different units of time in this field (eg integer element of the decimal value, other units of measurement).
3. Lots of applications only show the main Exif TimeStamp Field (and there are multiple timestamp fields). But serious apps such as Lightroom will and should cater for sub second times correctly.
4. Many photographers add the time stamp in various combinations of ddmmyyhhmmss to their image filenames ..but do not properly cater for the subsecond component.. so lots of images are presented in the wrong sequence as a result (if sorting by filename). There are lots of better ways to achieve this other than using the timestamp code.

I had a look back at some of my images... those shot with a Sony A7R3 show a 2 digit subsecond timestamp in LrC, while the image above from a Sony A7RV shows a 3 digit subsecond decimal. Perhaps if I updated the Process version of the older images this might trigger an update of the presented subsecond time for my older images.

 

[music_cities]

In my quest to remove missing entries from my catalog I also developed a duplicate removal helper. It's the "Has Same Or Better" plugin function, and the collection "Same or Better Comparison" that gets made when you run it.

It just looks for the same file name stem and matching time and matching camera, and same or better resolution, and puts them in a collection. So, for example, you could pick all your photos for a date (or a month) and run Same Or Better from the plugin, to see if those images are in your collection under some other day or month. Or, if you've accidentally included, say, a temporary import folder in your collection, you could check to make sure those images are all in their date folders before deleting them. Or, say LrCloud has added to your catalog a bunch of missing image entries with the Smart Preview dimensions (<2500 pix long end), and you want to be sure those are truly spurious, you could use AnyFilter to search for missing files and cloud synced entries, and then run Same Or Better to be sure that those entries can be deleted from your catalog (unfortunately, presumably also deleting them from LrCloud) because those same images are actually already in LRClassic in another folder.

When I wrote all the helper functions I mostly decided to try to restore the files for the catalog entries with missing files, using Time Machine, Carbon Copy Cloner, and hard linking (or copying, if across volumes) files from other locations. I knew this meant that the spurious entries in the catalog that were duplicates might be harder to find, since now they wouldn't be missing anymore. So, I wrote the Same Or Better function to be used before running my other tools (to remove at least some of the spurious missing photo entries) and then after as well to be able to delete entries that are no longer needed.

Also, if it's disk space you're worried about, just make sure you don't let LrC write metadata into your files (force it to use .xmp files instead, or just leave the metadata in the catalog) and then you can use the excellent rdfind utility to hard-link two images on the same drive but in different folders to the same inode on your hard drive (like a corner house having two addresses, one on each street, you really don't need two houses to have two addresses). Sometimes having duplicates is actually ok, especially if they don't take up extra disk space.

Hope this helps.

Here's my tool containing Same Or Better and the other functions.
https://github.com/jeabraham/hardlink_missing_lightroom_photos