Catalogs Validating all your cataloged photos and videos for corruption

[John R Ellis]

Lightroom Version Number

All

Operating System

1. Windows 10
2. Windows 11
3. macOS 14 Sonoma
4. macOS 15 Sequoia

Over the years, there have been infrequent feature requests for Lightroom to provide file validation to detect when files have become corrupted on disk. With archival catalogs now including tens of thousands of digital photos spanning three decades, how do you test if hardware or software failure has corrupted some of them?

There's at least one old plugin that computes hashes of the files and records them in the catalog, but that can be moderately tedious to manage, since any legitimate change to the file (e.g. saving the metadata into a non-raw) will change the hash, and you've got to figure out which hash differences are legit and which indicate corruption.

After discovering two very old corrupted JPEGs in my catalog that weren't recoverable from my backups (which only go back two years), I came up with the following simple method: Export all the photos in my catalogs, which forces LR to read their image data and metadata. If a file is corrupted, it's very likely LR will complain. Using this method, I validated my catalogs (about 50K photos) and found two more corrupted files.

To make these validating exports go faster and not waste disk space, I defined an export preset Validate Files (attached) that exports all photos as 100 x 100 JPEGs at quality 10 and exports video at the lowest resolution (480 x 270). On my Macbook Pro, it took 1 hour 37 minutes to export my main catalog of 40K pics (2/3 raw) and a couple dozen shorter video clips, and the folder of exported files used about 0.2% of the total space for storing the originals.

To use the Validate Files export preset, download the attachment to your desktop. Then in the LR Export window, right-click User Presets in the left column, do Import, and selected the downloaded file. Select all the files in the catalog and do File > Export With Preset > Validate Files. To select all the files in the catalog, go to All Photographs and do Photo > Stacking > Expand All Stacks. Or define a smart collection All Photos with the criterion Rating Is Greater Than Or Equal To 0 (no stars selected).

If you have other validation methods you like, please share!

 

[Johan Elzenga]

To be honest, I don’t validate at all. I have three image backups, so the probability that the same file is corrupted on all three of them is so small that I am willing to take that chance.

 

[clee01l]

To be honest, I don’t validate at all. I have three image backups, so the probability that the same file is corrupted on all three of them is so small that I am willing to take that chance.

I don't validate either. I have 5 image backups. 2 Local, 2 Network and 1 in the Cloud(BackBlaze). Local backups are on 4 different disks which could conceivably corrupt or fail completely but never at the same time. It is my expectation that BackBlaze has their own validation and can insure the constant integrity of their customers files.

 

[Califdan]

To be honest, I don’t validate at all. I have three image backups, so the probability that the same file is corrupted on all three of them is so small that I am willing to take that chance.

I'm not sure about the accuracy of this logic. I think it depends on what mechanism each particular BU applicaiton uses to detect that the file has changed. I suspect that some forms (or causes) of corruption would trigger a backup of the file cuasing the corrupted file to be copied to the backup. I don't know enough about that level of technology to come up with examples but I suspect it's the case.

Having said that, I don't validate either.

 

[Johan Elzenga]

I have not seen any indication that this is the case. Most BU simply use the modification date, and that would normally not change if ‘bit rot’ has corrupted a file. In the few cases where Lightroom Classic suddenly had a problem reading a file, I have always been able to replace that file successfully with a backup copy.

 

[John R Ellis]

I have two direct-attached Time Machine disks, which go back about 10 months, and I use Backblaze with the "infinite" versions option (going back now nearly two years). The files must have been corrupted before then, since the oldest versions in the three different backups were all the same (and corrupted). The files were readable, and Mac Preview showed the JPEGs without any error message at all (minus the bottom rows of pixels).

This is a well-known problem with backups. If something corrupts a file (bad block, bad software, user error), it could get captured into the backup system and eventually the older "good" version of the file gets aged out and disappears.

 

[Philippe Coudé du Foresto]

Over the years, there have been infrequent feature requests for Lightroom to provide file validation to detect when files have become corrupted on disk

In my opinion, this is not the responsabilty of Lightroom.
Lightroom manages only the edit (metadata and/or develop) made to the images by saving them in its catalog, not the files (it has just a link to them in its catalog). Therefore, it is responsible to verify the integrity of its catalog (which it can do when you request it to backup the catalog), not the integrity of the files.
The files containing the images are managed by the OS. Therefore it's the responsability of the OS to garantee their integrity.
It's for the same reason that Lightroom backup the catalog only, not the photo files. Backup of the photo files should be done as any other files on the computer, thru the OS.

 

[BobT]

If you are talking about your images. Firstly, assuming you are following best practice and have everything on the one catalogue, each time you backup the catalogue, I do it weekly, LrC does an integrity check and optimize. The integrity check, checks the catalogue. Would a corrupted image be picked up with that, I don't know, but as other have said, images are the responsibility of the OS.

 

[clee01l]

If you are talking about your images. Firstly, assuming you are following best practice and have everything on the one catalogue, each time you backup the catalogue, I do it weekly, LrC does an integrity check and optimize. The integrity check, checks the catalogue. Would a corrupted image be picked up with that, I don't know, but as other have said, images are the responsibility of the OS.

Lightroom will not detect an image corrupted by "bit rot". Once an image has been through the workflow, ithe original image itself is rearely accessed after import. Things in LrC that will cause the original image to be accessed are mainly exports and prints. The rest of the time LrC is using Previews.

 

[neilhunt]

Over the years, there have been infrequent feature requests for Lightroom to provide file validation to detect when files have become corrupted on disk. With archival catalogs now including tens of thousands of digital photos spanning three decades, how do you test if hardware or software failure has corrupted some of them?

There's at least one old plugin that computes hashes of the files and records them in the catalog, but that can be moderately tedious to manage, since any legitimate change to the file (e.g. saving the metadata into a non-raw) will change the hash, and you've got to figure out which hash differences are legit and which indicate corruption.

After discovering two very old corrupted JPEGs in my catalog that weren't recoverable from my backups (which only go back two years), I came up with the following simple method: Export all the photos in my catalogs, which forces LR to read their image data and metadata. If a file is corrupted, it's very likely LR will complain. Using this method, I validated my catalogs (about 50K photos) and found two more corrupted files.

To make these validating exports go faster and not waste disk space, I defined an export preset Validate Files (attached) that exports all photos as 100 x 100 JPEGs at quality 10 and exports video at the lowest resolution (480 x 270). On my Macbook Pro, it took 1 hour 37 minutes to export my main catalog of 40K pics (2/3 raw) and a couple dozen shorter video clips, and the folder of exported files used about 0.2% of the total space for storing the originals.

To use the Validate Files export preset, download the attachment to your desktop. Then in the LR Export window, right-click User Presets in the left column, do Import, and selected the downloaded file. Select all the files in the catalog and do File > Export With Preset > Validate Files. To select all the files in the catalog, go to All Photographs and do Photo > Stacking > Expand All Stacks. Or define a smart collection All Photos with the criterion Rating Is Greater Than Or Equal To 0 (no stars selected).

If you have other validation methods you like, please share!

This is a great idea - thanks!

 

[John R Ellis]

The files containing the images are managed by the OS. Therefore it's the responsability of the OS to garantee their integrity.

But in my use case, where several files got corrupted, either by an errant software app or a corrupted disk i/o operation, there's no way for the OS to detect this. A RAID volume would protect against bad disk blocks but not truncated disk writes (e.g. during a power failure) or errant software apps. But LR can detect photos that have been changed on disk incorrectly.

 

[neilhunt]

Another reason to do this: “find all missing” isn’t reliable. I just did a find all missing, found a total of two, replaced them, and then initiated a sync to a brand new cloud catalog, and promptly found 8 more missing in the first 20,000. 310,000 more to sync - I’m expecting 50 out 100 more tomorrow morning!

I’m sure they weren’t removed in between, since the gaps are in my cloud backup too. Fortunately, I keep all my full cards in a hanging file rather than reusing them…

 

[BobT]

Fortunately, I keep all my full cards in a hanging file rather than reusing them…

I've heard of people doing that. Flash memory cards are intended for reuse, not archival storage. Data is stored as a charge in the circuitry. That charge can leak in time, "bit rot". Their design criteria is to hold charge for 10 years. However, not all card will achieve that.

 

[thommy]

Another way to check the health and integrity of your image files in Lightroom Classic is by rebuilding your previews. This process not only optimizes performance and conserves storage but can also reveal any issues with file integrity. Peter Krogh has written a great guide on how to do this, which explains the steps and benefits of creating new previews for better speed, space management, and reliability.
The article is from 2020, but I assume it's still valid - maybe our experts here can verify.
You can check it out here: How to Rebuild Lightroom Previews to Optimize Speed, Space, and Integrity

 

[neilhunt]

I'll try out the rebuild previews too - thanks.

 

[neilhunt]

I've heard of people doing that. Flash memory cards are intended for reuse, not archival storage. Data is stored as a charge in the circuitry. That charge can leak in time, "bit rot". Their design criteria is to hold charge for 10 years. However, not all card will achieve that.

I haven't had an HDD last 10 years yet, either, however.
I also store backups on an HDD RAID, but the issue is that if it's online, and a file is removed or corrupted, so does the backup become removed or corrupted.
The only way to solve that is to pull the HDDs offline. Which is a bit cumbersome.

I have SDD drives too - but they have the identical problem to flash cards. I suppose you can read and refresh a large SDD to get another few years.

Saving the flash card is cheap, and easy (just scribble a date and camera name on it, and throw it in a box or file folder). If it lasts 5-10 years, great. As long as it's not the only solution.

 

[John R Ellis]

Another way to check the health and integrity of your image files in Lightroom Classic is by rebuilding your previews.

Good suggestion.

 

[neilhunt]

Another way to check the health and integrity of your image files in Lightroom Classic is by rebuilding your previews. This process not only optimizes performance and conserves storage but can also reveal any issues with file integrity. Peter Krogh has written a great guide on how to do this, which explains the steps and benefits of creating new previews for better speed, space management, and reliability.
The article is from 2020, but I assume it's still valid - maybe our experts here can verify.
You can check it out here: How to Rebuild Lightroom Previews to Optimize Speed, Space, and Integrity

One snag is: it doesn't check any photos where a preview already exists - unless you first purge all previews - in which case, you've just lost your last trace of anything missing...

 

[clee01l]

One snag is: it doesn't check any photos where a preview already exists - unless you first purge all previews - in which case, you've just lost your last trace of anything missing...

If you rename your old Previews folder, you haven't lost anything. Open Lightroom and it not finding a Previews folder will proceed to rebuild previews while your old Previews are still available.

 

[clee01l]

Fortunately, I keep all my full cards in a hanging file rather than reusing them…

A 256GB SDHC card is ~$30. A 256 GB XQD card is ~$250. A 256 GB CF Express card is ~$225. A 1 TB HDD is ~ $30, With that logic, your SDHC cards are 4 times the price of a HDD. Why would you use them instead of a reliable HDD?

How many of your oldest camera cards are still readable? When a camera card fails, it is usually the whole card.