Catalogs Benefit of Lightroom Controlled Backups

[atj777]

Lightroom Version Number

13.3

Operating System

1. macOS 14 Sonoma

Note that I'm not question the benefit of backups

I have a large catalog (~120GB) and if use the backup feature within Lightroom it can take many hours to complete.

If I do an integrity check on my catalog at startup, it takes around 10 minutes. If I quit Lightroom and copy the catalog to another location, it also takes around 10 minutes. i.e. I can achieve what I believe is the same result as the in Lightroom backup but I'm only locked out of Lightroom for 20 minutes.

What am I missing?

Note, that I can zip up the catalog (just like Lightroom does), but I can do that after the copy completes and use Lightroom while that happens.

 

[Hal P Anderson]

Are you backing up to the same disk that you did a straight copy to? That could make a difference in the timings.

Note that you'll need to also copy <catalogue>.lrcat-data to where you're manually backing up.

 

[Califdan]

In the LrC backup routine, it tells you which phase it is in. Optimize, check integrity, copying catalog, zipping. See which steps are consuming causing the problem. I suspect it's the Optimize. If so, uncheck it as well. I don't think it needs to run more than once every month or so.

 

[PhilBurton]

In the LrC backup routine, it tells you which phase it is in. Optimize, check integrity, copying catalog, zipping. See which steps are consuming causing the problem. I suspect it's the Optimize. If so, uncheck it as well. I don't think it needs to run more than once every month or so.

No way to run just optimize without quitting LrC?

Isn't optimization a form of catalog "cleaning" that minimizes the possibility of corruption, and also recovers unused space caused by deleting individual files in the catalog?

 

[atj777]

Are you backing up to the same disk that you did a straight copy to? That could make a difference in the timings.

Yes. The same external drive in both cases.

Note that you'll need to also copy <catalogue>.lrcat-data to where you're manually backing up.

I hadn't because I didn't realise I'd need it. It's only 20% of the size of the catalog so will only add around 2 minutes

 

[atj777]

In the LrC backup routine, it tells you which phase it is in. Optimize, check integrity, copying catalog, zipping. See which steps are consuming causing the problem. I suspect it's the Optimize. If so, uncheck it as well. I don't think it needs to run more than once every month or so.

I already disable Optimize as that is a major problem on my catalog. I've run it on it's own (i.e. nothing to do with backup) and had to cancel it after 10 hours because it never finishes.

 

[karl.nyhus]

Yes. The same external drive in both cases.

You can get shorter times if your system can read from one disk drive and write to a different disk drive.

 

[atj777]

No way to run just optimize without quitting LrC?

File->Optimize Catalog...

 

[atj777]

You can get shorter times if your system can read from one disk drive and write to a different disk drive.

Perhaps I worded that poorly.

In both cases (Lightroom backup and manual backup), I am backing up the catalog from internal SSD to external drive.

 

[atj777]

Just did the manual copy again, this time including lrcat-data and it took around 12 minutes. So, still well under half an hour versus multiple hours with Lightroom doing it.

Note that the most time consuming piece (50+ minutes) is the compress but with the manual method, that can be done with Lightroom running.

 

[clee01l]

There is no reason for the lrcat file to be 120gb. How many images are catalogued? One thing that can increase your time to backup is an excessive number of development history steps. You can delete development history steps as you are unlikely to need them. This will give you a smaller and easier to manage catalogue file.

Optimize uses a Database command VACUUM which removes the space consumed by deleted records. It also rebuilds database indexes making the processes more efficient.

Often over time the database ends up with orphan records for unknown reasons. Possibly due to poorly designed plugins

One thing that cleans up the database is to export it to a new catalog file. This copies the good records and leaves the orphans behind producing a smaller more compact file.


Sent from my iPhone using Tapatalk

 

[BobT]

I think it's an optimise issue. I suspect that if you let it run the optimisation for however long it takes, you will see a great improvement. Please don't be tempted to interrupt the optimisation. There is no telling what corruption that could cause. The catalogue is a "database". A database is a specific, highly efficient, data storage method. Think of it as a long list on one-line records. Each record contains the location of an image and all the metadata, edits data and whatever, associated with that image. Each time and image is imported, a new line is added to the bottom of the list. Each time an image is deleted or otherwise removed from LrC, the record is cleared, however the space remains. So, if you import say, 100 images, 100 records are added to the bottom of the catalogue database. If you then cull 95 of them, 100 records still remain, only 95 will be blank. The database will remain like that forever unless the catalogue is optimised. You can see that with daily adding and deleting, databases (catalogues) can grow rapidly resulting in poor performance and waisted storage. In optimising, all blank records are deleted and the database blocked back up tidily. All databases, in whatever systems, need to be regularly optimised.

 

[atj777]

There is no reason for the lrcat file to be 120gb. How many images are catalogued?

344,730.

 

[atj777]

So, if you import say, 100 images, 100 records are added to the bottom of the catalogue database. If you then cull 95 of them, 100 records still remain, only 95 will be blank.

I almost never delete images from the catalog. The vast majority of my images I use for scientific (natural history) purposes and even poor images contain useful biological information.

 

[atj777]

I should add that I used to optimise the catalog on a regular basis (independent of backup) and it never took that long. It was after upgrading to 12 or 13 that it became ridiculously slow.

Interestingly, it was slow but tolerable on older versions of Lightroom Classic even on my old 2015 Mac Mini. I have a brand new 2023 Mac Mini with Apple M2 Pro, 32GB of memory and over 450GB of free disk space and it is incredibly slow to optimise.

 

[Califdan]

No way to run just optimize without quitting LrC?

Not that I know of

Isn't optimization a form of catalog "cleaning" that minimizes the possibility of corruption, and also recovers unused space caused by deleting individual files in the catalog?

For the most part yes. The corruption part of your statement is probalby more in the Integrity check though. It's sort of a house keepping routine. I don't know exactly what the LrC version does but in general realtional database admin, such routines can be used to build or clean indexes, remove dead space left over from the deletion of rows from tables, reclaim space from tables that have had a lot of deletes, etc. In my experience these sorts of problems tend to gradually build up over time - sort of like the need to de-frag a disk drive doesn't happen in a day.

The problem is that the optimize routine has to touch virtually everything in the catalog and many times rewrite much of it which can be quite time consuming for larger catalogs. Even if it finds absolutly nothing to "optimize" (like you run it twice in a row) it still takes the same amount of time to do the analysis needed to determine that there is nothing to optimze. So, I only tend to select it along with the integrety check once every 2 to 3 months. I know some folks will consider this approach as risky but I've been doing it this way ever since taking the catalog backup got to be over an hour, which as I recall must be 8 to 10 years now with no nagative impact that I know of. And, to be honest, I actually can't detect any peroformance imporvement after I run it compared to just before I run it.

 

[Gnits]

Consider adopting a strategy to delete the image edit history. Say , all history for images captured more than a month or year ago.

Most people never use the history feature, Maybe you do and that will influence your approach. Maybe you save the history of images at different points in time and give these names (snapshots).

By deleting the history you are not deleting the develop settings, just deleting all the steps required to get to the current status.

When is the last time you used the history feature.

With 344k images, you are constantly backing up all the editing steps each time and this history may go back to your initial use of Lr.


In the early days of Lr disk capacity was scarce and the Lr backups started to explode in size, to the extent I spent more time managing spare disk on my system than editing images. I used a 3rd party utility and found the tables inside the Lr catalog that had either the largest number of table entries or occupied the most space. I was gobsmacked by how much space the history table consumed. So I adopted a strategy to delete the history for images older than a few months.

Time has moved on, I have had several hardware upgrades since and Adobe have become more efficient with the internal organisation and management of the internal table structures. I no longer delete history, but with the volume of images in your catalog it may be worth considering.

 

[atj777]

Not that I know of

File->Optimize Catalog...

 

[atj777]

Consider adopting a strategy to delete the image edit history. Say , all history for images captured more than a month or year ago.

Most people never use the history feature, Maybe you do and that will influence your approach. Maybe you save the history of images at different points in time and give these names (snapshots).

By deleting the history you are not deleting the develop settings, just deleting all the steps required to get to the current status.

I did play with this in the past and found it made little difference.

While it will in theory delete the steps so you can't go back to a point in in the edit process, all those steps must still be kept by Lightroom in order for it to apply all the edits to the original raw photo (which remains unchanged) to the final image that you export. i.e. if you were to do 100 spot removals to an image, Lightroom still has to know about each of the 100. Deleting the edit history stops you from going back to say the 50th one, but Lightroom still needs to know about them all.

 

[atj777]

Anyway, I have to go into the office on Wednesday so I might kick off an Optimize on Tuesday evening and can leave it running until at least Thursday morning and perhaps Thursday afternoon. That'll give it at least 36 hours to run and maybe as much as 48 hours.

 

[Paul McFarlane]

Can you just confirm again the size (disk) of your catalog? 120GB just doesn't sound right. My one with 2 million records is 20GB. Are you referring to the whole folder, including Previews?

 

[Gnits]

all those steps must still be kept by Lightroom in order for it to apply all the edits to the original raw photo (which remains unchanged) to the final image that you export.

The steps do not need to be kept. Lr remembers the final settings.

Do the following experiment.

Select a test image, with no adjustments.
Perform a typical set of adjustments..eg exposure, highlights, shadows, highlights. Move the sliders to more extreme settings than normal, so the adjustments stand out.

Now apply a few major adjustments (which make the adjustments even more obvious).

• Apply a strange crop.
• Convert to Mono (any of the presets).

You now have a radical version of a processed image.

If you look at the history panel in the develop model (left side) you will see the history of your adjustments.

Take a screen grab of the processed image.

In Library mode select ONLY the single test edited image.
In the Develop module, use the Menu item Develop / Clear History.

You will find that Lr has still retained all the developed adjustments, but it has cleared the history of all the steps taken to achieve those settings. You can check this by comparing the current version to the earlier screen grab.


You do not need to clear the history, it is totally optional.
But if you do not actively use the history and you back up your catalog regularly (which you should) then you are backing up the accumulated history of all your develop edits every time.

Remember, you can always create a virtual copy of an image if you wish to retain a specific set of adjustments for that image.

 

[clee01l]

344,730.

My catalog is 1.3 GB and contains 66,000 records. Extrapolating my file size to 340,000 records, at most you might expect a catalog file of 12.75GB

 

[Califdan]

110,000 images (including VC's) in catalog, lrcat file 3.7gb. lrcat-data older 470mb

 

[atj777]

Can you just confirm again the size (disk) of your catalog? 120GB just doesn't sound right. My one with 2 million records is 20GB. Are you referring to the whole folder, including Previews?