Catalogs Benefit of Lightroom Controlled Backups

[atj777]

Lightroom Version Number

13.3

Operating System

1. macOS 14 Sonoma

Note that I'm not question the benefit of backups

I have a large catalog (~120GB) and if use the backup feature within Lightroom it can take many hours to complete.

If I do an integrity check on my catalog at startup, it takes around 10 minutes. If I quit Lightroom and copy the catalog to another location, it also takes around 10 minutes. i.e. I can achieve what I believe is the same result as the in Lightroom backup but I'm only locked out of Lightroom for 20 minutes.

What am I missing?

Note, that I can zip up the catalog (just like Lightroom does), but I can do that after the copy completes and use Lightroom while that happens.

 

[atj777]

Interestingly, looking back over my backups and the size of the zip files...

Two years ago on V10, my (zipped) catalog was only 8GB
A little over a year ago in V10 it had grown to 26GB
It grew to 30GB after going to V12 a month later
At the beginning of the year on V12 it was 83GB
In May on V13 it was 89GB
Now on V13.3 it is 122GB

Again, these are the sizes of the zipped archives.

Surely, each time the catalog structure changes, Lightroom should be rebuilding the catalog and it should be effectively an optimise.

Edit: And while I am regularly adding images, it is not exponential. I'm probably adding 2-3k images per month. From the start of the year I have added just over 14k images.

 

[Gnits]

I suggest you try the following.

Create a new folder outside of all o/s folders (eg my docs, my pics, etc) or outside any cloud sharing service.. Give it a name you will recognise. Eg TestLrFolderJuly2024.
Inside Lightroom create a new empty catalog inside the new folder. Eg TestLrCatalogJuly2024
While inside the new catalog, import all the images from your current catalog, as per your screen grab above. You do not want to move the images, selecting the option to leave the images where they are.

As you have a large catalog, it will take some time to rebuild the new catalog. As it is not moving images, just rebuilding internal database tables and indices, hopefully it will not take too long. Make sure you have sufficient disk space for the new database and room for work files.

When the import has finished…
Check the new size.
Check how long it takes to do a Lr backup, incl optimise.

I perform this exercise as a matter of routine maintenance approx every 18 months. Lr and its catalog have a lot of stuff to deal with. Your Pc may have power crashes, disk I/o errors, software bugs and a million other things can go wrong. Performing an import into an empty database gives Lr the opportunity to leave behind any rubbish no longer required. Such rubbish includes missing links, corrupt index entries, etc… It is awful complicated inside a real database and this exercise does a lot of spring cleaning.

Before you do this, I suggest you check that there are no files missing in the source database and double check that the number of images in both match after the import. If the import fails, then it points towards some form of corruption in your existing database, But let’s deal with that if that situation arises. You are importing from your current t to a new database, so you still will have your original database, as is.

Run the missing file report again, in the rebuilt catalog, to ensure all images in the catalog are connected to their respective images… to be sure…. To be sure.

 

[atj777]

I think I understand why my catalog is so large based on some experiments I just did.

• I exported one image as a catalog.
• I then used the "new" Remove Tool (not the Generative AI one but the one that came with V12 or V13) 100 times.
• I quit Lightroom and checked the size of the catalog.
• I then opened it again in Lightroom, did a Develop->Clear History, quit and check file size
• Open again, Optimize (which quits at the end) and checked file size.

The results:

1,945,600 bytes (new)
3,366,912 bytes (100 Remove)
3,366,912 bytes (clear history)
2,142,208 bytes (optimise)

So without clearing the history and optimising, it was adding over 1.4MB to the size of the catalog. Clearing the history and optimising improved things but the 100 Removes still adds nearly 200KB.

Now, I haven't been clearing the history so I could get quite a bit of reduction in the size of the catalog. I'm not sure how that can be done on bulk, but I would at least want to do a full optimise first.

Out of interest, I did the above steps but used the older Heal version of Remove.

1,945,600 bytes (new)
2,867,200 bytes (100 Heal)
2,080,768 bytes (clear history and optimise)

I was using Heal before the new Remove was added so even it adds quite a bit to the size of the catalog.

 

[atj777]

By the way, I am surprised by how much difference Clear History did with the spot removal. Yes, I understand what clearing the history is doing, but in the case of spot removal, Lightroom still has to keep track of each and every ring and even the order in which they are done as previous ones affect newer ones. The ability to Undo them should be minor in comparison.

 

[atj777]

By the way, I am surprised by how much difference Clear History did with the spot removal. Yes, I understand what clearing the history is doing, but in the case of spot removal, Lightroom still has to keep track of each and every ring and even the order in which they are done as previous ones affect newer ones. The ability to Undo them should be minor in comparison.

OK... after checking the folders from my test catalogs, it is the lrcat-data that is storing most of that data.

For my test using the new Remove tool, the lrcat-data file grew from 7KB to nearly 1MB (even after clearing the history)
For the test of the older Heal tool, it only grew from 7KB to 60KB.

 

[Gnits]

To clear history of multiple images.

Select the images of interest.

With the images selected in the Library module go to the Develop module and pick the Menu Item Develop/ Clear History.

I would test it out with a few sample images.

I would then create a smart collection , selecting all images, say greater than 5 years since capture date. Go into that collection, select all within the collection, then menu item Develop / Clear History.

You can change this smart collection so that you can select images older than say 6 or 12 months, so you know you always have the history for your most recent images.

I was going to make the point earlier that doing a lot of pixel based editing in Lr causes large blobs of data to be stored in the catalog. This can distort storage needs of the catalog. Adobe have recognised this and have adopted the recent AI based filters pixel data to be stored now outside the catalog. For certain detail work it will always be a call whether to do pixel level edits in Lr or Photoshop.

 

[atj777]

I've tried twice now to run Optimize and both times it failed.

The first time it ran for around 9 hours and it looks like Lightroom crashed (as it was not running when I got up this morning) but there was no usual message to restart, nor were there any crash messages.

I kicked it off again this morning (and I had over 450GB of free disk space). It appears to have crashed 30 minutes later and I now have only 320GB of free disk space but nothing obvious taking up 130GB. I started and stopped Lightroom a few times and even rebooted the Mac but the space has not come back. There's nothing in the bin.

 

[Gnits]

Without re-reading the entire thread. .....


Have you tried exporting your existing catalog (all images) to a new catalog, but leaving the image files in their current location. This will create a new a brand new catalog, built by Lr. Your existing catalog remains untouched. You can then check several things.
1. Has the new catalog got the same number of images as the source catalog.
2. What is the size of the new catalog, compared to the old.
3. How long does it take to do a backup.
4. How long does it take to do a backup and optimise.

My instinct is that there are internal issues with your existing database and the above procedure provide one option to create a fresh version of the database.

If it works ... then you may decide to continue with this new database... but first we need advise on how best to cater for images been synched from the original database to the cloud. I do not use synch or cloud features so am not the best to give advise on this.

If it does not work it may provide a line of enquiry and maybe hints for next steps.

Hint (from experience).... give the new catalog a distinctive new name in a folder you will remember. Send yourself an email of the new catalog name, so you can make sure later it does not get confused with any other catalogs which might be on your system. Avoid folders such as My Pics, My Docs, etc... or any folders within any cloud based synch services, such as iCloud, Dropbox,OneDrive, etc)

 

[Gnits]

Suggestion. If rebuilding a catalog using the above approach it would be prudent to pause synching before rebuilding and only turn it back on when you are happy the new catalog passes your prelim checks that everything is in order.

Also, I make sure that there are no missing images in a catalog before starting this process and then double check in the new catalog that I have the same number of images and that none are missing.

 

[atj777]

My instinct is that there are internal issues with your existing database and the above procedure provide one option to create a fresh version of the database.

That suggests that running an Integrity Check is a waste of time.

Integrity Check has been successful on numerous occasions even immediately after the crash while trying to do Optimize.

 

[atj777]

I managed to get my free space to 550GB and I tried Optimize Catalog again. It crashed again (both Apple and Lightroom crash windows) after around 10 hours.

Assuming that .lrcat.wal is the new copy of .lrcat, it was nearly done, too.

 

[clee01l]

That suggests that running an Integrity Check is a waste of time.

Integrity Check has been successful on numerous occasions even immediately after the crash while trying to do Optimize.

The integrity Check is intended to make sure the internal tables and joins of those tables are complete. If for example there is an index that is a join records of Table A with records of table B and there is no table B, the integrity check would fail. It does not seek out orphan records . Optimization runs the Database VACUUM command to purge the data base of the space left behind by deleted records and to rebuild indexes..


Sent from my iPad using Tapatalk

 

[clee01l]

…

Assuming that .lrcat.wal is the new copy of .lrcat, it was nearly done, too.

WAL means “Write Ahead Log”. It is not a copy of the database file


Sent from my iPad using Tapatalk

 

[Gnits]

That suggests that running an Integrity Check is a waste of time

No…. It does a very useful job, but it is working within the constraints of the current catalog. The option I am suggesting is rebuilding the catalog from scratch. It is not guaranteed to work and it may not even finish, but if you are not making progress with other approaches it remains an option.

 

[clee01l]

No…. It does a very useful job, but it is working within the constraints of the current catalog. The option I am suggesting is rebuilding the catalog from scratch. It is not guaranteed to work and it may not even finish, but if you are not making progress with other approaches it remains an option.

The mains benefit of this is that it only copies the valid records.. The orphaned records that I mentioned earlier get left behind, resulting in a cleaner, leaner database file.

For some one that has a 118GB lrcat file, I think you have a lot of useless, inaccessible records in that database.


Sent from my iPad using Tapatalk