Catalogs What is the upper limit for size of a catalog? Bad performance issues...

Joined
Feb 11, 2019
Messages
42
Location
California
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Lightroom Version Number
14.2
Operating System
  1. macOS 15 Sequoia
My catalog has 338,000 pictures (total 5TB on disc); the catalog itself is about 5GB.
Since late last year, performance has been egregiously bad.
I have several theories:
1. Release of 14.0 with sync reengineered, and "performance" improvements.
2. Using a new camera, with larger RAW files (50MB each)
3. I crossed 300k+ photographs.

Does anyone have a sense of what the practical upper limit of catalog size and scope might be?

The performance issues are
* incredible slowness performing anything to do with synced catalogs (10s of minutes to create a new collection; essentially nothing working after checking the sync on a collection for perhaps an hour or two).
* adding 20 photos to a collection taking 20-30 minutes, or not taking at all.
* the second Import in a session blocking for hours, and importing zero photographs.
* edits (e.g. crops, lighting) that I made in LRC getting reverted for 10-30 minutes, until eventually they show up in LR, and round-trip back to LRC
* changing pick or rating taking 20 minutes or more
* syncing 10 photos taking an hour
* sync (image, not metadata) getting stuck on 3 photos that were first synced 3 years ago, and are now missing from LR altogether. Solution: remove them from LRC, wait a long time for sync to stabilize, then add them back.

These issues are not sudden, but have gradually worsened to the point of unusability over 6 months. Used to be the edit round-trip issue was a few seconds, and creating a new collection was 10-30 seconds, but now, minutes to hours.

I have optimized my catalog repeatedly, and tested catalog integrity at almost every startup. I have had two Integrity failures, which have required 10 minutes of "repair" with no adverse impacts reported or discovered.

I have a MBP with 64GB RAM and 8TB of SSD which still have 2TB free.
 
Just a quick reply....

I have a custom built, few year old Windows PC. High end cpu, motherboard, graphics card, power supply. 64GB of memory and really fast NVme SSD drives.

Following my build.. I was happy with my 140k images processing mostly a7Riii images. The minute I started to process a7RV images the performance turned into a crawl. This may have coincided with a LrC upgrade... but cannot correlate or prove that.

I got acceptable performance back by mainly the following ...

1. Turned off completely Windows Indexing (except for Outlook, One Note)
2. Removed the Catalog folder and All associated Previews from virus checking.
3. Reduced the size of my standard previews to 1680 pixels (it was an option available to me in Lrc preview options. It is a massive waste of rendering cpu and disk to be rendering previews larger than what I will typically use them for in Library view (ie with both panels normally present).
4. The Ai based denoise filters arrived... so I upgraded my GPU to RTX 4070ti 12GB.

As a regular maintenance item...
I completely clear all previews every 18 months or so (rubbish can build up there which can cause congestion)

I used to rebuild my catalog by importing to an empty catalog every 18 months or so... I do not do that now because it screws up the whole synch connection with the status of LrCloud images.

My catalog and last 2 years images are now resident on an External Samsung T9, which I use connected to my main Win Machine and connect to my MacAir when travelling. This is substantially slower than the M3 Nvme drive the catalog used to reside on ... but I do not notice a drop in performance and the portability and ability to import my travel images directly to my working catalog and image library is a joy. I also avoid the need to have a seconf travel catalog, so I only ever connect the same catalog to LrCloud.. so do not create synching issues.

I post a few LrC Collections to albums in LrCloud.... but I deliberately do not use any LrCloud apps (on mobile or desktops or web) , as I wish to have my LrC catalog as the centre of my image universe and I do not want the overhead of managing a stream of originals coming down from the cloud, into a single folder, which then need to be manually transferred to the folders I wish / need to hold these images.

I am not sure how much of this makes sense ... and I may be missing some points.

I do not think that just maxing out the hardware that performance can be resolved. It needs to be optimised in terms of work flow and efficient setup as well.
 
"Does anyone have a sense of what the practical upper limit of catalog size and scope might be?"

At 10 Million, I would start to get concerned. I am aware of many customers whose catalogs are in the millions. Your catalog (300K) is nowhere near that point.
 
I have 190,000 photos in my catalog. Sometime ago, I felt LRC slowing down. I did one thing that Gnits did:
" Removed the Catalog folder and All associated Previews from virus checking."
That seemed to help a lot.
I also keep a "Work in Progress" folder on my C drive so my everyday and new work happens there, and then every week or two I move , via LC, completed work folders to an external USB drive.
 
I’d suggest a more wholistic approach before focusing on the Catalogue file itself
  • Do you regularity backup and verify the catalogue?
  • Have you tried turning syncing off to see if that improves performance?
  • Do you make use of plugins and have verified they are up to date and disabled them to see if that helps performance?
  • Reviewed Victorias posting of performance management. https://www.lightroomqueen.com/tag/performance/
  • As others have said check to see if any Windows programs may be impacting LrC performance. Use Task Manager to see where resources are used when LrC is responding slowly.
 
Back
Top