Avoiding Master Catalog-Workaround

[simonplantphoto]

Lightroom Version Number

10.1.1

Operating System

1. macOS 10.15 Catalina

Hi,
For years Ive worked with a master catalog and have approx 450,00 images in it over 4 external drives.
Recently due to this being almost unusable Ive created a Working catalog where I import my images while the jobs are active.
Once the images/jobs are complete the images are moved to my Master archive catalog.
Thats all fine except I do have a need to search for images regularly in my archive catalog and it's simply painfully slow!

Anyone have a better workflow solution?
My thoughts are to create a separate catalog say for my finished images Tiff,PSD and PSB files and that may well be a workaround.
Interested in hearing anyone else's workflow for large archives.
Simon

 

[clee01l]

Your 450,000 inventory is not a particularly large catalog. It is the Catalog database file that Lightroom searches and the images files scattered over 4 external drives are not accessed, When you say "search", I presume you are using the Filter bar or Smart Collections and searching on keywords, dates or other metadata attributes. Eyeballing the thumbnails in each folder is going to be tedious with 450,000 images.

The releases of the Adobe LrC v10 and updates have caused others to complain of slow downs, I haven't seen any specifically related to the filter bar or Smart Collections but it would not surprise me.

 

[simonplantphoto]

Hi Cletus thanks for the reply.
Yes I hear different stories of much larger catalogs being fine while other smaller than mine also grinding to a halt.
My machine is a few years old now but have 24 gigs of ram and fairly good spec.
I use after effects, premiere and create large composites in Photoshop without much issue but sadly LR has always given me some grief.
I did however learn of a bug to do with custom screen profiles causing an issue and slowdown.
I was sceptical but changed my custom one to sRGB and it does seem happier at least at the moment.
Simon

 

[Jim Leask]

My catalog is only 1/2 the size of yours, at approximately 250,000 images. I haven't seen any trace of slowness though that would cause me to want to split the catalog. I'm interested to know when you started to see the problem.

I was struggling with managing the image files though, and used to have multiple external drives like you describe. I was really worried about losing track of them though, so I decided to spend a bit of money for the following setup that seems to be performing fine.

My catalog and working images stay on my laptop, that has lots of memory and 1TB of SSD. That easily holds enough images for recent shoots, and is backed up with the normal system backup. My main catalog is now on a 12TB Raid drive, that I have as Raid-1 (mirrored), so I have 6TB of duplicated space for my external images. Larger Raid drives are now available for not much more money, so one that would easily handle your files is possible. This drive has server grade drives at 7200 RPM, and connects with the Thunderbolt-3 connector. Performance is fine.

Hope this helps...

 

[simonplantphoto]

Hi Jim,
To be honest it been problematic for years but the last 2 years it's become almost unusable.
With the last major update (2020 version I think) it suddenly was running like a dream but that only lasted a few months.
My latest issue seemed to be a bug which was due to a custom monitor profile and I was made aware here on this forum.
It now seems to be running ok...for the minute
Thanks for sharing.
Cheers Simon

 

[Roelof Moorlag]

Victoria did some articles about perfomance issues, did you look at them?
https://www.lightroomqueen.com/lightroom/performance/

 

[simonplantphoto]

Thanks I'll take a look.
Regards Simon

 

[kimballistic]

What kind of hard drive is your catalog file and your previews on? I.e. internal or external, and spinning disk or SSD?

And if they're stored (hopefully) on a fast internal SSD, are standard previews already built for all 450,000 images? If not, as Lightroom is trying to display the results of your query it will also be grinding away on building previews for every image it's trying to show you.

 

[simonplantphoto]

What kind of hard drive is your catalog file and your previews on? I.e. internal or external, and spinning disk or SSD?

And if they're stored (hopefully) on a fast internal SSD, are standard previews already built for all 450,000 images? If not, as Lightroom is trying to display the results of your query it will also be grinding away on building previews for every image it's trying to show you.

Hi the archive is on several external hard drives with previews (non SSD) so no it’s not going to have the quickest access I realise that.
Simon

 

[kimballistic]

Hi the archive is on several external hard drives with previews (non SSD) so no it’s not going to have the quickest access I realise that.
Simon

The Lightroom Catalog file and the corresponding Previews database is what I was asking about, not your photos themselves. Since you're on LrC 10, you probably have a file called something like "Lightroom Catalog-v10.lrcat". It's probably in a folder called "Lightroom" that has a bunch of other important stuff, like "Lightroom Catalog-v10 Previews.lrdata" and other files.

Is all of that on an external, spinning disk drive? If so, getting all of that on the fastest SSD you can afford would be my first priority.

And of course, triple-checking that every one of those 450,000 images have standard previews built. You didn't mention that.

What Mac computer are you using?

 

[simonplantphoto]

The Lightroom Catalog file and the corresponding Previews database is what I was asking about, not your photos themselves. Since you're on LrC 10, you probably have a file called something like "Lightroom Catalog-v10.lrcat". It's probably in a folder called "Lightroom" that has a bunch of other important stuff, like "Lightroom Catalog-v10 Previews.lrdata" and other files.

Is all of that on an external, spinning disk drive? If so, getting all of that on the fastest SSD you can afford would be my first priority.

And of course, triple-checking that every one of those 450,000 images have standard previews built. You didn't mention that.

What Mac computer are you using?

The previews are built but these and the catalog are on my iMac ‘s SSD NOT stored with images that are on the external drives.
Maybe my way forward is to look at another set of drives large enough to hold around 12TB archive but SSD.
To confirm are you suggesting the catalog and previews should also stored on this.?
Simon

 

[Gnits]

If I was dealing with 450,000 images I would focus on optimising my storage of the images. Using 4 external drives, they are bound to be of different vintages, probably usb, plugged into available usb ports of different speeds. I would consider purchasing a modern disk enclosure, which can take 4 or 8 drives, and populate with as many drives as you need for your current images and the next year or two using say 14TB 7200rpm Nas style drives. I would gradually migrate your existing external drives data to the new disk enclosure.

If heading in this direction, seriously investigate the possibility of using a Thunderbolt 3 connection. If your machine does not support thunderbolt 3 then consider installing a card which will give you Thunderbolt 3 functionality.

As your storage requirements increase over time you can add drives into your enclosure. You can also streamline your backup procedures.

I do not think that using SSD for such volumes will be a good use of money, but you can optimise on Ssd by installing, say a 2 TB drive, which you use for the current year and then migrate those folders to the disk enclosure every year (or frequency of your choice).

Check out the 4 and 8 bay OWC Thunderbolt 3 enclosures.

I have just built a new workstation. I made sure the motherboard I used natively supports TB3, I have installed a Pcie V4 M2 2TB ssd, which has awesome transfer speeds and have transferred all my images to a single TB3 enclosure, backed up to a slow and ancient Nas drive.

 

[simonplantphoto]

If I was dealing with 450,000 images I would focus on optimising my storage of the images. Using 4 external drives, they are bound to be of different vintages, probably usb, plugged into available usb ports of different speeds. I would consider purchasing a modern disk enclosure, which can take 4 or 8 drives, and populate with as many drives as you need for your current images and the next year or two using say 14TB 7200rpm Nas style drives. I would gradually migrate your existing external drives data to the new disk enclosure.

If heading in this direction, seriously investigate the possibility of using a Thunderbolt 3 connection. If your machine does not support thunderbolt 3 then consider installing a card which will give you Thunderbolt 3 functionality.

As your storage requirements increase over time you can add drives into your enclosure. You can also streamline your backup procedures.

I do not think that using SSD for such volumes will be a good use of money, but you can optimise on Ssd by installing, say a 2 TB drive, which you use for the current year and then migrate those folders to the disk enclosure every year (or frequency of your choice).

Check out the 4 and 8 bay OWC Thunderbolt 3 enclosures.

I have just built a new workstation. I made sure the motherboard I used natively supports TB3, I have installed a Pcie V4 M2 2TB ssd, which has awesome transfer speeds and have transferred all my images to a single TB3 enclosure, backed up to a slow and ancient Nas drive.

That’s great advise thank you.
I giving serious thoughts of having a pc built as my IT guy was telling me what he could build me for very little money especially compared to an iMac
The specs sounded incredible and of course very updatable.
I think at this time migrating my archive to faster drives would be beneficial as you have pointed out.
Thank Simon

 

[Gnits]

I used an Asrock X570 Creator motherboard. This is at the upper end of the market, but has TB3 built in, 10GB network port, Pcie V4 for the fastest M2 drives and supports Ryzen 3900 processor. A good platform for a long term solution. Using a disk enclosure, you can build a storage solution, maybe as a priority and later concentrate on the workstation , as funds allow.

 

[kimballistic]

The previews are built but these and the catalog are on my iMac ‘s SSD NOT stored with images that are on the external drives.
Maybe my way forward is to look at another set of drives large enough to hold around 12TB archive but SSD.
To confirm are you suggesting the catalog and previews should also stored on this.?
Simon

Nope, sounds like you're doing it right already. As long as the catalog file itself (.lrcat) and the corresponding previews are on a really fast SSD, you've got the fundamentals right and the performance issues you're encountering while just searching your library aren't likely due to storage hardware speed bottlenecks.

Just to be clear, putting your terabytes of actual image files on faster disks will not speed up Library module searching, unless Lightroom needs to access the image files to create new previews.

It will speed up switching between images in the Develop module when those images aren't already in ACR's cache. That's about it, as far as I know. Might not be worth it.

I'd definitely continue reviewing the Performance tips that Roelof linked to above.

Also, Adobe has their own list of tips:

https://helpx.adobe.com/lightroom-classic/kb/optimize-performance-lightroom.html

 

[Gnits]

With 450,000 images there is no guarantee that previews are always available for all your images, so building previews may still be a factor in various operations, especially if these have to be built from slow usb 2 drives.

With that many images I would every so often (ie maybe once a year as a routine maintenance job), create a new blank catalog, import all my images into the new catalog (without moving the images) and rebuild previews for your most recent (or likely to be used) folders. This allows all the database indices to be rebuilt from scratch, flush out any work areas and rubbish accumulated over time, during app/system crashes, power outages, etc..

I agree, there are many factors affecting performance and all suggestions should be followed. Also, Kimballistic is correct, having your Catalog (and previews as a result) on your fastest drive is the way to go and putting your images on faster drives only has incremental benefits in performance. If you can keep your current active working folders also on a fast ssd then you maximise the opportunity to improve performance arising from disk i/o.

 

[gegjr]

I'm just a smuck here but there are two people I'd suggest you talk to;
1. Jared Platt
2. Tim Grey of GreyLearning.com
Finally
I didn't see any one suggesting you consider a RAID setup.

 

[Jim Leask]

I'm just a smuck here but there are two people I'd suggest you talk to;
1. Jared Platt
2. Tim Grey of GreyLearning.com
Finally
I didn't see any one suggesting you consider a RAID setup.

I suggested a RAID setup earlier in the thread, and the setup that @Gnits mentioned is almost certainly a RAID setup too. That is the way to go IMO for the master image repository. If the images are important to retain, don't trust them to a simple USB hard drive, and especially a lot of USB drives that can get lost and not backed up. Yeah, it costs a bit of money for that drive, but remember the statement, the images are important.

 

[gegjr]

I suggested a RAID setup earlier in the thread, and the setup that @Gnits mentioned is almost certainly a RAID setup too. That is the way to go IMO for the master image repository. If the images are important to retain, don't trust them to a simple USB hard drive, and especially a lot of USB drives that can get lost and not backed up. Yeah, it costs a bit of money for that drive, but remember the statement, the images are important.

Mr. Leask,
I do apologize. I just skimmed through the post so I obviously missed your suggestion. But thanks for setting the record straight.

 

[gegjr]

My catalog is only 1/2 the size of yours, at approximately 250,000 images. I haven't seen any trace of slowness though that would cause me to want to split the catalog. I'm interested to know when you started to see the problem.

I was struggling with managing the image files though, and used to have multiple external drives like you describe. I was really worried about losing track of them though, so I decided to spend a bit of money for the following setup that seems to be performing fine.

My catalog and working images stay on my laptop, that has lots of memory and 1TB of SSD. That easily holds enough images for recent shoots, and is backed up with the normal system backup. My main catalog is now on a 12TB Raid drive, that I have as Raid-1 (mirrored), so I have 6TB of duplicated space for my external images. Larger Raid drives are now available for not much more money, so one that would easily handle your files is possible. This drive has server grade drives at 7200 RPM, and connects with the Thunderbolt-3 connector. Performance is fine.

Hope this helps...

Mr. Leask,
I found it. My apologies, sir.
Sincerely,
GEGJr

 

[Jim Leask]

Mr. Leask,
I do apologize. I just skimmed through the post so I obviously missed your suggestion. But thanks for setting the record straight.

No problem at all - easy to miss. I just wanted to point out that a decent RAID disk is critical for a large library, any library for that matter, and I suspect will go a long way to solving performance issues like this, as well as just not losing track of images.