Huge catalogs ... is SSD advantageous?

[Kirby Krieger]

Lightroom Version Number

Up-to-date Lightroom Classic CC as of 10.Oct.2018.

Operating System

1. macOS 10.13 High Sierra

Hi. I have two _very large_ image catalogs (one is over 1,000,000 Photos, the other just under; nothing special, I'm doing some experimental artwork). Simple (?) question: what is the fastest storage, other than an SSD system drive (which I have) on which to keep them? The answer may be just an external TBolt SSD — but I don't want to pay for one if a good HHD or an SSD with an older BUS gives me the same performance. One thing I'm wondering is if there is something limiting throughput other than drive/BUS speed.

2018 27" iMac, TBolt3, USB3, with 1 TB SSD.

Lightroom Classic CC works well with these large Catalogs. My issue is where to store them (or what to store in the Catalog and how to manage the storage across devices). All image files are stored on a massive external TBolt RAID HDD.

Fwiw, one of the catalogs has been on an external 1 TB SSD since I moved from Aperture. The other in on my system drive, but at c. 750 GB now needs to be moved to another device, as between it and the system files I am now routinely running out of storage on my system drive. I am happy to hear advice on limiting the size of the Catalog. I need to use full-size previews; I prefer to not use Smart Previews; and I don't keep Previews I'm not using past 30 days. On of my needs is that I regularly import tens of thousands of Photos over the course of a few days (and then may not import much of anything for weeks or months).

Bonus Q: these two catalogs are inter-related. I would like to join them at some point, as a combined catalog would save me the bother of moving Photos from one to the other. I will probably give this a try at some point, but it entails having, I would think, a 2TB external SSD, which makes it an expensive experiment. Anyone tried this or anything similar?

Thanks.

 

[davidedric]

For clarification, are you describing image files or Catalogues?

Where your images reside and where your Catalogue(s) (Lightroom's internal database) reside are different questions.

For example, many store their Catalogue on ssd, and their images on conventional disks.

Dave

 

[Zenon]

Yes clarification. For 3TB of files my catalogue shows about 280MB. Most recommend to keep the actual files on an external drive and the catalogue on the HD. You should have about 100GB of free space on your HD. SSD will improve performance.

One million files in each? I'm guessing files in two folders? I might just keep it that way but someone might say it is fine to merge them. You can use the merge to another catalogue command and LR will bring one into the other and automatically move the files as well. That is a lot files to move.

 

[Kirby Krieger]

Thanks — I should have been more clear.

I am talking about just the Catalogs. All the image files are stored on a massive (iirc 24 TB) external TBolt RAID. The problem I'm dealing with is that the main Catalog is now c. 750 GB, which is too big to keep on my 1 TB system SSD.

A large portion of the Catalog is Photo Previews. Most of the time, I import a few tens of thousands of images, work with them for a few days, and then don't work with them for some time. A workflow that creates on import Previews adequate to my 27" monitors, and then auto-deletes them when they haven't been used in, say, five days, would, afaict, benefit me by keeping the amount of storage space used by Previews smaller. Alternatively, might I remove no-longer-needed Previews manually? Is there a way to tell Lightroom to delete Previews for selected Photos?

I will post system specifics when I am back at work.

Re: moving large numbers of image files. IME — with Lightroom versions prior to the current one — when you are moving more than something like (that is, broadly) 200,000 image files it is faster to move then with a file manager and then reconnect then with Lightroom's tools than it is to have Lightroom move them. (NB: for almost all Lightroom users, since this condition is not met, it is best to use Lightroom to move the image files of the Photos in the Catalog.) With that many image files it can take Lightroom days to finish (iirc, I let one operation run for just under eight days — that was the last time I tested).

 

[Zenon]

You can keep your catalogue on an external drive if you want to. I did for a while but I keep my drives pretty clean so I have the space. I that 100GB of free space is a rule of thumb. Adobe says 20%. I don't think you will see a big performance loss.

I don't know if this will be of any help but most of my setup is based on it.

Optimize Lightroom performance

 

[PhilBurton]

Thanks — I should have been more clear.

I am talking about just the Catalogs. All the image files are stored on a massive (iirc 24 TB) external TBolt RAID. The problem I'm dealing with is that the main Catalog is now c. 750 GB, which is too big to keep on my 1 TB system SSD.

A large portion of the Catalog is Photo Previews. Most of the time, I import a few tens of thousands of images, work with them for a few days, and then don't work with them for some time. A workflow that creates on import Previews adequate to my 27" monitors, and then auto-deletes them when they haven't been used in, say, five days, would, afaict, benefit me by keeping the amount of storage space used by Previews smaller. Alternatively, might I remove no-longer-needed Previews manually? Is there a way to tell Lightroom to delete Previews for selected Photos?

I will post system specifics when I am back at work.

Re: moving large numbers of image files. IME — with Lightroom versions prior to the current one — when you are moving more than something like (that is, broadly) 200,000 image files it is faster to move then with a file manager and then reconnect then with Lightroom's tools than it is to have Lightroom move them. (NB: for almost all Lightroom users, since this condition is not met, it is best to use Lightroom to move the image files of the Photos in the Catalog.) With that many image files it can take Lightroom days to finish (iirc, I let one operation run for just under eight days — that was the last time I tested).

Do you have a laptop or a desktop system? If the latter, why not get a second SSD of 1 TB or even 2 TB? No, it won't be cheap. The lowest price Samsung 2 TB SSD is $377 (on www.newegg.com), but considering the scale of your overall photo system, it is relatively inexpensive way to keep your workflow simple.

Phil

 

[Zenon]

I have an old iMac with 500 GB, HD and updated to 16 RAM. I looked into an SSD but even my computer guy said it wasn't worth it. I missed the Mojave update by a year (2011). I was planning on getting a new computer next year so I'm going to hold out. Glad I waited for a while. Mojave pretty much dictated the my path.

I picked up the least expensive MacBook Air I could find for travel. SSD, 125 GB and 4 RAM. It is faster than my iMac for most things. Boots up way faster. I use a 2TB portable ED with it. Some is partitioned for back up.

 

[Kirby Krieger]

Thank you Phil, Xenon, and Dave :thumbsup: .

My #1 question is in the Topic title: is an (external) SSD advantageous (over an external HDD)(for storing a Lightroom Catalog)?

I know I can put the Catalog on an external drive. I do this regularly.
I know the ED can be an SSD. I do this as well.

I am planning to move the now-too-large-for-my-system-drive Catalog to an external drive (and am interested in ways to limit the size of the Catalog other than limiting the number of records in the database).

My options for what will be a new ED are (afaik — I haven't shopped for drives in a year or so):
- SSD , TBolt
- SSD , USB3
- HDD

The TBolt SSD is the most expensive storage option. Does it provide any performance advantage over the other, less-expensive, storage options?

The reason I ask is that I worry that there may be a factor that prevents my system from taking advantage of the (more expensive and higher) thoughput the external SSD provides.

My next question will be : what specs should I look for in an external SSD? A brief look shows lots of variations among drives currently being sold (some of which date back to before 2010).

Thanks.

 

[Zenon]

Sorry I don't know much about drives themselves expect that personally I would go with an SSD, if you can afford it.

 

[PhilBurton]

Thank you Phil, Xenon, and Dave :thumbsup: .

My #1 question is in the Topic title: is an (external) SSD advantageous (over an external HDD)(for storing a Lightroom Catalog)?

I know I can put the Catalog on an external drive. I do this regularly.
I know the ED can be an SSD. I do this as well.

I am planning to move the now-too-large-for-my-system-drive Catalog to an external drive (and am interested in ways to limit the size of the Catalog other than limiting the number of records in the database).

My options for what will be a new ED are (afaik — I haven't shopped for drives in a year or so):
- SSD , TBolt
- SSD , USB3
- HDD

The TBolt SSD is the most expensive storage option. Does it provide any performance advantage over the other, less-expensive, storage options?

The reason I ask is that I worry that there may be a factor that prevents my system from taking advantage of the (more expensive and higher) thoughput the external SSD provides.

My next question will be : what specs should I look for in an external SSD? A brief look shows lots of variations among drives currently being sold (some of which date back to before 2010).

Thanks.

SSD performance specs have increased dramatically since 2010, and prices per GB have fallen dramatically. Just for education's sake go to the Amazon or the Newegg website and search for Samsung SSD. Samsung's SSD product generations are numbered 830, 840, 850, 860, and now 870. There are also 900 series product generations. Pick a specific drive size, say 512 GB and search on the available models. Look at the prices, and then look at key performance specs.

You can buy an external SSD, or you can assemble one yourself by getting a raw 2.5" drive and an external case. Look here. Hard Drive Enclosures - Newegg.com. To sort things out, use the selection criteria on the left side of the screen and check/tick the boxes for the TBolt or USB 3 interface. Considering that SSDs run very cool, a plastic case is fine. You don't need a metal case to cool off the drive. If you assemble this external SSD yourself, you can be sure of getting the "most" SSD for the least cost.

Phil

 

[Kirby Krieger]

Sorry I don't know much about drives themselves.

Thank you for clarifying this limit in your expertise.

 

[clee01l]

Thank you Phil, Xenon, and Dave :thumbsup: .

My #1 question is in the Topic title: is an (external) SSD advantageous (over an external HDD)(for storing a Lightroom Catalog)?

I know I can put the Catalog on an external drive. I do this regularly.
I know the ED can be an SSD. I do this as well.

I am planning to move the now-too-large-for-my-system-drive Catalog to an external drive (and am interested in ways to limit the size of the Catalog other than limiting the number of records in the database).

My options for what will be a new ED are (afaik — I haven't shopped for drives in a year or so):
- SSD , TBolt
- SSD , USB3
- HDD

The TBolt SSD is the most expensive storage option. Does it provide any performance advantage over the other, less-expensive, storage options?

The reason I ask is that I worry that there may be a factor that prevents my system from taking advantage of the (more expensive and higher) thoughput the external SSD provides.

My next question will be : what specs should I look for in an external SSD? A brief look shows lots of variations among drives currently being sold (some of which date back to before 2010).

Thanks.

Let's return for a minute to that 750GB Catalog folder. Do you need Smart Previews? Most people don't. If you always have the master image file EHD attached, you can stop creating these and Delete the Smart Previews. folder. Although Smart previews can now be helpful in the develop module, These are probably not a necessity of older images well mature and developed sometime ago. How many regular Previews do you need? If you delete the Previews folder, LR will create a new one and start building new previews as needed and not perhaps for the older images. While this will slow LR down briefly when recalling older images, it won't impact your development of new images. Doing these two things will reduce your storage limits quickly and you can decide whether a larger drive is still required.

A TB3 EHD is as fast or maybe faster than an internal drive. I would look for one that maximizes R/W cache. You also might find that a conventional disk with a TB3 connection is as fast as anything that you might require.

 

[Zenon]

Thank you for clarifying this limit in your expertise.

I haven't had the need to investigate them. It came with my laptop but we wanted to keep the price down so it didn't matter. As long and the editors worked. Interesting to read replies.

 

[Kirby Krieger]

Thank you Cletus.

Let's return for a minute to that 750GB Catalog folder. Do you need Smart Previews?

I have never used Smart Previews and I never expect to use them. The editing I do needs to be done at 100%.

How many regular Previews do you need? If you delete the Previews folder, LR will create a new one and start building new previews as needed and not perhaps for the older images. While this will slow LR down briefly when recalling older images, it won't impact your development of new images. Doing these two things will reduce your storage limits quickly and you can decide whether a larger drive is still required.

I need the Previews for a few days after importing files. That's it ... until I may need them down the line, at which point I am more than willing to have Lightroom create them anew. Considering how many Photos I'm managing in my Lightroom Catalogs, and how small the percentage of them that I edit per week is — and that nearly all of those are newly-imported — it does not seem worthwhile to commit storage space to Previews of Photos that are not likely to be edited soon.

Is there a more finely-grained way to control Previews other than Catalog-wide? Is there any way to selectively delete Previews? If I had my druthers, I'd keep Previews larger than a thumbnail of only Photos imported in the last 6 days and Photos that have been edited. That's probably 5% of all the Photos in my Catalog.

Here are all my Previews settings (that I know of):
- Preferences ▹ Performance ▹ Develop ▹ No check in "Use Smart Previews"
- Preferences ▹ Performance ▹ Preview Generation Settings ▹ Check in "Generate Previews in parallel"
- Catalog Settings ▹ File Handling ▹ Standard Preview Size ▹ "Auto (5120 px)"
- Catalog Settings ▹ File Handling ▹ Preview Quality ▹ "Medium"
- Catalog Settings ▹ File Handling ▹ Automatically Discard 1:1 Previews ▹ "After One Week"
- Catalog Settings ▹ File Handling ▹ Smart Previews ▹ Total Size ▹ "0 bytes"

How much storage is given to Previews?
"Catalog Settings ▹ File Handling ▹ [Un-titled top section that is most likely supposed to be titled "Previews"] ▹ Total Size" is 433 GB. (Added: the Catalog size is currently 480 GB.)
OmniGroup's Disk Sweeper reports numbers that let me calculate that 98.5% of the data stored in the Catalog is Previews (total size of Previews database divided by total size of Catalog file package; the Previews database in held inside the Catalog file package).

I _just_ calculated that. I had thought I could control the issue of the Catalog size by better controlling the storage of Previews; I now see that the issue is _entirely_ the storage of Previews.

A TB3 EHD is as fast or maybe faster than an internal drive. I would look for one that maximizes R/W cache.

Thanks. I will.

You also might find that a conventional disk with a TB3 connection is as fast as anything that you might require.

That's the question this Topic addresses. I'm not likely to test this. Is there any other way to find out?

 

[clee01l]

I have never used Smart Previews and I never expect to use them. The editing I do needs to be done at 100%.

Then Delete your Smart Previews folder and uncheck the box in the import dialog labeled "Build Smart Previews". Lightroom will still build smart previews when it needs to just not automatically on import.

Is there a more finely-grained way to control Previews other than Catalog-wide? Is there any way to selectively delete Previews? If I had my druthers, I'd keep Previews larger than a thumbnail of only Photos imported in the last 6 days and Photos that have been edited. That's probably 5% of all the Photos in my Catalog.

In the Library Menu there is a sub menu "Previews" You can build or delete previews on selected images. I would begin by deleting my current Previews folder, then select the images that I wanted LR to maintain Previews for. Building a large number of new Previews this way will take some time. So, you might want this batch process to run over night.

That's the question this Topic addresses. I'm not likely to test this. Is there any other way to find out?

Try to find another LR user that has the drive that you are interested in and see if they find this particular drive to be suitable for their needs. I am inclined to think that any 7200 RPM spinning SATA disk on TB3 would be fast enough. I have one TB3 drive that I use for auxiliary storage (not catalog) And I think it is transparent where the data files are located

 

[Kirby Krieger]

Then Delete your Smart Previews folder and [more fine advice]

Triple thanks!

I have no file nor folder with "Smart" in the name in my Catalog file package. May I conclude that I have no Smart Previews in this Catalog?

I have the file "catalog_name Previews.lrdata" in my Catalog folder (on a Mac). From what I've read, that is my "Previews Folder" — yes?

Thx.

 

[clee01l]

That is correct, not Smart Previews. If they were present, the folder would be named "catalog_name Smart Previews.lrdata".
Another item that you could be used to make the Lightroom more compact on the SSD would be to move the backup folder to an EHD. The backup folder might also contain many older useless backups. There is no housekeeping on this folder, so it is unto the user to manually clean up backups that are not relevant to any conceivable recovery.

 

[Kirby Krieger]

Another item that you could be used to make the Lightroom more compact on the SSD would be to move the backup folder to an EHD. The backup folder might also contain many older useless backups. There is no housekeeping on this folder, so it is unto the user to manually clean up backups that are not relevant to any conceivable recovery.

Recommended. I verify and backup all Catalogs on closing. I keep the backups on external drives. I manually delete old backups (I saved a link to the backups folder in a handy place so I can get to it with one click). :thumbsup:

 

[Conrad Chavez]

Thank you Phil, Xenon, and Dave :thumbsup: .
- SSD , TBolt
- SSD , USB3
- HDD
The TBolt SSD is the most expensive storage option. Does it provide any performance advantage over the other, less-expensive, storage options?

It's a good question. Answering it involves comparing three factors: The drive type (HDD, SSD), drive interface (SATA, PCIe), and transfer interface (USB, Thunderbolt).

First let's look at how drives perform using the common SATA 6Gb/sec (750MB/sec) interface you find in most affordable enclosures:
Hard disk drives (HDD): 125-200MB/sec.
SSDs: 450-550MB/sec

Your iMac probably has USB 3.0 ports which get you up to 5 Gb/sec, or up to 625 MB/sec. If you compare that to the numbers above, you can see that USB 3.0 can easily handle any hard drive and most common SSDs. Thunderbolt could also handle that of course, but Thunderbolt interface drives cost much more. There is not much reason to pay more for a Thunderbolt interface drive if what's inside it is just a slow HDD or common SSD.

So if USB can handle all that, is there any point in buying a Thunderbolt interface external drive?

There is also a type of advanced SSD using the PCIe interface that can reach 3000MB/sec, many times faster than common HDDs or SSDs. This is the "flash storage" built into current MacBook Pros. You can get this type of storage in external enclosures (for a price), but obviously USB 3.0 is so much slower that it would completely bottleneck these superfast drives. But Thunderbolt 3, with a theoretical top speed of 5000MB/sec, can handle that.

The other reason to go Thunderbolt is when common SSDs are ganged up as a RAID, because then they can exceed the speed of USB 3.

Bottom line:
If you're going to get an external hard drive or common SSD, and you want to keep the cost down, USB 3 is fast enough, common, and affordable.
If you're going for top speed at any cost, get a PCIe (also called NVMe) SSD, and connect it via Thunderbolt 3.

 

[Zenon]

Thanks. Good info. After seeing how my 4GB laptop smokes my 16GB iMac I always lean towards SSD. I kept your post for reference.