Improve slow LRC on iMac: new disk, new Mac Studio?

[gYab61zH]

Lightroom Version Number

LRC 12.1

Operating System

1. macOS 12 Monterey

Hi,

First:
iMac Retina 5K, 27 inch, 2020; 3,6 GHz 10-Core Intel Core i9; 32Gb RAM (last of the Mohicans). Monterey.
Storage: 500 Gb internal SSD; external 8Tb Samsung 870 QVO SATA 2.5 Inch SSD

I spend many hours a day behind the iMac working with LRC. I used to be quite happy with my setup but after the last few LRC updates I am becoming more and more frustrated by its lack of speed: I sit there waiting for images to load properly in the library module (sometimes they do not load at all and I have to restart LR) and this slows down the selection of what to keep and what to abandon considerably; building Standard Previews takes a long time on Import and I don't even dare to build 1:1 Previews because my Previews catalogue is already 262Gb large (catalogue contains 162800 images). Topaz Photo AI is slow too, though it zips along quite nicely on my 13" M1 MBP (2020). Consequently I cannot keep the catalogue itself on my internal SSD so I am forced to save it together with the images on the external SSD.

It would be nice if I could speed things up a bit, but I have been wrecking my brains on how to achieve this. The easiest way to do it, I thought, would be to replace the Samsung SSD with an 8Tb M.2 SSD, but I cannot find an external case that will hold 2x4Tb NVMe M.2 cards nor do I know of a small NAS built around such cards. I guess I could move to a Mac Studio and a Studio Display but that adds up to well over 3000 Euros (i.e. a big invest so soon after getting the iMac).

Any suggestions on how to improve on this setup would be much appreciated.

 

[Gnits]

Nas is a network connection. 1Gbps is 125 GB/s. Add network overheads, Nas overheads, Windows overheads. I also found adding a Thunderbolt enclosure did not work for me. I reverted to putting my image archive on an internal spinning disk and my current years images in a fast internal M2 drive.

Top Tips for obvious checks.
1 How much spare space on your system drive.
2. Remove Catalog, PreViews and Image folders from Spotlight.

 

[gYab61zH]

Thanks. Yes, know about a NAS (I run 3 at home; 1 main; 2 backups). I was thinking more about a NAS/Enclosure with a direct connection to the iMac (USB/Thunderbolt). A RAID enabled enclosure would be needed for multiple NVMe's to be available as one disk.

 

[CliffGilbert]

OWC do Thunderbolt 3 NVME raid enclosures (OWC Express 4M2), or you can use SATA SSDs in their Thunderbay 4/8 devices rather than HDDs, there are some U2 things coming out now - not cheap. I can't say how good or reliable the former are as I don't use one, but I do use their Thunderbay 4 with 4x big HDDs and they work fine with their Softraid, I store my 174k raw images on there. They also do pre-built TB3 high capacity NVME drives, but they are expensive, very expensive! I did briefly think about storing my raws on NVME but decided against it, as I heard it didn't make a huge difference to real world speed.

To give you an idea on speed, I have a Mac Studio Ultra and LR uses all 20 cores when building previews, so it only takes a few minutes to build as I import my usual 1-2000 images, I usually just go get a coffee and it's done. Catalogue is on internal storage, as is my 275GB previews file. Moving between each image when culling in library mode is instant with 121s built and about a second if none available - it can actually depend on your display, mine slowed a bit when I moved from a 2k to a 6k one.

Hope that helps.

 

[gYab61zH]

I did not know about the OWC Express 4M2. I have just been reading some reviews. This could be something for me. Other suggestions would still be very welcome though.

 

[Victoria Bampton]

I sit there waiting for images to load properly in the library module (sometimes they do not load at all and I have to restart LR)

That sounds odd, I'd investigate that. What resolution are the photos?

Easy test - reboot, create a catalog on the internal SSD, import 100 photos of usual resolution while building standard previews. Then do the same again with the external drive. Time both tests and see how big a difference there is. I think you'll need to clear the camera raw cache (Preferences > Performance tab) and reboot in between to make sure nothing's cached.

 

[gYab61zH]

That sounds odd, I'd investigate that. What resolution are the photos?

Easy test - reboot, create a catalog on the internal SSD, import 100 photos of usual resolution while building standard previews. Then do the same again with the external drive. Time both tests and see how big a difference there is. I think you'll need to clear the camera raw cache (Preferences > Performance tab) and reboot in between to make sure nothing's cached.

Thanks for the excellent suggestion Victoria. I did the comparison and was somewhat baffled and surprised by the results. I imported 668 Canon R5 files into a test catalogue on my iMac. First on the internal SSD, then onto my (4x slower) Samsung 8Tb external SSD, rebooting between sessions. Importing only took about 3,5 minutes, but building the standard previews took over half an hour! Surprisingly, using the external Samsung disk made no difference. It is the actual building of previews that takes all the time; not the writing of them!

I then repeated the exercise using the same Samsung on my 2020 MBP (with M1). Again there was no material difference in speed between importing and building previews on the internal or external SSD disks; nor was the import any faster or slower than on the iMac, but the building of previews was! It was barely slower than the import. I then realised that the previews on a MBP are a lot smaller (2880 px) than those on the iMac 5K (5120 px), and repeated the test on the iMac but this time with the previews set to 2880 like on the MBP. This time the results were identical to those on the MBP.

While I was at it, I compared Topaz AI denoising and sharpening a 8192x5464 image on the iMac and the MBP, because I felt the latter was much quicker than the former. This turns out to be right. The MBP handled the image in a quarter of the time it took on the iMac (22 vs. 85 secs).

My sad conclusion is that if I want to improve the speed of my setup it is fairly pointless trying to improve the speed of the storage. What may well make a (huge?) difference is moving to an M1 system ... or be happy with 2880 previews.

PS. To save the other problems I have been having with LRC, I took the somewhat drastic step to make a fresh start, wipe the internal SSD and install Monterey, LRC etc from scratch. We'll see if it makes a difference. It did get rid of several years of clutter.

 

[Victoria Bampton]

Double check that Preferences > Performance > Generate Previews in Parallel is checked, that might help a bit.

 

[gYab61zH]

I did, it is.

 

[Victoria Bampton]

Depending on whether you have the panels open and how wide, the smaller previews may be plenty for your browsing needs.

 

[GregJ]

Hi,

First:
iMac Retina 5K, 27 inch, 2020; 3,6 GHz 10-Core Intel Core i9; 32Gb RAM (last of the Mohicans). Monterey.
Storage: 500 Gb internal SSD; external 8Tb Samsung 870 QVO SATA 2.5 Inch SSD

I spend many hours a day behind the iMac working with LRC. I used to be quite happy with my setup but after the last few LRC updates I am becoming more and more frustrated by its lack of speed: I sit there waiting for images to load properly in the library module (sometimes they do not load at all and I have to restart LR) and this slows down the selection of what to keep and what to abandon considerably; building Standard Previews takes a long time on Import and I don't even dare to build 1:1 Previews because my Previews catalogue is already 262Gb large (catalogue contains 162800 images). Topaz Photo AI is slow too, though it zips along quite nicely on my 13" M1 MBP (2020). Consequently I cannot keep the catalogue itself on my internal SSD so I am forced to save it together with the images on the external SSD.

It would be nice if I could speed things up a bit, but I have been wrecking my brains on how to achieve this. The easiest way to do it, I thought, would be to replace the Samsung SSD with an 8Tb M.2 SSD, but I cannot find an external case that will hold 2x4Tb NVMe M.2 cards nor do I know of a small NAS built around such cards. I guess I could move to a Mac Studio and a Studio Display but that adds up to well over 3000 Euros (i.e. a big invest so soon after getting the iMac).

Any suggestions on how to improve on this setup would be much appreciated.

Something else is going on here. I run windows and have that same 8TB SATA Samsung SATA SSD as my main data drive and LR is very fast. It has not slowed down with recent updates. You are saying that you have had this same setup for a while, are experienced with LR and it ran fine before the recent updates, right? Nothing has changed except the updates? Your connectivity is all the same and then it just dramatically slowed down lately? Nothing else different?
I always build 1:1 previews on import and keep them forever. So should you. You are a pro right?
Anyway, on import, I walk away and let the machine grind those imports. For example, last week I imported 800 huge (200 MB) Fuji GFX Medium Format files and built 800 1:1 previews. While those imports are building, I walk away because you can't do work on LR while those previews are building. It took about 10 or 15 minutes. Can't remember exactly and didn't time it, but let's say ten minutes.
You have a 5K retina display. You absolutely should be building 1:1 imports. You zoom in to full res when viewing in Library and when editing anyway right? S,o you are building them anyway.
You should not be running off of a 500 GB internal SSD. 1 TB is the minimum these days. 500SSDs have performance problems so you should be thinking about getting a 1 or 2 TB M.2 SSD for your boot. They are pretty cheap now but it is a pain to update boot drives.
Don't run raid and why do you want a NAS if you have an 8TB SSD and you are just working solo on your desktop? You have an 8TB SSD and can easily back up to cheap 8 TB HDDs. You can operate off of single disk and avoid all that NAA and especially RAID mess.
How full is that 8TB SSD? Mine has 6.2 TB on it and that is every shot I have shot in my life which is about a bazillion raw files.
Something else going on here. Should not be slow unless you are trying to work while previews are building, which you can't do. But that should not take too long. Get a cup of coffee and wait 5 minutes.
But something else is slowing you down. I'm not a mac guy, but this is something else besides LR. Is everything else running OK?
Youve done all the driver updating and cleaned out all your gunk right?

 

[GregJ]

Thanks for the excellent suggestion Victoria. I did the comparison and was somewhat baffled and surprised by the results. I imported 668 Canon R5 files into a test catalogue on my iMac. First on the internal SSD, then onto my (4x slower) Samsung 8Tb external SSD, rebooting between sessions. Importing only took about 3,5 minutes, but building the standard previews took over half an hour! Surprisingly, using the external Samsung disk made no difference. It is the actual building of previews that takes all the time; not the writing of them!

I then repeated the exercise using the same Samsung on my 2020 MBP (with M1). Again there was no material difference in speed between importing and building previews on the internal or external SSD disks; nor was the import any faster or slower than on the iMac, but the building of previews was! It was barely slower than the import. I then realised that the previews on a MBP are a lot smaller (2880 px) than those on the iMac 5K (5120 px), and repeated the test on the iMac but this time with the previews set to 2880 like on the MBP. This time the results were identical to those on the MBP.

While I was at it, I compared Topaz AI denoising and sharpening a 8192x5464 image on the iMac and the MBP, because I felt the latter was much quicker than the former. This turns out to be right. The MBP handled the image in a quarter of the time it took on the iMac (22 vs. 85 secs).

My sad conclusion is that if I want to improve the speed of my setup it is fairly pointless trying to improve the speed of the storage. What may well make a (huge?) difference is moving to an M1 system ... or be happy with 2880 previews.

PS. To save the other problems I have been having with LRC, I took the somewhat drastic step to make a fresh start, wipe the internal SSD and install Monterey, LRC etc from scratch. We'll see if it makes a difference. It did get rid of several years of clutter.

OK, I read this after I wrote my diatribe. If you are going to wipe that old 500 gig boot SSD, replace it with a 2TB as fast as you can get at whatever gen your MB allows. That is very immportant. No way you should be running on a 500 ssd. That goes for everyone else here too. You need the cat on that boot SSD and if you get a 2 TB boot, you will have plenty of room for the preview files at 1:1, but if not you can move those externally.
But you Mac guys can't tinker with your rig like I do, but suely you can replace the boot SSD and get a 2 TB latest gen your motherboard will allow.
Still, I don't know why your system would slow so dramatically if no other connectivity changes. It's not the updates. But how full is your 8TB SSD and how full is that inadequate 500 boot? Also, replace your cables that connect that samsung ssd. How long is that cable to your SSD? How old is it?

 

[Victoria Bampton]

I always build 1:1 previews on import and keep them forever. So should you. You are a pro right?

Ummmm, you realize those go "stale" the moment you edit the photos? At which point they're no help because they'd need updating before you use them.

For the majority of people, there's no need to keep 1:1 previews. They're only really useful when checking focus in the Library module in 1:1 view, and how many times do you do that after the first edit?

 

[Victoria Bampton]

But you Mac guys can't tinker with your rig like I do, but suely you can replace the boot SSD and get a 2 TB latest gen your motherboard will allow.

Nope.

 

[GregJ]

Ummmm, you realize those go "stale" the moment you edit the photos? At which point they're no help because they'd need updating before you use them.

For the majority of people, there's no need to keep 1:1 previews. They're only really useful when checking focus in the Library module in 1:1 view, and how many times do you do that after the first edit?

Victoria, come on now.... You know with a high-res monitor and with her 50 MB high res full frame raw files, she should be building 1:1 previews so that when she is culling in library mode she can get to full res quickly and cleanly. 1:1 previews don't hep editing bercause when you zoom in at all it builds then from scratch at 1:1 anyway, but it helps a lot in library mode.
I'm always telling photographers that they should be enjoying their own work at full res on 4K or 5K 32-inch IPS (or OLED, mini or micro LED) professional monitors and look at your images at full res. That is a joy.
I wish you were sitting with me right now looking at a full res GFX 100 shot of a townscape on my ASUS pro 32 inch calibrated 4K monitor. The image fidelity is astounding and knocks your eyeballs out. Sensational. I can see a dude sitting in a car window from a mile away zoomed to full res.
Anyway, if you are not doing that you are missing out on one of the joys of life.
Storage is cheap and fast. Use it to store 1:1 previews.
But why are all these Macs slowing down with LR? I want to know. Because I'm not slowing down and my rig is two years old.

 

[gYab61zH]

OK, I read this after I wrote my diatribe. If you are going to wipe that old 500 gig boot SSD, replace it with a 2TB as fast as you can get at whatever gen your MB allows. That is very immportant. No way you should be running on a 500 ssd. That goes for everyone else here too. You need the cat on that boot SSD and if you get a 2 TB boot, you will have plenty of room for the preview files at 1:1, but if not you can move those externally.
But you Mac guys can't tinker with your rig like I do, but suely you can replace the boot SSD and get a 2 TB latest gen your motherboard will allow.
Still, I don't know why your system would slow so dramatically if no other connectivity changes. It's not the updates. But how full is your 8TB SSD and how full is that inadequate 500 boot? Also, replace your cables that connect that samsung ssd. How long is that cable to your SSD? How old is it?

Hi Greg (I remember you from the various Fuji fora and read the other thread here about the Studio Ultra being slow). Thanks for the suggestions and as you saw I did get rid of all the old stuff by reinstalling the OS and starting from scratch. Replacing the boot drive is no option in this Mac, otherwise I would have done it long ago. I am considering getting a single card 4Tb M.2 thunderbolt drive to put LRC on. If I do, I will then switch to 1:1 previews. Things appear to have improved somewhat since the clean install, but it is early days yet. My SSD drive is just over half-full, so that should to be the problem either. The cable is fine and very short.

 

[GregJ]

Hi Greg (I remember you from the various Fuji fora and read the other thread here about the Studio Ultra being slow). Thanks for the suggestions and as you saw I did get rid of all the old stuff by reinstalling the OS and starting from scratch. Replacing the boot drive is no option in this Mac, otherwise I would have done it long ago. I am considering getting a single card 4Tb M.2 thunderbolt drive to put LRC on. If I do, I will then switch to 1:1 previews. Things appear to have improved somewhat since the clean install, but it is early days yet. My SSD drive is just over half-full, so that should to be the problem either. The cable is fine and very short.

Yes, I was a star on the DPR Medium Format Boards for many years as I pontificated to some of the best photographers in the world about what they should be doing. LOL.

But I banned DPR from my Universe over a year ago and have not posted there since for a reason I could tell you on a private message if you want. I basically got into a spat with one of their Nikon and Sony affiliated fan-boy staff members and a few Sont oriented trolls who hated Fuji and my enthusiasm for the wonderful GFX system, so I naturally took issue with that and decided to deny them my brilliant insights. Haha.

Anyway, I am not a mac guy but all this connectivity stuff is big news in the computer world. Everybody throw away all of their cables. Do it now. Order new USC c to c and a to c cables now, and make sure they handle the data load. The good ones are now labeled 1, 20, and 40 Gbps. If you are using a TB4 / USB/4 port and peripheral, you gotta pay attention to the cables and the length. Also, that 8TB SATA is way slower than an M.2 PCIe so it won't benefit from 40 Gbps. External SSDs that will are still very expensive, but I have my eye on an 8TB M.2 PCIe Gen 4 NVMe drive that has dropped to 1100 bucks from way higher. Still very expensive. I want my stuff on 1 drive backed up 4 times and I don't want RAID or NAS. But that's just me.,
Why are there so many mac problems with LR this past couple of months.
Is something going on or maybe you guys know it is time to upgrade.
You are getting hampered by that 500 boot SSD. How full is it? How full is that external 8TB SATA SSD?

 

[Victoria Bampton]

Victoria, come on now.... You know with a high-res monitor and with her 50 MB high res full frame raw files, she should be building 1:1 previews so that when she is culling in library mode she can get to full res quickly and cleanly.

Build them for culling, yes helps with speed if he/she zooms in when culling. Keeping them forever, however, does not have an advantage for most people.

 

[Victoria Bampton]

Things appear to have improved somewhat since the clean install, but it is early days yet.

That's good news. Spotlight may still be building in the background, so it may improve further yet.

 

[GregJ]

Build them for culling, yes helps with speed if he/she zooms in when culling. Keeping them forever, however, does not have an advantage for most people.

Victoria,
Yes, I can see that argument. What you are saying is practical and true for most people. But if storage is not a problem, it can be a benefit if you like to call up old images and view them at full res. I do that a lot. I might call up, a folder from Rome ten years ago and quickly view through the images.

Then if something jumps out at me, I might flip over to develop and re-edit the shot with a far better version of LR just for fun. I mean all of us can edit better now than we could ten years ago.

Plus, if storage gets to be a problem you can delete the previews very quickly in a couple of mouse clicks. Anyway, I generate the 1:1 previews on import so I can cull my images at full res. That is very important to me because my shots are extremely high res and I want to see that res. If I have camera shake blur or a focus problem, the shots is dead and gone.

So then the question becomes do I keep the 1: previews? Not a big deal really because I could generate them again fairly quickly for the bunch I might be looking at later. But I really don't mind having half a TB of 1:1 preview files.

But I think it is an interesting discussion and everyone has different needs (and for me - quirks).

 

[GregJ]

By the way ladies and gents. The SSD world is hot right now and prices are dropping significantly for 4 TB external SSDs that have very fast 20 Gbps connectivity (USB-C 3.2 Gen 2x2).

CES is going on right now in Vegas and all the big players are announcing new products as we read this. Today's release caught my eye because I am waiting for fast (non-SATA) external 4 to 8 TB SSDs. Sabrent announced the Sabrent Rocket nano V2 portable SSD. This type of SSD is really fast and great for photographers because it is perfect for sequential workloads such as media file transfers and backups (USB connections to external drives are traditionally not great at sustained workloads but getting better). All of the competitors to this drive are 10 Gbps USB-C 3.2 Gen 2, so this is twice as fast for our purposes when it comes to getting backup data to the drive or calling the file up from LR. Well.... Check that. I'm not so sure it will be twice as fast getting the image data to LR because Adobe has not quite caught up in that regard yet. But they will! I think in the dev module when LR grabs the image from the new 20 Gbps drive (or 40 with TB4), it is not using all of that yet and bottlenecks down. I think testers on this forum have been pretty clear about that.

So, imagine using this little rugged external drive to back up all your files onto one tiny disk and then slipping it in a pocket. A lot of you might even use it as a primary drive since a lot of posters on this board seem to be using extrnal 4 TB hard disk spinning rusters for their primary date storage drives. This would be about a thousand times better than that in every way.

Think about it.