LR & The Fastest SSD in the Consumer World

[GregJ]

Lightroom Version Number

Latest Version of Classic via Adobe Cloud

Operating System

1. Windows 11

There have been several threads lately that talk about using fast SSDs with LR. The one that got my attention the most (and probably Adobe's too), was the one about how LR develop module doesn't really load the raw file much (if any) faster from the fastest SSDs ( PCIe NVMe Gen4 M.2) connected to a very fast port (40 Mbps) than it does from a slow external 5-year-old external spinning hard drive connected to a much slower legacy USB port of the past.

This was interesting news to me in terms of how LR currently behaves when it comes to pinging at the raw file for the develop module. LR flies on my PC and laptop so I don't fret too much about LR loading from the stored raw, but it is interesting....

As the Gurus said, LR does an outstanding job of limiting the times it has to ping back to the raw file that is probably stored on a separate data disk internally or externally. LR has all kinds of creative ways to get the raw files you are using loaded into various caches and ram as we are working in the develop module so it doesn't have to ping back to the raw much. I'm sure Adobe is working on ways to continue to make better use of the new GPUs with huge amounts of uber-fast memory. They are probably working on ways to make better use of the fastest SSDs and the fastest connectivity (like TB4 and USB4). I assume that Adobe is doing this.

But here is the point. As of today, the absolute fastest PCIe NVMe Gen4 M.2 SSD in the world available to consumers at 107 bucks for the 1 TB model and 204 bucks for the 2 TB model. I won't say the name of the company or the chip but it is easy enough to find out. The point is that SSD prices are finally falling, and availability is increasing while SSDs just keep getting better and faster (note PCIe Gen 5 SSDs that will hit the market in September). This is great news for photographers and users of Adobe.

I believe that the vast majority of photographers out there can fit all of their raw files with a lot of room to grow on a 2 TB SSD. So, it just became much cheaper to boot off of the fastest 1 TB SSD in the World, and store all of your image files on the fastest 2 TB SSD in the world. Then you can back up your images easily to an external 2TB SATA SSD for pretty cheap or to an external hard disk for very cheap. And I'm talking laptop or PC.

Will it have an impact using LR? Yes absolutely. It will have an impact on your entire photography and Adobe work-flow, not to mention the rest of your computer-use tasks, especially if you are also a gamer, video shooter or do 20 types of other tasks on your computer).

I'm talking about the vast majority of photographers out there that seem to post here, and certainly the ones who dwell on the photography boards. They can live on a 1 TB SSD to boot and a 2TB SSD for data. If you are like me or any advanced enthusiast or pro and you need way more, then get ready for 8TB prices to drop where you can do the same thing too. Currently, M.2 SSD is 1400 bucks for 8 TB and 700 bucks for SATA TB. That will drop by 35% this year.... Just watch. It is going to happen.

When it does, the trash cans will be piled high with old spinning hard drives of 8 TB and less.

 

[Conrad Chavez]

If the problem is that Lightroom Classic is not properly optimized in this area, it would be good to know about the alternatives that do it right. Which other raw photo processing applications read and write original files at the maximum transfer rates of the best SSDs, like over 3000MB/sec, and what is their actual measured throughput?

 

[BobT]

I have two 1TB NVMe M2 drives in my machine. I think LrC is plenty fast enough but yes, I was surprised it wasn't faster. I'm also surprised that Adobe is not taking advantage of Intel's 12th Gen processors with more powerful integrated graphics. In fact, contrary to their earlier stance, they seem to now favour powerful add-on GPU cards and effectively dismiss integrated graphics. This is disappointing, since I feel that integrated graphics is the future for all but gamers and data miners, but what would I know. The one good thing is that with the chip supply improving and data mining collapsing, GPU prices should come down.

 

[GregJ]

A few points Bob,

1. GPU prices are coming down dramatically right now as I type this, especially at the top-end. That is good for not only gamers, video editors and content producers. It is good for photographers too. It helps in a lot of ways to have a nice recent GPU with your windows system, or a latest Mac M1/M2 chip with Apple. Your Alder Lake CPU with the latest integrated graphics as fine as well. Newer is better ion this arena.

2. You are using your integrated graphics more than you think you are with Adobe - especially if it is 12th generation Alder Lake. I don't know that for a fact, but I suspect it. Remember, that fast boot SSD you have where LR and the Catalog are located, and that fast SSD where your raw files are located are working for you hard in LR in many ways, especially on Export, where with version 11 of LR, it uses both CPU and GPU, whereas it did not before. Now for generating previews, that does not use the GPU and it is very CPU-intensive. The fast SSD helps though because it is writing those files to disk where the previews are stored and then it is reading them out in the library module.

3. I don't fret too much on the point about LR loading the raw files when you first click on them in the develop module at the same speed with fast internal SSDs as external spinning drives. That seems odd and I didn't believe it at first, but I do now. It is probably something they are working on. All software struggles to keep up with the latest gear tech.

The best at it are the big-selling games, but Adobe has made some big strides in this area as well.

 

[GregJ]

If the problem is that Lightroom Classic is not properly optimized in this area, it would be good to know about the alternatives that do it right. Which other raw photo processing applications read and write original files at the maximum transfer rates of the best SSDs, like over 3000MB/sec, and what is their actual measured throughput?

It would be interesting, but if the competition did that part better, there is probably somewhere else LR kills it. Nothing beats LR at what it does (in my opinion).

 

[MrFixit]

This is largely why I still maintain my very large collection of RAW files on a NAS with just a 1GB connection. As long as my Catalog is on my NvME SSD, Lightroom and the Develop Module work plenty fast enough for me (retired amateur).

Importing is perhaps a little slower than I would like, as the RAW files need to be sent to the NAS over the 1GB connection. But then I just go and make a cup of tea or two! I may consider a 10GB connection upgrade which will help there.

 

[Conrad Chavez]

It would be interesting, but if the competition did that part better, there is probably somewhere else LR kills it. Nothing beats LR at what it does (in my opinion).

Of course. But the reason I asked the question is to isolate that one thing, and see if any other commercial or noncommercial/open source raw processing apps have achieved, or at least made a priority of, matching the maximum read rate of storage. If none of them do it, and if a lot of them read originals at about the same rate as Lightroom Classic…that tells us something. Like, for example, that maybe maximum read speed would expose many other bottlenecks that are not so easily resolved.

I am not a programmer, but I am thinking of things like, if you do have an app that reads raw files at the maximum storage transfer rate, can the GPU and CPU demosaic and render the incoming files to previews/ thumbnails at a rate equal to how quickly they are being read; and if they can’t keep up, how soon do you run out of available RAM to cache the growing backlog of files read but waiting for processing? For example, if there is a photo app that can read in originals at 3000MB/sec, then for raw files that are for example 100MB, 30 more original raw files would be read every second. To not hold up the line, every second that app would have to fully close out the demosaicing, rendering, and writing of all 30 raws to previews/thumbnails, before the next 30 come in during the next second. If it can’t keep up, the read rate has to slow down.

Basically, I’m curious whether Lightroom Classic is typical or an outlier in how fast it reads from storage. Because if it’s actually typical across the industry for a photo app to not max out the storage rate when reading originals, maybe there’s a good reason for that, and that would be the next question we’d want to answer. What have the programmers identified as the highest priority bottlenecks to resolve for faster performance, and how high is storage speed on that list?

 

[GregJ]

I hear you. I'm curious too.

I bet Adobe is working on it because it makes zero sense to me. How is it possible that there is no difference in LR performance level (initial load of the raw in the dev module) with the fastest SSD currently available to mankind on the fastest lane available commercially now, and on the fastest port one can buy or build vs the slowest external spinning hard drive on the slowest port one can still find? That can't be.... But apparently it is. See the thread that was locked and you will see several LR guys say it is indeed so. I believe them. I denied it at first. But I believe them.

 

[PhilBurton]

There have been several threads lately that talk about using fast SSDs with LR. The one that got my attention the most (and probably Adobe's too), was the one about how LR develop module doesn't really load the raw file much (if any) faster from the fastest SSDs ( PCIe NVMe Gen4 M.2) connected to a very fast port (40 Mbps) than it does from a slow external 5-year-old external spinning hard drive connected to a much slower legacy USB port of the past.

There is always a challenge for software vendors about how much to add new featres that benefit only current year hardware or are willing to spend significant bucks to pggrade their systems. (and have the skills to do so.)

.

But here is the point. As of today, the absolute fastest PCIe NVMe Gen4 M.2 SSD in the world available to consumers at 107 bucks for the 1 TB model and 204 bucks for the 2 TB model. I won't say the name of the company or the chip but it is easy enough to find out.

Why not post a link?

The point is that SSD prices are finally falling, and availability is increasing while SSDs just keep getting better and faster (note PCIe Gen 5 SSDs that will hit the market in September). This is great news for photographers and users of Adobe.

SSD prices like allsemiconductors prices typcally fall over time due to competiton an increased manufacturing yields. Typically. Been that way since the days of 64 kb memory chips with "insect legs." Here is an early example of a PC addin board for memory. https://www.ebay.com/itm/165571776324

I believe that the vast majority of photographers out there can fit all of their raw files with a lot of room to grow on a 2 TB SSD. So, it just became much cheaper to boot off of the fastest 1 TB SSD in the World, and store all of your image files on the fastest 2 TB SSD in the world. Then you can back up your images easily to an external 2TB SATA SSD for pretty cheap or to an external hard disk for very cheap. And I'm talking laptop or PC.

Will it have an impact using LR? Yes absolutely. It will have an impact on your entire photography and Adobe work-flow, not to mention the rest of your computer-use tasks, especially if you are also a gamer, video shooter or do 20 types of other tasks on your computer).

I'm talking about the vast majority of photographers out there that seem to post here, and certainly the ones who dwell on the photography boards. They can live on a 1 TB SSD to boot and a 2TB SSD for data. If you are like me or any advanced enthusiast or pro and you need way more, then get ready for 8TB prices to drop where you can do the same thing too. Currently, M.2 SSD is 1400 bucks for 8 TB and 700 bucks for SATA TB. That will drop by 35% this year.... Just watch. It is going to happen.

When it does, the trash cans will be piled high with old spinning hard drives of 8 TB and less.

 

[GregJ]

Thanks Phil. I didn't mention the SSD brand or name because I wasn't sure of the rules here on this forum for that and didn't want people to think I was promoting a company or specific item.

You talked about all semiconductor prices falling over time - of course, but it is especially impactful and newsworthy now because of the supply interruptions of the past 3 years, chip shortages across the bard, manufacturing problems, crypto mining demand and all kinds of other factors that were keeping prices artificially high and supplies low. That is all getting better now and SSDs are about to roll off the lines at unprecedented numbers.

And yes, I'm sure Adobe and all software companies struggle with staying abreast with the latest hardware of all types, and Adobe is doing pretty well at it. But Phil, M2 PCIe 3 and 4 SSDs and the newer faster ports (TB4 / USB4 40 Mbps, and even USB 3.2 Gen 2 (5 and 10 Mbps) ports are not something that could have possibly surprised them.

 

[BobT]

I have two 1TB NVMe M2 drives in my machine. I think LrC is plenty fast enough but yes, I was surprised it wasn't faster. I'm also surprised that Adobe is not taking advantage of Intel's 12th Gen processors with more powerful integrated graphics. In fact, contrary to their earlier stance, they seem to now favour powerful add-on GPU cards and effectively dismiss integrated graphics. This is disappointing, since I feel that integrated graphics is the future for all but gamers and data miners, but what would I know. The one good thing is that with the chip supply improving and data mining collapsing, GPU prices should come down.

To qualify what I said above, with a USB 3 port, cable and card reader, LrC is actually lightning fast at uploading files. The delay is in the processing afterwards. It's not lack of resources. The CPU runs at about 50%, memory 25% and GPU at a paltry 2% yet still it takes a few minutes to complete the process. Not a problem but it's as if there's a choke point somewhere stopping the full use of resources.

 

[PhilBurton]

Thanks Phil. I didn't mention the SSD brand or name because I wasn't sure of the rules here on this forum for that and didn't want people to think I was promoting a company or specific item.

I'm not aware of any such limitation. I believe that I have used the names Western Digital and Seagate a number of times with no pushback from any of the moderators. I suspect that blatant promotion or disparagement might be an issue. But "moderate" comments, I think those would be accepted.

You talked about all semiconductor prices falling over time - of course, but it is especially impactful and newsworthy now because of the supply interruptions of the past 3 years, chip shortages across the bard, manufacturing problems, crypto mining demand and all kinds of other factors that were keeping prices artificially high and supplies low.

True, the pandemic and all its various impacts have certainly roiled semi-conductor manufacturing and affected numerous industries. Certainly crypto mining has an an impact on GPU card availability, but I think I've read that nVidia has had record production in recent quarters. It's the damn miners who have created market shortages. It might then cheer you to learn that crypto currency values have cratered recently.

I'm not aware of serious manufacturing problems. I'm sure that the newest 7mm and 5 mm processes have the usual startup yield issues, but that's to be expected.

That is all getting better now and SSDs are about to roll off the lines at unprecedented numbers.

What evidence is there of that, particularly for larger capacity chips?

And yes, I'm sure Adobe and all software companies struggle with staying abreast with the latest hardware of all types, and Adobe is doing pretty well at it. But Phil, M2 PCIe 3 and 4 SSDs and the newer faster ports (TB4 / USB4 40 Mbps, and even USB 3.2 Gen 2 (5 and 10 Mbps) ports are not something that could have possibly surprised them.

The struggle that software companies have is not technical in a strict sense. It's defining the "profile" of the system in use by customers. USB 3.2 ports are transparent to Adobe software, at least on Windows.
Thunderbolt is generally available only on Mac systems. I know someone who has a custom built Windows system that includes native TB support on the motherboard. My 2020 custom built system (ASUS Ryzen 9/X570 chipset) has a motherboard without that support and I was not foresighted enough to spend several hundred dollars extra for a board with that support (in 2020).
If I do want to get an adapter card just for TB support, it will cost me probably $150. That is too much, and I suspect that few PC users will upgrade at that cost. But please appreciate the point that TB and USB support is built into the operating system, and not in application software such as Lightroom. If the customer is running on an older version of Windows or MacOS, then some of that hardware support will be absent.

The system hardware that is at issue for application vendors like Adobe is the CPU, memory, and GPU. If Adobe spends lots of engineering dollars and creates features that requires (as opposed to benefiting from) top speed CPU (e.g. AMD "Threadripper", 64 or even 128 GB of RAM and a top of the line GPU, then those features won't be of value to the vast majority of customers. Also true to some extend for large, very fast SSDs. And there will be less money available to features that benefit the majority of customers.

Adobe needs to engineer features that work well on more normal hardware, as defined by the "profile" of customer systems. Set the profile too low, and it's hard to introduce new or improved features. It's a complex process to make these decisions. (Been there, done that.)

 

[GregJ]

I'm not aware of any such limitation. I believe that I have used the names Western Digital and Seagate a number of times with no pushback from any of the moderators. I suspect that blatant promotion or disparagement might be an issue. But "moderate" comments, I think those would be accepted.


True, the pandemic and all its various impacts have certainly roiled semi-conductor manufacturing and affected numerous industries. Certainly crypto mining has an an impact on GPU card availability, but I think I've read that nVidia has had record production in recent quarters. It's the damn miners who have created market shortages. It might then cheer you to learn that crypto currency values have cratered recently.

I'm not aware of serious manufacturing problems. I'm sure that the newest 7mm and 5 mm processes have the usual startup yield issues, but that's to be expected.


What evidence is there of that, particularly for larger capacity chips?

The struggle that software companies have is not technical in a strict sense. It's defining the "profile" of the system in use by customers. USB 3.2 ports are transparent to Adobe software, at least on Windows.
Thunderbolt is generally available only on Mac systems. I know someone who has a custom built Windows system that includes native TB support on the motherboard. My 2020 custom built system (ASUS Ryzen 9/X570 chipset) has a motherboard without that support and I was not foresighted enough to spend several hundred dollars extra for a board with that support (in 2020).
If I do want to get an adapter card just for TB support, it will cost me probably $150. That is too much, and I suspect that few PC users will upgrade at that cost. But please appreciate the point that TB and USB support is built into the operating system, and not in application software such as Lightroom. If the customer is running on an older version of Windows or MacOS, then some of that hardware support will be absent.

The system hardware that is at issue for application vendors like Adobe is the CPU, memory, and GPU. If Adobe spends lots of engineering dollars and creates features that requires (as opposed to benefiting from) top speed CPU (e.g. AMD "Threadripper", 64 or even 128 GB of RAM and a top of the line GPU, then those features won't be of value to the vast majority of customers. Also true to some extend for large, very fast SSDs. And there will be less money available to features that benefit the majority of customers.

Adobe needs to engineer features that work well on more normal hardware, as defined by the "profile" of customer systems. Set the profile too low, and it's hard to introduce new or improved features. It's a complex process to make these decisions. (Been there, done that.)

Phil, you sound like a Man who has worked in the industry and has a lot of experience with this. Interesting post.

I'm a PC hobbyist and keep up with the gear and tech in order to plan my next PC build. But I am knowledgeable of emerging laptop and desktop trends and am a camera and computer gearhead.
Anyway, what you said about Thunderbolt being mostly Mac? Not anymore. For the past two years, Thunderbolt 3 ports, via USB-C has been on almost all mid to upper end Windows laptops, and TB3 contained the latest generation USB tech as part of its protocol. I have a two-year-old Dell XPS 15 and it has two Thunderbolt 3 ports. Those TB3 ports include as part of the protocol USB 3.2 Gen 2 and DisplayPort. Wonderful capability.

Now what you said is true (lacking thunderbolt) on building or buying a PC the past few years. But in the last 3 years, all mid to upper-level Motherboards have had plenty of latest gen USB ports (5, 10 and even 40 Mbps) that are many times faster than the (what we used to call) old USB 3.0 ports. Motherboards are going to have USB 4.0, especially with the emergence of the 13th gen Raptor Lake CPUs. Adobe of course knows this.
TB4 is hitting in waves right now. TB4 always is a USB-C form and TB4 always has USB4 (and DisplayPort) as part of its protocol. In fact, it could be argued that TB4 is USB4 (although USB 4 is not always TB4 because USB4 can have a 20Mbps version). Anyway, it is now the standard. I'm sure Adobe is cranking on making best use of this new connectivity landscape.
They have to be if they are really still plodding along with raw load up speeds at old spinning hard drive on a USB 3 port speed.

 

[Victoria Bampton]

I'm not aware of any such limitation. I believe that I have used the names Western Digital and Seagate a number of times with no pushback from any of the moderators. I suspect that blatant promotion or disparagement might be an issue. But "moderate" comments, I think those would be accepted.

Yeah that's fine. If someone started dropping a brand name into every other post, we'd take issue with it. But just saying what you use and like when the topic arises is fine.

 

[GregJ]

Speaking of external SSDs and the late generation very fast USB ports, I was reading the new issue of Max PC this morning and "The Doctor" was talking about the SSD madness that is confusing everyone about write speeds. He said something like this (I summarize and adlib a bit from my own knowledge):

"The USB Bus, whether it is USB 3.2 Gen 1 (5 Gbps), 3.2 Gen 2x1 (10 Gbps) 3.2 Gen 2x2 (20 Gbps) or USB 4 (20 Gbps or 40 Gbps) of course is the huge factor determining how fast data can be transferred on these new external SSDs, but there are many other factors at play. The onboard USB controller and the speed of the hardware bus your USB ports are on also play a significant role. It is also true that large file write speeds are most often much faster than when copying a whole bunch of small files. Manufacturers rarely reveal all of their nuts and bolts and tech tricks when it comes to the mix of all this tech. Then there are factors like the type of Nand and support for TRIM or UASP that can greatly affect write speeds in different situations and types of tasks."

It's all good because we are getting some awesome SSD and connectivity tech for cheaper (see the Samsung T7 for an example of NVMe external SSDs at USB 3.2 Gen 2 for cheap.... But one can understand how a big software giant like Adobe has to work hard to keep up with all the hardware advances to maximize LR and PS speed on a huge variety of tasks with files big and small.