Lightroom Faster > Faster

Henk

Member
Joined
Jan 26, 2016
Messages
30
Lightroom Experience
Advanced
Lightroom Version
Because I own a very fast iMac with 64GB RAM, I still experience some slowness in the Develop module.

Other programs don't suffer and go very fast.

I understand there are a lot of things have to be done after editing before you see a full screen after you have pushed the F-key. It takes 4-5 seconds after doen cropping to full screen.

Question:
Does Lr during editing communicate with the RAW file, which I have on an external disk. Or does Lr do this independent of the external disk, because I imported the images into Lr.

If yes Lr communicates, then you must have a very fast connection (Thunderbolt) an a SDD disk OR doesn't it matter?

Though, with the help of a Thunderbolt connection and a SSD external disk where my images are stored; getting very fast a full screen is THE solution! ??
 
Joined
Jul 2, 2015
Messages
10,991
Location
Netherlands
Lightroom Experience
Power User
Lightroom Version
When you go to a raw image in the develop module, Lightroom has to load that image into RAM. At that point, it does indeed have to communicate with the external disk. A fast connection (Thunderbolt) and a fast disk (SSD) will indeed help to speed that up. After the image has been loaded, there shouldn't be any activity to that external disk, unless you write the changes to XMP.
 
Joined
Jan 18, 2009
Messages
2,427
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Do you keep your catalog on the local fast SSD, or is it also on the external disk? The catalog (and unless you move it the preview caches) are used a bit in develop, and generally should not be on the external drive for reasons of safety, but for reasons of size some people keep them there.

I also find the develop "ready" delay the most annoying aspect of Lightroom in day to day use. While it varies by camera, for example a D800's 36mpx image takes almost 8-10 seconds to be "ready". To my experience it is primarily CPU speed that affects that, at least for me.
 

Henk

Member
Joined
Jan 26, 2016
Messages
30
Lightroom Experience
Advanced
Lightroom Version
Thanks Johan, however I did test with 400 RAW images stored on my local SSD disk,and I still find ita slow proces.

Thanks Ferguson. My catalog and the previews are on my local 1 TB SSD, very fast, my archive with the RAW 36 mpx images of my Nikon D810 are stored on a simple external 2TB disk.

Again my average opening to full screen time is about 4 seconds if the images are placed on the local disc. See above.

Sometimes, especially after a while, very busy with adjustments, it comes over me as if things are going slower in the proces.
 
Joined
Jun 20, 2009
Messages
17,658
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
When you go to a raw image in the develop module, Lightroom has to load that image into RAM. At that point, it does indeed have to communicate with the external disk. A fast connection (Thunderbolt) and a fast disk (SSD) will indeed help to speed that up. After the image has been loaded, there shouldn't be any activity to that external disk, unless you write the changes to XMP.
Actually, LR only works with RGB images. A JPEG is already RGB, a RAW file is not. ACR creates an RGB image from the RAW. This is stored (temporarily) in the ACR Cache location. This default location is the primary local drive.
There are times in Develop when LR needs to render a new 1:1 Preview. When this happens LR first is sent to ACR cache to retrieve the RGB image created by ACR. If ACR no longer retains the RGB image, then LR calls ACR to create a new RGB image from the original RAW file. So it is not always a simple matter of where you keep your master RAW image files. Unlike JPEGs which contain an RGB image, there can be an additional delay when LR need to retrieve a fresh RGB image from the RAW.
 
Joined
Jul 2, 2015
Messages
10,991
Location
Netherlands
Lightroom Experience
Power User
Lightroom Version
What Lightroom shows you is not the image itself, but a preview. Lightroom stores previews for viewing in the library module, but uses a different one in the develop module. That preview needs to be rendered to reflect any changes. Increasing the Camera Raw cache size (in the preferences) may help, but it will never be an instant process.
 

Henk

Member
Joined
Jan 26, 2016
Messages
30
Lightroom Experience
Advanced
Lightroom Version
Cletus wrote: So it is not always a simple matter of where you keep your master RAW image files.
Johan wrote: A fast connection (Thunderbolt) and a fast disk (SSD) will indeed help to speed that up.

I am a little bit confused now. Though what is then the best solution to get things faster done.
I own yet the fastest iMac possible and I quote Simon Chen who wrote to me "LR loves RAM" I have 64GB.

I hope you both respond.
 
Joined
Jan 18, 2009
Messages
2,427
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
"Loves ram" is true, but at 64G you are well above the range where any more will help. While we could debate if it's 8, 12, 16gb there's a limit above which LR doesn't gain much speed, and it's well below 64GB.

When you open a raw file from a slow drive there is a need to read it once. That delay is mitigated by having it on a faster drive, but reading 50meg from a slow drive vs fast might add half a second to your time at a guess? Depends on the drive. Try something like this -- copy 100 of your images from the slow disk to the fast disk and time it, divide by 100 and you'll have an estimate of the max delay it adds (ok, there are variables here like buffer capacity, write speed of the SSD, etc. but the idea is to get an estimate).
 
Joined
Jul 2, 2015
Messages
10,991
Location
Netherlands
Lightroom Experience
Power User
Lightroom Version
I think Cletus and I both made a point. When Lightroom needs to load the raw data from disk, the speed of that disk matters. When it doesn't need to load it from that disk, but from the ACR cache, the speed of the internal disk matters. In both cases RAM and the speed of RAM matters. And CPU speed matters. And the GPU speed matters (if Lightroom can use the GPU). There are always things that take time. Patience, grasshopper...
 

Henk

Member
Joined
Jan 26, 2016
Messages
30
Lightroom Experience
Advanced
Lightroom Version
Do I understand it well, does LR always or sometimes load during Developing (editing) from the disk where the RAW image is stored in this case an external Disk.Or does it load most of the time from the ACR cache, then there should not be a delay.
Do you mean with ACR cache the Camera Raw Cache which you can adjust in Preferences and Camera Raw Cache Settings? I adjusted it to 200GB.

Cletus I do a time check.
 

Henk

Member
Joined
Jan 26, 2016
Messages
30
Lightroom Experience
Advanced
Lightroom Version
Stopwatch: 100 images of 78 MB each: 79 seconds. Though average per image: 0,79 seconds! Is this slow Cletus?
 
Joined
Jul 2, 2015
Messages
10,991
Location
Netherlands
Lightroom Experience
Power User
Lightroom Version
Loading always causes some delay. Yes, we are talking about the Camera Raw cache. The first time you open an image in the develop module, it has to be read from the external disk. When you open it a second time, it can be loaded from the cache. I'm not certain what data are stored in the cache exactly. I assume it's linear (demosaiced) RGB-data and the preview. Perhaps Cletus can tell you more about that.
 
Joined
Jun 20, 2009
Messages
17,658
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Cletus wrote: So it is not always a simple matter of where you keep your master RAW image files.
Johan wrote: A fast connection (Thunderbolt) and a fast disk (SSD) will indeed help to speed that up.

I am a little bit confused now. Though what is then the best solution to get things faster done.
I own yet the fastest iMac possible and I quote Simon Chen who wrote to me "LR loves RAM" I have 64GB.

I hope you both respond.
LR does not keep derivative images apart from the previews of various sizes. These previews are temporary. LR renders a new image from the original data AND the instructions stored in the catalog each time you throw a new image on the screen. "LR loves RAM" means that algorithms rendering a finished image on the screen from the original RGB image data and the LR instructions will work faster if you have more RAM to temporarily store the results of the computations Multiple cores (up to about 6) can be used by LR to process instructions in parallel faster than processing those same instructions serially. Whenever LR needs to create a temporary file (either as a temp file in working storage or as a swap file to make better user of a limited RAM capacity), the process becomes I/O bound. Thus you can improve performance if your process can avoid I/O bound operations with more RAM or if your I/O operations can be speeded up with a faster I/O path (Buss mounted SSDs versus spinning platters on a slow USB or ethernet EHD).
So, both statements relate to improving performance. Previews are a file stored in the Previews folder. Any time LR needs to render a new one (every Develop Change, exports, prints) It needs a current RGB image to work with. If you have a JPEG, LR can open and read the JPEG file to apply your develop instructions to render an up to date image on the screen. If your original is a RAW file, LR need a RGB image to work with. This is always taken from the ACR cache files. If ACR cache does not contain the requested RGB image file, then ACR needs to open the original RAW file demosaic it and render a fresh RGB image file to be stored in ACR Cache. All or these operations require I/O processes that can slow things down if the file needed to be read or written is located in a slow hardware path. If you want to tune LR for the best performance, you need to be aware of the I/O constraints on all of the files generated or consumed by LR. These include the original master image file, the ACR Cache file (if used), working storage (LR creates lots of temporary intermediate files) and LR previews.
 
Joined
Jun 20, 2009
Messages
17,658
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Stopwatch: 100 images of 78 MB each: 79 seconds. Though average per image: 0,79 seconds! Is this slow Cletus?
I have both a D800E & D810. I am well aware of the magnitude of data needing to be processed just to render an initial image from a D810 NEF. Until two days ago my processing was done on a late 2011 iMac with 16 GB of RAM. Two Days ago I received a late 2015 27" Retina 5K iMac with 32GB of RAM. I have not yet imported my first image with the new set up.

Here are the setting that I use for import to speed up the import process:
  1. I import to the local buss mounted drive (now a Fusion drive)
  2. On import, I build only minimal previews. LR will always generate new previews as needed. I judge it a waste of time to generate large previews on newly imported images that are likely to be deleted as soon as they are first viewed large.
  3. I don't generate Smart Previews. Smart previews are generated by LR as necessary for those images in collections that are Sync'd with LR Mobile.
These three things minimize the time needed to complete an import. Still, I don't think I can import 100 images in 79 sec. I expect my new iMac to be faster but I've been comfortable with the LR performance of the old iMac. I haven't really had an opportunity for see if LRCC 2015.6 with its new predictive background renewing is develop is faster but I expect it to be.
 

tspear

Senior Member
Joined
Sep 23, 2014
Messages
1,825
Location
Waltham MA
Lightroom Experience
Beginner
Lightroom Version
I found the new 2015.6 update in develop to be faster the majority of the time on my recent trip to Ireland. In a few areas:
  1. The grid display when jumping down multiple pages is much faster/smoother while also rendering previews for the images displayed on the screen faster. It used to be you could "see" that Lr was still working on the images which had already be passed a couple of pages ago.
  2. Facial guessing is much more accurate and faster.
  3. Switching back and forth between images in Develop was a lot quicker when comparing just a few images. (I am guessing the 1-2 images before/after the current image is the limitation)

Tim
 
Joined
Jan 18, 2009
Messages
2,427
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Stopwatch: 100 images of 78 MB each: 79 seconds. Though average per image: 0,79 seconds! Is this slow Cletus?
So on the initial develop it takes 8 tenths of a second to read. If it was on SSD it would not be zero, so let's pretend it was 0.1 or 0.2 seconds there, so maybe 0.6 seconds or so of your 4 seconds mentioned is coming from the external drive on the first open.

I am a bit surprised by the comment above that on re-open in develop it does not access the raw, but I did some experimenting and indeed it does not (open 1:1 in develop, close; move the NEF, do it again it doesn't complain -- do the same with one not recently opened it complains).

Now this is where I get confused -- if I build 1:1 previews that doesn't happen. The 1:1 preview doesn't apparently populate the cache the same way, and doesn't satisfy the develop module's need for the NEF.

So if you go into develop it doesn't need the NEF again, but if you build 1:1 previews and then go into develop it does still read the NEF.
 
Joined
Feb 1, 2010
Messages
12,951
Location
West Sussex, UK
Lightroom Experience
Advanced
Lightroom Version
Do I understand it well, does LR always or sometimes load during Developing (editing) from the disk where the RAW image is stored in this case an external Disk.Or does it load most of the time from the ACR cache, then there should not be a delay.
Do you mean with ACR cache the Camera Raw Cache which you can adjust in Preferences and Camera Raw Cache Settings? I adjusted it to 200GB.

I think a little clarity is needed here. It's not a case of Develop loading EITHER the ACR cache entry OR the original raw file....in most circumstances it will do both!

The ACR cache entry for a raw file is created whenever you have Lightroom render a standard or 1:1 library preview, it is stored in the ACR cache which by default will be on the system drive (but can be user-changed). The size of the cache can also be user set, and it works on the basis of the oldest will be removed if the cache is full in order to store the latest entries. The cache entries will typically be in the 500kb to 1mb range, so a 20gb cache will hold around 20-40,000 entries.

So unless you only create minimal previews on import, the ACR cache entry will have been created before you try to open it in Develop. In that situation the Develop file opening process is as follows:

1. Read the Library preview, which is the first thing you will see, but at this stage sliders are not yet activated.
2. Read Smart Preview. If you haven't rendered Smart Previews, LR will open either the ACR Cache entry OR (for DNG files only) the Fast Load Data if it has been rendered. The ACR Cache entry and the DNG FLD are basically the same thing, but as the latter is stored in the DNG file, reading that could be slower if for example the original files are on a network or slow speed external drive. For files which have already been edited the preview from the SP/ACR?FLD entry will first be updated into the system cache with any existing develop edit instructions. This preview will be the second thing you see.
3. Whilst the above is going on, LR will also be reading and converting the original raw file (+ applying any existing develop edits) to create the "scene referred" full-size Jpeg preview which is the last thing you will see in this opening process, and which is stored in the system cache during the edit process.

You'll probably not notice much difference between the first two steps, but at the end of step 2 the develop sliders will be activated.

It is possible, if you look closely, to determine a change when the final stage preview is loaded and displayed on screen.

My take on the above is that having the smart previews (if you've created them) or the ACR cache on fast drives is good. Having the original files on fast drives is far less important (in Develop terms), as the transition to the final stage Develop preview will be almost seamless so it probably won't matter if it takes a few seconds longer. But it might matter if the original files are now DNG and Fast Load Data is being created.

Of course all of the above pre-dated the new performance improvement in LR6.6, which could mean that loading the next image is virtually instant.

Hmmmm....maybe on reflection I didn't bring that much clarity, but it's important to understand what goes on "under the hood" if you're considering file placement options/strategies.
 
Joined
Jan 18, 2009
Messages
2,427
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Jim, how do I reconcile that with what I saw in testing:

- Scenario #1:
1) Go into develop and let it finish rendering to full size, exit to library
2) Move the NEF (and in my case XMP) elsewhere
3) Go back into develop -- it shows up and appears happy, no warning, appears full resolution

- Scenario #2:
1) Do not (recently) go into develop
2) Move the NEF (and in my case XMP) elsewhere
3) Go back into develop -- it shows the image, but it warns "Cannot open file" at the top, clearly trying to open the NEF

- Scenario #1:
1) Do NOT go into develop (recently)
2) Render 1:1 preview in library.
2) Move the NEF (and in my case XMP) elsewhere
3) Go back into develop -- as in case #2, it complains "Cannot open file".

It would appear that the 1:1 preview did not satisfy the need for the raw, but the recent entry into develop did? So with regard to your point (3) above, is this related to the pre-caching in 2015.6 and not needing the NEF?

What I didn't try was exiting LR entirely between moving off the NEF and trying to open it again, which would flush any memory cache.

But to me it's disappointing as you can build 1:1 previews in batch, but there appears no mechanism to build ACR (or whatever) cache that is used for develop in batch to make it faster. Is there?
 
Joined
Feb 1, 2010
Messages
12,951
Location
West Sussex, UK
Lightroom Experience
Advanced
Lightroom Version
In your scenario #1, I would have expected the same "missing file" problem as per your other scenarios. However, it's quite likely that the develop preview would still be cached at the point you switched back to it, which could explain why LR didn't complain. That's nothing to do with the loading sequence....perhaps I should have made it clearer that the process outlined is for when there isn't already an entry in the system cache (which is why the 6.6 performance enhancement changes things quite significantly).

Re you last point, sure there is...render either standard or 1:1 previews which you can do in batch, and that'll take care of any missing ACR cache entries. I don't know what percentage of users do not render either standard or 1:1 previews on import, but for all those that do the ACR cache entries are done at the same time.
 

Henk

Member
Joined
Jan 26, 2016
Messages
30
Lightroom Experience
Advanced
Lightroom Version
More and more an interesting discussion.

Yesterday the Journalist of the Century in the Netherlands passed away at the age of 88 years, his name is Hofland.
His credo was :"always be curious". And that's what I am.

I agree with Ferguson: "
But to me it's disappointing as you can build 1:1 previews in batch, but there appears no mechanism to build ACR (or whatever) cache that is used for develop in batch to make it faster. Is there?

Your point of view is very interesting Jim:
Hmmmm....maybe on reflection I didn't bring that much clarity, but it's important to understand what goes on "under the hood" if you're considering file placement options/strategies.

Cletus:
I have had the same Mac and I have the same camera. I am convinced you get the same results as mine, try it. But I think there must be something under the hood to get things faster.
With the update of LR 6.6 you see things get worse (slower), look at all the forums on the internet. It is also my experience. Wish I was the CEO of Adobe.

All in all again, what does Adobe think and what do you think to get things going faster.
 
Joined
Jan 18, 2009
Messages
2,427
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Not sure if this is beating a dead horse, but I did a more extensive test as follows:

1) Closed lightroom (and photoshop is not open)
2) Deleted all files in the photoshop raw cache
3) Deleted all files in the lightroom preview cache
4) Opened lightroom and
a. (It opened briefly on a gallery with 3 shots so it probably built some previews there).
b. On photo #1 in a gallery of about 160 shots, built 1-1 previews.
c. On photo #50 in the same gallery opened develop and zoomed 1:1 and waited until everything stabilized​
5) Closed lightroom
6) Noticed (not sure this is relevant but I found it interesting):
a. 19 photos in the ACR cache, all about the same size.
b. 75 files in the LR Preview cache
i. 33 tiny (under 100kb) - thumbnails I presume
ii. 4 between 1 and 2MB
iii. 13 between 2 and 3MB
iv. 10 between 3 and 4MB
v. 10 between 4 and 5MB
vi. 2 between 5-6 MB
vii. 1 at 7MB
viii. 2 at 9-10MB​
c. Note: I can't really see how the 19 in ACR map to these in any obvious way, nor do the counts really make sense, there seem to be too many small ones (I did nothing to generate standard or other small size other than thumbnail). But...​
7) Moved the original NEF's for both of the two above (shots #1 and #50 in the gallery)
8) Open lightroom
a. Opened file #1 in develop - got "The file could not be found" and it would not zoom beyond "fit"
b. Opened file #1 as full screen in library -- zoomed 1:1 -- it zoomed just fine.
c. Changed to grid and moved to #50, clicked it and develop
d. Same as file #1 - warning message and would not open beyond "fit".​
9) At this point I have not opened #51 in any manner other than as a thumbnail.
a. With LR still open I moved file #51's raw file out of the directory.
b. I clicked on #51 and then Develop - no error message, and file opened fully zoomed 1:1​

So...

a) It appears that neither preview cache nor ACR cache is adequate to get back into develop fully -- it still has only a low res shot to show.

b) It appears that the caching in 2015.6 pulls something that is adequate into the (system in memory?) cache, so a missing raw file gives no error (and obviously implying no need to read the raw file to develop).

So I'll come back to my original comment -- there appears to be no adequate way to pre-build whatever kind of preview is needed to get into develop quickly, i.e. the kinds that occur in adjacent photos as you edit and move, you can only do it one by one inside develop?
 
Joined
Feb 1, 2010
Messages
12,951
Location
West Sussex, UK
Lightroom Experience
Advanced
Lightroom Version
a) It appears that neither preview cache nor ACR cache is adequate to get back into develop fully -- it still has only a low res shot to show.

b) It appears that the caching in 2015.6 pulls something that is adequate into the (system in memory?) cache, so a missing raw file gives no error (and obviously implying no need to read the raw file to develop).

So I'll come back to my original comment -- there appears to be no adequate way to pre-build whatever kind of preview is needed to get into develop quickly, i.e. the kinds that occur in adjacent photos as you edit and move, you can only do it one by one inside develop?

Re point a) No they're not, they just set the stage more quickly whilst waiting for the full-res preview to be rendered.

Point b) sounds like it could be a bug.

I'm really not understanding your last point here. Your original point was about pre-building ACR cache entries, but now you seem to be talking about something different? The "kind that occurs in adjacent photos"....I assume you mean the full-res develop preview pre-loaded into system memory? In which case how would you see "pre-building" of those working?

BTW, the 19 ACR files are probably the result of Lightroom undercover activity. As you deleted the preview cache, and purged the ACR cache, as soon as you open Library, LR will start to silently build Standard previews for all the images showing in either the Grid or Filmstrip....and as I said earlier when LR builds standard or 1:1 previews, it'll also build ACR cache entries if none yet exist.
 
Joined
Jan 18, 2009
Messages
2,427
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Re point a) No they're not, they just set the stage more quickly whilst waiting for the full-res preview to be rendered.

Point b) sounds like it could be a bug.

I'm really not understanding your last point here. Your original point was about pre-building ACR cache entries, but now you seem to be talking about something different? The "kind that occurs in adjacent photos"....I assume you mean the full-res develop preview pre-loaded into system memory? In which case how would you see "pre-building" of those working?

BTW, the 19 ACR files are probably the result of Lightroom undercover activity. As you deleted the preview cache, and purged the ACR cache, as soon as you open Library, LR will start to silently build Standard previews for all the images showing in either the Grid or Filmstrip....and as I said earlier when LR builds standard or 1:1 previews, it'll also build ACR cache entries if none yet exist.

Building 1:1 previews in library seems to me to have no or little speed-up effect when you go into lightroom. I think you are saying it should but I do not see it (the above did not try to test for it admittedly).

If I pick a random shot with no recent develop activity and without 1:1 previews built and open it in develop it takes me about 6 seconds (D5 size).

If I let LR sit and work (watching task manager) until it finishes whatever it is doing (presumably caching the adjacent shots), then I go to one of those adjacent shots, it takes about 3-4 seconds, or just a bit over half as long. So the system cache (if that's the proper word) is beneficial, if not magic.

If I pick a random group of shots (again, no recent activity) and build 1:1 previews, let them finish, then select one of them and go into develop it is back to about 6 seconds.

Now it's always difficult to precisely time an interactive program where in principle it could be doing other "stuff", my experience is that 1:1 previews do no good whatsoever in going into develop.

I would like some way to take (say) 100 shots and say "go pre-process these and load into memory so I can go into develop and have less wait".

Caching them a few ahead does me no good, as it still takes too long -- it can't keep up with what I want to change and move on (usually just tiny tweaks in exposure and temperature). I just absolutely HATE that 4-10 seconds I have to wait before I can move the sliders and see the result as I flip between shots.
 
Joined
Feb 1, 2010
Messages
12,951
Location
West Sussex, UK
Lightroom Experience
Advanced
Lightroom Version
1:1 Previews have NO effect in Develop, other than to give you something to look at while the full loading process goes on in background (but a standard preview would be just as effective in this situation). The purpose/benefit of 1:1 previews is to allow 1:1 viewing quickly in the Library module.

If I open Develop and start to move through a set of 22mp files, the speed at which the sliders are enabled (which is the point that I consider the file sufficiently loaded to start working on it) will vary depending up how much editing (and what type) has already been done. If there are no edits, i.e. a new batch of imports, slider activation is pretty well instant. For most developed images, slider activation will be around 1 to 2 seconds, for extreme developed images it can be somewhat longer.
 
Top