Metadata slowed to a crawl

[GrahamP]

Lightroom Version Number

Lightroom Classic version: 11.4.1

Operating System

1. Windows 10

I am wondering whether I have reached the limit of metadata that Lightroom Classic can handle. I am running the latest Lightroom version (11.4.1) on Windows 10 Pro, i9 Intel processor, 48GB RAM, NVIDIA GeForce GTX 1060 6GB (drivers fully updated), Lightroom catalogue on a dedicated internal SSD, all images on local hard drives (drives D: to I: each a separate physical disk (not merely a partition), none on the C: Windows drive).

I have 800,000 images, almost all Canon RAW files, 27,000 keywords and synonyms. My catalogue is backed-up and optimised once a week on exiting Lightroom (and has been re-optimised outside this schedule). The catalogue size is just under 26GB.

Until recently, perhaps until the update to version 11 or 11.1 (but I am not entirely sure this was the start of the issue), adding keywords to images in the catalogue was as good as instantaneous. Recently, however, this has become so slow as to make Lightroom almost unusable. Type in an existing keyword, click a keyword suggestion or select a keyword in the keyword list, all are the same: nothing happens immediately, it takes from several seconds, sometimes even several minutes (even for one or a single digit number of images), for the keywords to update in the list of tags, and until it updates other actions, such as export, are not available (if the drop down menu does appear and allows you to click an action, it can take seconds or minutes for that command to be implemented, waiting for metadata writing to finish). Often as each keyword is added, it shows as a new task - just now I added 7 keywords to a single file, which showed as 7 pending operations. Sometimes a small type message appears at the top left under the menu bar "writing XNP for [n] files"

I have deleted most of my smart collections. I have a smart collection which reports metadata conflicts or pending metadata updates, and both are reporting zero images in this condition.

All my images are on local drives. I have ensured that the drive on which my images for this year are located has more than 20% free disk space. All my drives are regularly optimised or defragmented using a Windows system weekly schedule. My images are organised into folders and subfolders with relatively small numbers in each and generally I do not select Show Images in Subfolders, so the display is not overwhelmed with large numbers of images. Standard low quality previews have been built for all images. The catalogue settings are for XMP to be written to sidecars, develop settings are not written to image files.

All my hard disks and my RAM memory modules check out fine using diagnostic software. I am not aware of any system bottlenecks slowing writing XMP files to disk (I relocated a group of images from one drive to another, to see if the write times were any better, and they were not, so it is not a drive specific problem).

Attached to my computer are a Wacom Intuos Pro M and a Loupedeck +, both with fully updated drivers. The problems occur whether or not I have other applications running: I am satisfied I have eliminated other applications or new hardware being the cause, nothing having materially changed since the problems started.

More transient problems, which may or may not be related, are that sometimes the wrong image is selected (e.g. I change the colour label on image [n] in a folder, but Lightroom changes the colour label of an entirely different image, and that the Wacom driver stops responding and has to be restarted. More rarely the Loupedeck driver has to be restarted.

Usually the problem gets worse with use, and can sometimes be alleviated by restarting Lightroom, which suggests to me some sort of Lightroom memory leak.

Has anyone else had similar problems, and if so, does anyone have a suggested cure? Or have I reached the limit of the combined numbers of images and keywords that Lightroom can handle?

Any thoughts or suggestions would be gratefully received.

Graham

 

[Gnits]

I make a point (every 18 months or so) of exporting my catalog as a new catalog, creating a brand new empty catalog and then reimport the recently exported catalog into the empty new catalog. Select the options such that the existing images stay in their current location, especially with your volumes. Do a few test runs on a small sample subset of your images a few times, so you get familiar with the options presented.

I do this ....... to try and give my catalog a fresh start. Tables and indices are rebuilt from scratch and any accumulated junk in there gets flushed. This can happen due to bugs in Lr, but more typical if you have other apps or power issues crashing your system.

Also, make sure that any antivirus software is specifically excluded from scanning your catalogs or previews or images. Sometimes Window updates or upgrades may cause previous exclusion settings to be reset.

Finally, it may be worth considering removing the jpgs from raw+jpg combos, if you are unlikely ever to reuse those jpgs, just to reduce the size of your catalog.

 

[GrahamP]

Thank you for the quick and helpful reply. Recreating by importing sounds a very good idea to try: I always import my travel photos from my laptop after I return using the import from catalogue command and that has always worked well, but I have never tried it for my entire main catalogue before. I will report back whether this improves matters.

I have a very few JPG+TIFF combinations from scans, but no JPGs derived from RAW files: I keep my exported JPGs on a separate computer and outside the main catalogue. My AV is already configured to ignore my catalogues, previews and the image files themselves, but again a great suggestion for anyone who has not already done this.

Graham

 

[Gnits]

This is also an opportunity of starting all previews and caches from scratch, further reducing the possibility of legacy junk. I cannot remember if you will have the option to keep old previews, if so I suggest not to.

 

[hanoman]

i make a similar observation on win10. The external usb-drives stop spinning after a while if there is no work and they need several seconds (sometimes 10-20 sec) to become operational. I did not know wether there is a parameter in the windows setting which influences this behavior.

 

[Gnits]

I have a Thunderbolt enclosure, which holds all my images except those for the current year and a really fast M2 SSd internal drive, which stores my current years images. I have noticed, from time to time, when I access an image on the M2 drive, there is a distinct pause of several seconds while the Thunderbolt enclosure fires up. There is no obvious reason why the Thunderbolt enclosure should crank up, but my whole system waits for this to happen.


If you have grown your 800k organically by incrementally adding external drives over time, then it may be time to consider putting all of your images into one or two large internal drives

I have noted this behaviour, but do not have the tools or the skills to establish the root cause.

Also, while most people drool over impressive read and write performance stats of the latest M2 drives, people often pay no attention to a) time for disks or enclosure of disks to come up to speed. or b) performance is usually the burst read or write mode, which often degrades exponentially for extended read or write behaviour, or when hardware caches or buffers fill up. Again, my experience of sustained I/o with fast hardware has always been slower than my expectation.

Approx 9 months ago I installed the fastest M2 drive, for my active images, I could get into a rig with the latest Pcie standards and was really disappointed to see no noticeable improvement with the interactive nature of Lr.

This is an area on the edge of hardware, o/s, database and application capability and any generalisations are dangerous. I am just providing some feedback on some real world scenarios.

 

[johnrellis]

There have been a small number of reports in the Adobe forums about extreme slowth in LR 11 making changes to metadata, and these reports didn't call out unusually large catalog sizes.

The various versions of LR 11 have made changes to the handling of metadata that may be responsible for what you're observing. LR 11.0 changed the timing of how it wrote .xmp sidecars and displayed the pausable progress bar in the upper-left. LR 11.3 made a significant architectural change to how metadata is displayed in the Metadata panel, and it had a number of obvious bugs indicating it was poorly tested -- these bugs impacted a large number of people, and Adobe quickly fixed most of them, but at least a couple have only been recently discovered.

Particular troubleshooting steps:

1. Try turning off Catalog Settings > Metadata > Automatically Write Changes Into XMP (your description indicates it's currently turned on). Whether you keep it on is a big-endian/little-endian religious issue (I keep it on), but it would be good to know if it's the proximate cause of your problems.

2. How many keywords do you have in your keyword list? If you're not sure, do Metadata > Export Keywords and open the exported text file in a text editor to count the lines. On Windows, when you have more than about 1500 - 3000 (depending on your screen resolution and settings), you'll experience some of the symptoms you're describing. (It's possible you're tripping over more than one problem.)

 

[Gnits]

John,
Complements on your analysis . I am not yet running the latest version, so do not have a good grasp of recent release notes, which seem very relevant.

 

[GrahamP]

Thank you everyone for your helpful replies and suggestions. I have marginally improved, but not solved, the problem of LR's slow writing of metadata.

Recapping or responding to some of the queries here:

• I have approaching 800k images in a single catalogue, almost all Canon RAW images of various generations, a relatively small number of JPGs and some DNGs (panoramas and HDRs)
• Approx. 27,000 keywords, including synonyms
• All images on 6 internal hard drives, the catalogue and previews on a separate dedicated internal SSD
• Catalogue set to automatically write to XMP (essential for compatibility with other applications)

These are the steps I have tried/checked:

• import old catalogue into new clean catalogue:
  • did not work - Lightroom eventually crashed after importing all images. The total number of images seemed correct, but metadata obviously was not (I could see immediately that numbers of keywords were missing from the images I checked). Probably the catalogue was simply too big to import into a new catalogue.
    • there is no option to use previews from the catalogue used for import, but I would not have used them anyway
• cleared out all caches and deleted all previews, which forced LR to read each XMP file or image with embedded metadata and build them again.
  • took many days to complete!
  • forces LR to read each image again and recreate metadata, hopefully eliminating cataloguing database issues
• deleted Helper.lrdata folder, reset all preferences
• monitored all processes used in updating previews and ensured that all were excluded from anti-virus on-access scanning (all image folders and LR processes similarly excluded)
• made sure all metadata conflicts were resolved and metadata was fully updated (I set up smart collections to monitor metadata conflicts and metadata updates to check for this)
• Scanned for and removed from the catalogue all missing images
• checked all drivers were up to date, including Wacom, Loupdeck and Nvidia
• optimised catalogue at the end of all of this and of course restarted Windows

Did all this solve the problems? No. Did this improve performance: yes, a little.

Initially, I thought the problems had gone away. Keywords seemed to be added quickly and metadata updated as expected. But after only a few minutes' use, it slows down again. Copying metadata between images, such as synchronising keywords and captions (even between a single digit number of images) remains unacceptable slow. I just tried adding 8 keywords to half a dozen files, and saw 8 separate updating XMP progress bars at the top left of the screen. I tried syncing the metadata across a few more files and the progress bars did not show, but right clicking to export did not work to bring up a menu until LR had caught up with itself writing the metadata. Check a box to add a keyword in the Keyword List and there is still a several seconds wait for the tick to appear.

Conclusion: LR metadata handling is broken. Certainly it is not longer useable for my purposes.

The problem lies with LR, not my system. I have for a long time also been using Photo Mechanic Plus, which does not suffer from any of these metadata problems, but I have treated my LR catalogue as my main catalogue, partly due to inertia and partly because I do most of my processing in LR (rather less in Capture One) and it is convenient to be able to export processed JPGs from LR with flattened non-hierarchical keywords for stock purposes. In terms of catalogue and metadata management and usability, PM+ is now way ahead of LR and, unlike LR, it just works. From now on, I am going to treat my PM+ catalogue as my main catalogue and use LR only for its Develop module, forcing LR to update the metadata for the folders in which I will be developing images in readiness for export.

LR has done little if anything to improve the library module and its handling of metadata over the years (still not even a spell checker - so basic, and as John Ellis has pointed out, it has recently made this worse with new bugs (which it does not seem intent on fixing as a matter of priority) whereas PM+ is being actively developed and maintained all the time.

Thanks again to everyone for their suggestions. I have wasted a lot of time this week trying to fix LR, but at least it has had the benefit of confirming what my new workflow has to be and given me a chance to practice using it how I expect to be using the two programmes together going forwards.

Graham

 

[Gnits]

Well done for completing such a comprehensive exercise. I rebuilt / recovered an 800k catalog for a friend who lost 300k images due to a disk failure, which took me approx a week to complete, with some processes taking 24-36 hours, so I have an appreciation for the effort involved.

I would suggest that you check Lr again after the next point upgrade and definitely after the next major upgrade. Adobe may optimise some of the mechanics under the database covers.

I am pleased to hear of your PM feedback. My experience with PM was quite the opposite (many years ago). My use of the PM Next Seq No was critical to my file naming schema and on multiple occasions PM reset this number to 1, with devastating consequences for my image repository, always discovered too late to recover from. I abandoned PM at that point. I have also received feedback from other sources that PM have
rebuilt from scratch their internal engine.

The fact that the import of the catalog into an empty catalog fails, is indicative to me, that there is some type of data issue within the original catalog. If so, this may rear its head again at a future date.

Eliminating missing files is a good step. If I had your issue I would probably checkout cases where duplicate filenames exist, or use of virtual files. Unfortunately, Adobe does not provide useful tools to audit internal issues within the catalog.

 

[johnrellis]

I just tried adding 8 keywords to half a dozen files, and saw 8 separate updating XMP progress bars at the top left of the screen. I tried syncing the metadata across a few more files and the progress bars did not show, but right clicking to export did not work to bring up a menu until LR had caught up with itself writing the metadata. Check a box to add a keyword in the Keyword List and there is still a several seconds wait for the tick to appear.

Do these bad symptoms occur after disabling the option Catalog Settings > Metadata > Automatically Write Changes Into XMP? Some of the changes in LR 11 were to how LR writes XMP metadata in background (even though there no reports of problems with that in the past many years).

If disabling that option eliminates or reduces the bad behavior, then a workaround would be to keep it off. Periodically, use the Library Filter bar with a filter preset or define a smart collection to search for photos whose metadata status is Has Been Changed:

Select all the displayed photos and do Library > Save Metadata To File.

 

[johnrellis]

Approx. 27,000 keywords, including synonyms

Separate from the other symptoms you're experiencing, there's one known misbehavior with that many keywords on Windows LR. When you start typing into the Filter Keywords box of the Keyword List, LR can lockup for a minute or two before it displays some of the matching keywords:

The only workaround is to type the initial search word very quickly (or copy/paste it) (or to move to Mac :-< ).

When you type just a letter or two, slowly, LR tries to display all the matching keywords and their containing keywords. If that list is longer than about 1500 - 3000 keywords (depending on your display and display settings), LR will lock up.

Keeping your keywords in a hierarchy with most of the hierarchy collapsed doesn't help in this case, since the hierarchy is expanded in the filter results.

 

[GrahamP]

Separate from the other symptoms you're experiencing, there's one known misbehavior with that many keywords on Windows LR. When you start typing into the Filter Keywords box of the Keyword List, LR can lockup for a minute or two before it displays some of the matching keywords:

Yes, every time I do this (which is all to often) it is a real "DOH" moment!

I have not tried turning off the write to XMP option because I need this, and refreshing metadata to pick up metadata not written to XMP takes forever (I have collections to flag unwritten or conflicting metadata, but these are too slow to help. I may try it later out of curiosity, but in the meantime I will detail the solution I have found in my next post.

Graham

 

[johnrellis]

I have not tried turning off the write to XMP option because I need this

It would be good to do a temporary experiment of turning off Automatically Write Changes Into XMP to see if that is indeed the proximate cause of your issues. That would allow an actionable bug report to be filed with Adobe. My hypothesis is that with the changes introduced in LR 11, they seriously broke this part of the metadata infrastructure. Without a precise finger pointing at the cause, Adobe will ignore the issue.

 

[GrahamP]

This is my solution (for now) to the slowing down of my master Lightroom catalogue and its corresponding awful performance with metadata updates, particularly keywords, and also sometimes unresponsive develop functions.

My goals are to :

• use Photo Mechanic Plus as my main catalogue, including for import (ingest) and keyword management;
• be able to use Lightroom (as well as other RAW converters such as Capture One) to develop photos for export of JPGs from RAW files including flattened keywords and synonyms;
• as far as possible, keep my master LR catalogue in sync with the PM+ catalogue.

The issues to overcome include:

• PM+ will store and write keywords in hierarchical format, but it has no facility to export only flat keywords including their synonyms (there is a GREP solution I have posted on the PM+ forums, but let's ignore that here).
• Lightroom will only allow editing of images which are in a LR catalogue, it will not function as a stand-alone processor/editor.
• While PM+ can send or drop images into Lightroom which are not already in a LR catalogue, this does not work for images already in that LR catalogue (particularly if the "do not import suspected duplicates" is selected, and you do not want to make second copies).
• While changes made to metadata outside of PM+ (written to XMP) are immediately recognised in PM+, the reverse is not the case for Lightroom: LR will (eventually) detect a conflict and ask whether you want to overwrite using the LR catalogue metadata, or import the changed metadata from disk. While this can be checked from a suitable smart collection or metadata filter, it has become too painfully slow on my system to be viable.

This is therefore what I am now doing:

• My master database is that in PM+. I am using PM+ to ingest new images and to apply the keywords I want. PM+ preferences has a setting to maximises compatibility with Adobe hierarchical keywords, which works well.
• My keywords list in PM+ is kept in sync with that in LR (and vice versa): the keyword list with any material changes made in either can be exported as a text file and imported into the other (but this is not often necessary - syncing metadata as described below will also sync keywords between the programmes).
• I have created a new, empty LR catalogue, with the latest imported keyword list. It is set to write metadata to XMP automatically.
• Having identified the images I want to process in PM+, I send them to LR (either by choosing edit in, or drag and drop - I am using Windows 10). This opens the LR import dialogue. The images are then imported in place, with no develop settings or additional metadata.
• I then edit each image in LR ready for export. Since there are only these few newly imported images in the new LR catalogue, it runs very quickly, without any of the lags in developing or metadata handling that started me on this process. I can make keyword or caption changes in LR, and check that all selected images have the same keywords (which is not always easy to do in PM+, which does not have the equivalent of an asterisk to indicate that a keyword is contained in some, but not all, of a selection of images). As each image is exported as a JPG, I then change the RAW file label colour and add the keyword I use to indicate that an image has been processed and exported.

Any metadata changes in LR are reflected in PM+ straight away. These will eventually be reflected in my large, slow master LR catalogue, including develop settings. The LR master catalogue can either be checked for metadata changes later (choose import from disk when prompted), or for more immediacy the new small catalogue can be imported into the master catalogue using "import from catalogue", or if all the images were in one folder, this folder can be synchronised in the main catalogue, choosing the update metadata and develop settings option. In this way, the PM master catalogue can be kept up to date and synchronised with what has been done in the new small catalogue without too much effort.

Once that has been done, the images can be removed from the new catalogue (taking care to select remove from catalogue, not delete!), leaving it empty for next use (and thus keeping it small, light and nimble).

In summary, all I am doing is bypassing the huge, sluggish LR master catalogue in such a way as to allow me to develop images and modify metadata in LR within a timescale which can be measured with a clock rather than a calendar. I am using LR+ for its keyword handling, but you do not have to do this. In principle, you could select the images you want to work on with any image browser (such as Faststone Image Viewer), drag them to import them into a new temporary catalogue, then update the metadata in the slow main catalogue, so this process might assist others who are saddled with a very slow large LR database but who do not want to use other database management applications.

Hope that helps or is of interest.

Graham

 

[GrahamP]

It would be good to do a temporary experiment of turning off Automatically Write Changes Into XMP to see if that is indeed the proximate cause of your issues. That would allow an actionable bug report to be filed with Adobe. My hypothesis is that with the changes introduced in LR 11, they seriously broke this part of the metadata infrastructure. Without a precise finger pointing at the cause, Adobe will ignore the issue.

Will try this when I have a little more time.

 

[johnrellis]

Have you considered using Bridge instead of LR? You could edit directly in Camera Raw (same capabilities as LR, slightly different UI), and Bridge and Photo Mechanic will immediately read changes to XMP made by the other. Seems like you're going to a lot of work to keep LR "alive" (barely), while Bridge could work more smoothly for you.

 

[GrahamP]

It has been some time since I started this thread. In summary, I want to keep Lightroom to use its Develop module, which has improved considerably with its masking abilities. Yes, I can do the same in Photoshop, but only if I create a large TIFF or PSD file. An attraction of Lightroom is that edits are stored in the catalogue at minimal disk space cost. I want to do this even if the catalogue can no longer usefully be used for metadata because it has become too slow.

Having concluded that my old catalogue was beyond use for keywording, I completely built a new catalogue, very time consuming for 735,000 images, mainly RAW files, including building standard previews for each. I wanted to divorce the new catalogue from my old catalogue so far as possible, so as not to carry over any corruption issues (if there were any). I did, however, need to export all virtual copies from the old catalogue to a new catalogue, which I then imported into my new master catalogue, in order to retain my virtual copy edits which would otherwise have been lost. Strangely, this export/import process somehow deleted all the captions and keywords from the images and virtual copies imported into the new master catalogue - hey ho, another bug, it would seem.

The new catalogue is 12.8GB, compared for the old catalogue which was 27.3 GB for the same images (actually slightly less images, the new catalogue now contains a few thousand more images than the old). Maybe the difference in file size indicates that there was some corruption in the old catalogue?

Sadly, though, none of this has solved the slow metadata issue. The new catalogue is as slow as the old in adding keywords. I have turned off "Automatically Write Changes Into XMP" so that metadata edits are stored only in the catalogue, but it still takes an inordinate amount of time for added keywords (existing or new) to appear in the list of keyword tags. I have just tried assigning 6 existing keywords and one new keyword to a small group of images (112) and the result was 6 pending tasks shown on the top left, gradually ticking down. The keywords did eventually complete their appearances some minutes later. I did not time the tasks, but by the time I had logged into this site, found this thread and written the first couple of paragraphs of this post (and I am not a fast typist!) the tasks were still there pending. They have since completed.

Does any of this help with filing a bug report to Adobe? I can now definitely confirm that in a new catalogue the slow writing of keywords happens whether or not "Automatically Write Changes Into XMP" is turned off or on.

Graham

 

[johnrellis]

One more experiment before filing a bug report: Put 100 images in the Quick Collection and select the Quick Collection, so there are just 100 images showing in Grid view. Does the bad behavior still occur?

LR 11 reworked how metadata was displayed in Library for "performance", and a lot of bugs were introduced. I'm wondering if this is related.

 

[Denis de Gannes]

I have a completely different view on this issue.
Adobe offers two options to manage your digital raw images,
1. Bridge / Camera Raw which reads and writes metadata to the file by default.
2. Lightroom Classic which stores info in the Catalog file by default and allows users to avoid having to deal with sidecar files.
If your workflow, includes using sidecar files and info stored in the image file then use the first option.
I have decided to utilize the second option, so in my workflow I utilize the second option. I do not “Automatically write Metadata to xmp” not do I use the DNG file format. If I need to share info with others I know how to do that. If I need to utilize Photoshop options I use the “edit in option” to share data.