Library module Is LR performance impacted by number of keywords, or complexity of keyword hierarchy?

[PhilBurton]

Lightroom Version Number

Lightroom Classic 8.0

Operating System

1. Windows 10

I'm thinking of buying the Controlled Vocabulary Keyword Catalog. The URL expands to a long sentence: Controlled Vocabulary: your site for information on Keyword, Hierarchical Classification, Thesauri, Taxonomy and Subject Heading systems used to describe images in databases (Thesaurus, facet classification, hierarchy). It contains 11,000 keywords organized into about 25 top-level categories. That's a lot of keywords, and I'm certain that I'll have to add many additional keywords to capture important detail for a particular hobby subject area. Is this number of keywords going to slow down LR performance? Would it be beneficial to lop off top-level categories such as Agriculture that I probably won't use.

Phil Burton

 

[clee01l]

Keywords is a table in the LR database. It is indexed so that retrieval is quick and efficient. There is (or used to be) a list limit in Windows that prevents the keyword panel from displaying more than ~2048 items. Long lists won’t affect LR. Performance.

The Controlled Vocabulary is so large that you will find that you in reality will only use a small subset. I considered the Controlled Vocabulary and ultimately desired it was cheaper to make my own specific subset tailored to my image subjects.

 

[LouieSherwin]

I also started with David Riecks Controlled Vocabulary Keyword Catalog (CVKC) many years back when I first started using Lightroom 3. At the time it was one of the only hierarchal keyword sets available and had the broadest scope of categories. I have not experienced any performance problems with Lightroom itself. The real challenges have to do with the implementation, usage and maintenance of your keywords.

Using the CVKC will provide a big assistance on the implementation because as you have noted a lot of the organizational work is provided with the list. You simply download it and import into Lightroom and all 11,000 terms are ready to use.

There are some things to keep in mind regarding hierarchal keyword in Lightroom. First the hierarchy only exists inside Lightroom. In fact Export explicitly flattens the hierarchy and puts them into a completely different IPTC field. Regardless I find using hierarchies useful in two important ways.

The first is as a shortcut to apply multiple terms at a time. For example "poppy" also gives me "flower" and "plant". The second is organizational. A keyword hierarchy can imply some kind of organization. I mostly use this mostly for more detailed description of location. Take a trip to Hawaii for example. The IPTC Location fields are simply insufficient to capture the full location information. So the term "Halawa Bay" also gives me "Halawa Valley", "Molokai" and "Hawaiian Islands".

On the usage side, Lightroom is much better now than it was in V3. The search in the Keywords List will now return the complete hierarchy of any term that contains the search term. For example if you search for "flower" it will show you all the hierarchies that have flower as part of the hierarchy. On the down side it you cannot search for terms with imbedded spaces such as "San Diego". Nor can you search for aliases. Both are pretty bad over sites.

The workaround for both of these issues is John R Ellis's AnyTag plugin. I have it setup as Ctl-T in my Lightroom keyboard shortcuts and typically go there first when keywording. You can search, enter and create keywords all from one dialog.

The maintenance side is really the most challenging. Editing and organizational tools are limited. You can drag and drop terms to make or adjust the hierarchy but there is not a search and replace. You can search but have to individually open and edit each term.

Also Each keyword has "Keyword options" such as "Include on Export". child terms will inherit this option from the parent when creating a new child term. However, if you forget to set it correctly or simply want to change a whole hierarchy to have a different option, as above you have to edit each term individually.

If you really do and want to make wholesale changes that are simply too tedious to accomplish with the limited tools in Lightroom you can simply export your keywords and Lightroom creates a tab delimited text file that you can edit with any text editor. I used an application called OmniOutliner that was able to also handle the hierarchy as implied by the number of tabs which made the process easier. I don't believe that you could drag and drop in the Keyword List then as you can do now.

Even with these limitations I feel that the benefits of using hierarchical keywords out ways the restrictions. Although no matter which keyword list you start with be prepared to take the time to tailor it to your own personal needs.

-louie

 

[PhilBurton]

I also started with David Riecks Controlled Vocabulary Keyword Catalog (CVKC) many years back when I first started using Lightroom 3. At the time it was one of the only hierarchal keyword sets available and had the broadest scope of categories. I have not experienced any performance problems with Lightroom itself. The real challenges have to do with the implementation, usage and maintenance of your keywords.

Using the CVKC will provide a big assistance on the implementation because as you have noted a lot of the organizational work is provided with the list. You simply download it and import into Lightroom and all 11,000 terms are ready to use.

There are some things to keep in mind regarding hierarchal keyword in Lightroom. First the hierarchy only exists inside Lightroom. In fact Export explicitly flattens the hierarchy and puts them into a completely different IPTC field. Regardless I find using hierarchies useful in two important ways.

The first is as a shortcut to apply multiple terms at a time. For example "poppy" also gives me "flower" and "plant". The second is organizational. A keyword hierarchy can imply some kind of organization. I mostly use this mostly for more detailed description of location. Take a trip to Hawaii for example. The IPTC Location fields are simply insufficient to capture the full location information. So the term "Halawa Bay" also gives me "Halawa Valley", "Molokai" and "Hawaiian Islands".

On the usage side, Lightroom is much better now than it was in V3. The search in the Keywords List will now return the complete hierarchy of any term that contains the search term. For example if you search for "flower" it will show you all the hierarchies that have flower as part of the hierarchy. On the down side it you cannot search for terms with imbedded spaces such as "San Diego". Nor can you search for aliases. Both are pretty bad over sites.

The workaround for both of these issues is John R Ellis's AnyTag plugin. I have it setup as Ctl-T in my Lightroom keyboard shortcuts and typically go there first when keywording. You can search, enter and create keywords all from one dialog.

The maintenance side is really the most challenging. Editing and organizational tools are limited. You can drag and drop terms to make or adjust the hierarchy but there is not a search and replace. You can search but have to individually open and edit each term.

Also Each keyword has "Keyword options" such as "Include on Export". child terms will inherit this option from the parent when creating a new child term. However, if you forget to set it correctly or simply want to change a whole hierarchy to have a different option, as above you have to edit each term individually.

If you really do and want to make wholesale changes that are simply too tedious to accomplish with the limited tools in Lightroom you can simply export your keywords and Lightroom creates a tab delimited text file that you can edit with any text editor. I used an application called OmniOutliner that was able to also handle the hierarchy as implied by the number of tabs which made the process easier. I don't believe that you could drag and drop in the Keyword List then as you can do now.

Even with these limitations I feel that the benefits of using hierarchical keywords out ways the restrictions. Although no matter which keyword list you start with be prepared to take the time to tailor it to your own personal needs.

-louie

Louie and Cletus,

Thanks for these observations. I've been having a debate with myself for months now about whether or not to get the CVKC. I have no aspirations to be a stock photographer, but as I posted in another thread, I have a huge backlog of images to ingest, and that includes keywording. If the CVKC cnan speed up that effort, then it will be worth the cost and the learning curve.

I really appreciate it that you have given me arguments and reasons for both sides of the issue. The real cost is maintenance and modification of the catalog. I expect that I will need to make large additions for the sub-keywords for the keywords "train," and "railroad" and "railway," and probably others. Whatever someone's hobby or special interest, I'm sure that they will be faced with the same dilemma.

Now I have to do some serious contemplation.

Phil

 

[LouieSherwin]

Hi Phil,

I came across another keyword provider: Photo-Keywords. It looks to be quite a bit more comprehensive than CVCK and less expensive. The author has also provided a lot of additional information that I find useful.

-louie

 

[PhilBurton]

Hi Phil,

I came across another keyword provider: Photo-Keywords. It looks to be quite a bit more comprehensive than CVCK and less expensive. The author has also provided a lot of additional information that I find useful.

-louie

Louie,

Thanks. The creator of CVKC maintains a listserv that I am on. It is [email protected]. It's a good source of support for this kit, and that has value for me.

Phil

 

[MarkNicholas]

I find that it is not so much the number of keyword or the structure that affects performance but the assignment of keywords to photos that has a significant detrimental effect. After 10-15 mins of intensive keyword assignment LR often stops responding and goes all opaque. I just shut it down and restart LR and all is well for another 10-15 mins and repeat etc.

 

[Tony Jay]

I am with Cletus and others here!

Buying or downloading a controlled vocabulary sounds like a great idea but in practice it seldom works out well....
The problem is that that neither the granularity of the keywords nor the choice of keywords is what is needed or expected.
To a bird photographer having a single keyword for penguin is unacceptable, but for the average photographer who is not specifically into bird photography a full taxonomic hierarchy including latin names is way overkill.

Most people who sell controlled vocabularies also talk up how extendable and modifiable their systems are. However, unless one has built something like this from scratch to start with and is fully familiar not only with the Lightroom process but also how the hierarchies are constructed modifying or extending the vocabulary becomes difficult and confusing. Thus most individuals either never modify them or they end up with a dog's breakfast...

Really, the only viable alternative is to create your own vocabulary that reflects the subjects that you shoot as well as the granularity of detail that suits the purpose. I would have no issue with downloading a pre-existing controlled vocabulary in order study its structure. From there is becomes relatively easy to construct one's own - in concept at least!

Just that you know, I have constructed my own controlled vocabulary in Lightroom and the last time I looked it had well over 100 000 keywords and it has since grown!

 

[tspear]

Louie,

Thanks. The creator of CVKC maintains a listserv that I am on. It is [email protected]. It's a good source of support for this kit, and that has value for me.

Phil

I would suspect that the vast majority of words in any predefined lexicon likely would require a lot of work to apply. In addition, since you likely do not know the lexicon, you will likely end up being rather haphazard in the application until you get it to the point where is is fully memorized.
As such, I would short cut the process and use one of the tools that can do the processing of standard items for you. Such as
Any Vision Lightroom Plugin
Lightroom Keywords | Quick, automatic keywords for your photos.
Wordroom - The lightroom plugin for automated keywording
Excire Search - Find your Lightroom Images fast and intuitive

Note: I have used none of the above. I have just tracked the concept since I plan to eventually make use of it.

If you decide to go this route, please report back. It would hopefully save me some time when I get to it.

 

[MarkNicholas]

the last time I looked it had well over 100 000 keywords and it has since grown!

How can I check how many keywords I have without physically counting them all ?

 

[Hal P Anderson]

You can export them from the Metadata menu and count the lines in the resulting text file.

 

[LouieSherwin]

I find that it is not so much the number of keyword or the structure that affects performance but the assignment of keywords to photos that has a significant detrimental effect. After 10-15 mins of intensive keyword assignment LR often stops responding and goes all opaque. I just shut it down and restart LR and all is well for another 10-15 mins and repeat etc.

You don't by any chance have "Automatically write changes to XMP" enabled?

-louie

 

[MarkNicholas]

You don't by any chance have "Automatically write changes to XMP" enabled?
-louie

Nope its disabled. I have had this issue for as long as can remember with many previous LR versions. I just put it down to my key wording being so fast that LR cant keep up !

 

[PhilBurton]

I would suspect that the vast majority of words in any predefined lexicon likely would require a lot of work to apply. In addition, since you likely do not know the lexicon, you will likely end up being rather haphazard in the application until you get it to the point where is is fully memorized.
As such, I would short cut the process and use one of the tools that can do the processing of standard items for you. Such as
Any Vision Lightroom Plugin
Lightroom Keywords | Quick, automatic keywords for your photos.
Wordroom - The lightroom plugin for automated keywording
Excire Search - Find your Lightroom Images fast and intuitive

Note: I have used none of the above. I have just tracked the concept since I plan to eventually make use of it.

If you decide to go this route, please report back. It would hopefully save me some time when I get to it.

Tim,

Thanks for this list. I was already aware of Excire, and I had a few messages exchanged with sojmeone from Excire. The takeaway is that Excire uses its own taxonomy and it cannot be integrated with any keywords or keyword structure I already have. Until Excire addresses this issue (if ever) I can't use it.

The others on your list, I need to investigate.

Phil

 

[PhilBurton]

I am with Cletus and others here!

Buying or downloading a controlled vocabulary sounds like a great idea but in practice it seldom works out well....
The problem is that that neither the granularity of the keywords nor the choice of keywords is what is needed or expected.
To a bird photographer having a single keyword for penguin is unacceptable, but for the average photographer who is not specifically into bird photography a full taxonomic hierarchy including latin names is way overkill.

Most people who sell controlled vocabularies also talk up how extendable and modifiable their systems are. However, unless one has built something like this from scratch to start with and is fully familiar not only with the Lightroom process but also how the hierarchies are constructed modifying or extending the vocabulary becomes difficult and confusing. Thus most individuals either never modify them or they end up with a dog's breakfast...

Really, the only viable alternative is to create your own vocabulary that reflects the subjects that you shoot as well as the granularity of detail that suits the purpose. I would have no issue with downloading a pre-existing controlled vocabulary in order study its structure. From there is becomes relatively easy to construct one's own - in concept at least!

Just that you know, I have constructed my own controlled vocabulary in Lightroom and the last time I looked it had well over 100 000 keywords and it has since grown!

Tony,

OK you have just convinced me basically using own arguments (prune the CV tree, add lots of additional keywords) that I should simply strike out on my own. I have enough of an IT background to understand the concept already. To be clear, I am not and have no future aspirations to be a stock photographer. :speechless:

Phil

 

[tspear]

Tim,

Thanks for this list. I was already aware of Excire, and I had a few messages exchanged with sojmeone from Excire. The takeaway is that Excire uses its own taxonomy and it cannot be integrated with any keywords or keyword structure I already have. Until Excire addresses this issue (if ever) I can't use it.

The others on your list, I need to investigate.

Phil

Why do you care if it integrates? As long as it is under its own root element, so it does not "pollute" your main tree it should not matter. The only thing I would care about is if any solution had false positives, provided a complete list of possible items they recognize, and change lists on each release of what is recognized.

 

[PhilBurton]

Why do you care if it integrates? As long as it is under its own root element, so it does not "pollute" your main tree it should not matter. The only thing I would care about is if any solution had false positives, provided a complete list of possible items they recognize, and change lists on each release of what is recognized.

Tim,

IF I am going to build my own keyword hierarchy, presumably I'm going to start out with some notion of top level keywords, and then sublevels of each top level. I will proceed as Tony suggests in post #8 in this thread. In practice, if I use Excire or some other auto-keywording service, I will end up with a lot of keywords that aren't yet in my own hierarchy, but I will need to add them in their proper context, including synonyms and UK spellings, etc. AND my own hierarchy could become polluted or end up with lots of duplicates. Then I lose control of the entire process.

If there is a feasible workaround, then I am extremely interested.

 

[tspear]

@PhilBurton

I think we are talking past each other. My point is more fundamental, why do you use keywords? To find stuff, correct?
If keywords are to find things, you care about only one thing: the accuracy (both missing and false positives). Human nature, and even AI capability at this point will miss items, that is just a given. However, I think human nature is not likely to have many false positives unless it was due to data entry. AIs, the level of false positives is TBD.

So with that said, if you are looking for an image, and there are two sunset keywords on it, does it matter? It will be found when you search for sunset....

Therefore. using Excire as the example, if they can attach all their generated keywords under #Excire, and they add Tree, Smile, Car, Sunset to an image. If you have already tagged the image with sunset under #Location, who cares? Sure you can remove one of them, but why does it matter?

 

[MarkNicholas]

@PhilBurton

So with that said, if you are looking for an image, and there are two sunset keywords on it, does it matter? It will be found when you search for sunset....

Therefore. using Excire as the example, if they can attach all their generated keywords under #Excire, and they add Tree, Smile, Car, Sunset to an image. If you have already tagged the image with sunset under #Location, who cares? Sure you can remove one of them, but why does it matter?

I would not be able to sleep at night if my keywords were so cluttered