External Editors AI for catalog management?

[Stephanie Booth]

Lightroom Version Number

14.4.1

Operating System

1. macOS 15 Sequoia
2. iOS

Hello everyone, im wondering how close we are with plug-ins and extensions to what I would like to help me manage my approx 150k photos spanning 25 years (there will be more once I get around to scanning negatives… saw there was a recent thread about this I will peruse).

- correctly identify photos of a specific cat ‍
- update file creation date to match timestamp embedded in video
- « smartly » identify videos taken by an unknown camera based on image characteristics (I have lots of cat surveillance security cam footage)
- identify « ski outings » and « hiking outings »
- conversational interface for searching for photos (« many years ago I took a photograph of a pipe in my cellar » or « I had a beautiful photo from the top of this mountain in HDR, maybe 15 years ago »
- assistance in correctly completing location or keywords

See what I mean? The technology is pretty much there now but do we have anything that is already working in Lightroom out of the box or at minimum « plug and play »?

Of course, also interested in workarounds that would do this through the file system and update metadata in a way that Lightroom can then take advantage of.

Thanks for any thoughts, tips, pointers, experiences or ideas!

(I have the latest desktop version but not on my computer right now so just put something random in the required field, I have no clue where we are at now)

 

[Paul_DS256]

Yes, some interesting ideas including maybe adding voice interface. Maybe even a 'process photo' eliminating the need for the maker know anything about post processing.

There would also likely two steps needed. A background process that indexes each photo imported and a real-time AI based query language that integrates indexes. I can't see any local computer being able to scan 150K in real-time unless everything was cloud based and built for such intensity.

My impression at the moment is that visual recognition is not quite there yet. This is based on manually using Googles 'Search by Image' feature. I haven't tried any of the automatic keyword assignments for a couple of years but found them to generic to be of interest.

Personally, I'm waiting for someone to come up with a tool that will say if an image is AI generated.

 

[stevevp]

I've seen Excire mentioned elsewhere. I know nothing about it but it might be what you're looking for.

 

[clee01l]

Sensei available in Lightroom, not in Lightroom Classic can do much of the keyword searching on images that do not have those keywords. (Show me all of the images that contains cats)

 

[Califdan]

Personally, I'm waiting for someone to come up with a tool that will say if an image is AI generated.

​

Content Authenticity (CAI initiative)​

See https://contentauthenticity.org/
Verify content of exported JPG’s here: https://contentcredentials.org/verify

CAI is currently avaialbe in LrC as an option when you export images. At some point it may be be automatic and become an industry standard.

Attached is excerp from one of my LrC tutorials

 

[John R Ellis]

There are a couple plugins that use state-of-the-art AI to generate accurate keywords, short and long descriptions, extract text, and generate specialized descriptions and classifications from ChatGPT-like "prompts" you supply. My Any Vision plugin uses Google's Gemini, while Excire runs entirely on your computer. Last I looked, Excire doesn't yet have the ability to supply prompts.

Working through your list:

- correctly identify photos of a specific cat

You could provide a prompt like, "Describe the cats in the photo, including their colors, length of fur, and breed". Then you could use LR text search to find photos that likely contain the desired cat. Or you could have hierarchical keywords representing the color, fur length, and breed.

- update file creation date to match timestamp embedded in video

Any Vision does a good job of extracting text (OCR, optical character recognition). You could write a prompt that transforms that text with patterns to only extract the format of timestamps in the video. Unfortunately, LR still doesn't allow plugins to directly change the capture date of photos, but if you're adept at scripting, you could write a script that uses Exiftool to assign the video capture date from the extracted text.

- « smartly » identify videos taken by an unknown camera based on image characteristics (I have lots of cat surveillance security cam footage)

Any Vision / Google Gemini can apply all the image analyses described above to video as well as images. It analyzes frames sampled every second from the video.

- identify « ski outings » and « hiking outings »

The default keywords and descriptions generated by Any Vision and Excire should do a good job of that.

- conversational interface for searching for photos (« many years ago I took a photograph of a pipe in my cellar » or « I had a beautiful photo from the top of this mountain in HDR, maybe 15 years ago »

That conversational interface doesn't exist yet in any image app I know of. Though the descriptions generated for the photos are whole sentences and paragraphs and are very good, and they can be searched with traditional keyword and text search.

- assistance in correctly completing location or keywords

Any Vision and Excire do a very good job with keywords. Any Vision / Google Gemini recognizes most iconic locations (e.g. Eiffel Tower, New York City skyline, Crater Lake, etc.) and very many more obscure ones (basically, locations where someone has published photos).

 

[UliP.]

Exicre Search DOES Support prompt based search. Take a look:

https://excire.com/en/tutorials/excire-search/promptsuche/

And it works very well!

 

[John R Ellis]

Thanks for the correction.

 

[BJB]

Like the OP I have many years of over 100k images and share their desire for better search. My phone has jpeg versions of them all and I search there based on anything I want to find filenames to then find in LR to edit sometimes! Seems like if I can do a very good AI search using Android Gallery on my tablet or phone, should be able to do something similar on LR Classic.

I am also interested in opinions about the Excire 2024 LR plugin from users as I was just researching it. I have learned you can turn off the Excire option to update the LR catalog with its keywords which I like.

If it would be more appropriate to start a new thread I would be glad to.

Thanks,
BJB

 

[John R Ellis]

My phone has jpeg versions of them all and I search there based on anything I want to find ... Seems like if I can do a very good AI search using Android Gallery on my tablet or phone, should be able to do something similar on LR Classic.

With the Any Vision plugin, you can do the text search "Any Searchable Field contains" to do the same kind of searches as on Android or LR Cloud's Sensei.

 

[Stephanie Booth]

Sensei available in Lightroom, not in Lightroom Classic can do much of the keyword searching on images that do not have those keywords. (Show me all of the images that contains cats)

the problem with this is that (unless I'm mistaken) I can't then capture keywords in the Cloud version that will be visible in Classic too.
(also... I have lots and lots of cat photos... different cats... they all get mixed up! basic thing I'd like to do is set time boundaries "hey, I got this cat in 2019, so before that, it's not him!")

 

[clee01l]

the problem with this is that (unless I'm mistaken) I can't then capture keywords in the Cloud version that will be visible in Classic too.
(also... I have lots and lots of cat photos... different cats... they all get mixed up! basic thing I'd like to do is set time boundaries "hey, I got this cat in 2019, so before that, it's not him!")

Yes, that is the problem with Sensei. But I keep hoping the it will show up in LrC since more and more AI stuff is being added.

The other problem is that while you can recognize people using the people search, you can't recognize individual cats So if "Tom" is a guy you are in luck. But if "Tom" is a cat you are not.

 

[John R Ellis]

Another app that's been surfacing recently is Peakto:
https://cyme.io/peakto-photo-organizer-software/

It's from the same company that makes the Avalanche app for converting databases between various programs (LR, Mac Photos, Aperture, etc.). Avalanche has a very good reptuation, but I haven't heard much about Peakto.

 

[BJB]

With the Any Vision plugin, you can do the text search "Any Searchable Field contains" to do the same kind of searches as on Android or LR Cloud's Sensei.

Thanks, will have to look at this too.
Bill

 

[John R Ellis]

- update file creation date to match timestamp embedded in video ... basic thing I'd like to do is set time boundaries "hey, I got this cat in 2019, so before that, it's not him

With Any Vision's Prompt command, you can extract the timestamps embedded in images and video. I downloaded 10 security images containing cats and used this prompt:

Then you can use the Library Filter bar's Metadata browser to select photos in a range of dates:

 

[John R Ellis]

So I'm a cat guy and couldn't resist. Continuing the cat-meets-AI, a simple Any Vision prompt can generate very good searchable descriptions of a cat in the photo. This prompt:

generates this caption:

which you can search with "Text: Any Searchable Field" in the Library Filter bar.

With a little more effort, you can create a prompt that classifies according to breed, colors, paw color, eye color, hair length, and kitten/adult, and you can use the Library Filter's Metadata browser to quickly slice-and-dice all your cat photos by those attributes, e.g. to find all black-and-white shorthairs with white paws and yellow eyes:

The prompt text is straightforward:

Describe the most prominent cat in the photo, giving its breed, its colors, the color of its paws, color of its eyes, whether it's long- or short-haired, and whether it is a kitten. An example of the result is:

Breed: Domestic shorthair
Colors: Gray and black tabby pattern
Color of paws: White
Color of eyes: Green
Hair length: Short
Kitten: No

The full prompt is:

 

[Stephanie Booth]

So I'm a cat guy and couldn't resist. Continuing the cat-meets-AI, a simple Any Vision prompt can generate very good searchable descriptions of a cat in the photo. This prompt:

OK, sold! Definitely going to try it. I would LOVE to have proper tagging for my cats (and all those I bump into all over the place and photograph). I'm guessing I can also put conditions somewhere, for example, to say that "Tounsi" can only have been photographed between this and that date, and in this place and that place?
Also liked your timestamp extraction. Could I then automate setting the video capture time to the date-time extracted by AnyVision?

 

[John R Ellis]

I'm guessing I can also put conditions somewhere, for example, to say that "Tounsi" can only have been photographed between this and that date, and in this place and that place?

You can use date and location filtering to limit searching photos to that criteria, and then you could in addition drill down by the cat classification (colors, etc.). But at some point, to avoid doing that, you may want to apply the keyword "Tounsi" to all the resulting photos, so you don't have to keep redoing the searches.

Could I then automate setting the video capture time to the date-time extracted by AnyVision?

That would only be possible if you used the Exiftool utility to set the video capture time outside of LR (required technical scripting skills) and then re-imported all the videos. That's due to LR limitations -- plugins can't set capture time in general, and for videos, LR only reads the capture time from video metadata when the video is first imported (Metadata > Read Metadata From File doesn't work for videos).

 

[Stephanie Booth]

@John R Ellis I'm trying out AnyVision and it seems really promising, thanks! I'm running into one tiny issue, where the extended description result sometimes starts with this:

Here's some text for the IPTC Extended Description (Accessibility) field, providing a more detailed description of the image:

"The image shows an Omron brand automatic blood pressure monitor displaying a reading. [...]"

Known issue? Workaround? (what is the best way to ask you about this kind of thing or report weirdnesses?)

 

[John R Ellis]

the extended description result sometimes starts with this:

Some Google models include such introductory text some of the time. It's hard to predict when or why. Often, it helps to add a last sentence to the prompt that says something like: 'Do not provide an introductory sentence like "Here's some text for the IPTC Extended Description"'.

 

[John R Ellis]

what is the best way to ask you about this kind of thing or report weirdnesses?

You can contact me directly at the Any Vision support address:
https://johnrellis.com/lightroom/anyvision.htm#support