• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom Classic questions. All the information you need is in Adobe Lightroom Classic - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. And better still, the eBooks are updated for every release, so it's always up to date.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Catalogs Exiftool to remove obsolete metadata

Status
Not open for further replies.

Michael Naylor

Active Member
Joined
Nov 21, 2016
Messages
145
Location
Spain
Lightroom Experience
Intermediate
Lightroom Version
Classic
My images (TIFs, DNGs, JPGs and Photoshop) and been bounced from one DAM to another many times over the years. They have accumulated various bits of IPTC and XMP that is no longer required by LR. I would like to use exiftool to delete all of this crud, so I will need to know what can go and what needs to stay.

Please can a metadata expert suggest what exiftool command options I need for this?
 
My images (TIFs, DNGs, JPGs and Photoshop) and been bounced from one DAM to another many times over the years. They have accumulated various bits of IPTC and XMP that is no longer required by LR. I would like to use exiftool to delete all of this crud, so I will need to know what can go and what needs to stay.

Please can a metadata expert suggest what exiftool command options I need for this?
Mike,

The EXIFTool forum may be a better place to ask this question. Do you have a list of all metadata that Lightroom uses and therefore you want to preserve?
 
Hi Phil. No I don't have a list of what LR uses, so I'm hoping someone here might know. I could get close by importing an image with absolutely no metadata and then adding keywords and stuff, but I'm not confident I could cover all the options. An Adobe supplied specification would be better. Thanks.
 
Hi Reolof. At some point I will wish to migrate to another DAM. I would like to make the process as easy as possible. I've been playing around with various alternatives for over a year now. At the moment, I have parallel systems in Lightroom Classic and Media Pro SE and its frightening to see just how much is wrong in list mode. Missing Color Profiles, Color Spaces, non-current Keywords in various XMP fields that Lightroom doesn't use, Event Dates that don't agree with the Capture Date, etc, etc. I'd like to fix these omissions and conflicts and reduce the XMP mess created in poorly implemented software - including versions of Lightroom. The fact is, software houses have not followed the rules AND the rules themselves are a complete mess. Phil Harvey has much to say on this. This is why I bother, because my files are broken and I'd like to fix them.
 
Hi Reolof. At some point I will wish to migrate to another DAM. I would like to make the process as easy as possible. I've been playing around with various alternatives for over a year now. At the moment, I have parallel systems in Lightroom Classic and Media Pro SE and its frightening to see just how much is wrong in list mode. Missing Color Profiles, Color Spaces, non-current Keywords in various XMP fields that Lightroom doesn't use, Event Dates that don't agree with the Capture Date, etc, etc. I'd like to fix these omissions and conflicts and reduce the XMP mess created in poorly implemented software - including versions of Lightroom. The fact is, software houses have not followed the rules AND the rules themselves are a complete mess. Phil Harvey has much to say on this. This is why I bother, because my files are broken and I'd like to fix them.
Mike,

I've been trying to sort out "dates" and it seems like there are at least three different dates in EXIF, and as far as I can tell, various Windows metadata utilities are very inconsistent (or wrong) in how they treat and display these various dates. I'm having to fall back on EXIFTool itself, which is OK, except that command-line utilities are so 20th century. Yes, I grew up on DOS and spent enough time in UNIX-land, but I really prefer a GUI.

If you have access to a Windows PC, there are two utilities I'd like to recommend.
XMP data consists of many different "namespaces," such as EXIF, TIFF, PHOTOSHOP, and LR, plus others.

Phil Burton
 
I will need to know what can go and what needs to stay.

It's very hard to generalise, but I agree with Roelof's post - the files may now contain crud, but that doesn't mean things are broken. Apps also write to a mix of their own and standard fields, so while you might be able to delete all of MediaPro's fields in a single command, it may have written to other fields too. So I would certainly suggest that you only hack away at metadata when there is a specific problem.

One tool I would suggest is Bridge 2017 and its File Info command. Its Advanced tab shows the XMP metadata in the file, and allows you to delete values. The UI is quite helpful and I have used it for years to look into metadata issues. Sadly, Adobe failed to include this feature in Bridge 2018, but you can still get it from the Adobe CC application.

One thought is to restore your original backups.

But generally, I would ignore the instinct to tidy things up.

John
 
I've been trying to sort out "dates" and it seems like there are at least three different dates in EXIF, and as far as I can tell, various Windows metadata utilities are very inconsistent (or wrong) in how they treat and display these various dates. I'm having to fall back on EXIFTool itself, which is OK, except that command-line utilities are so 20th century.

There are many more "date/time" fields, have a look with
Code:
exiftool -list -Time:All

I think even nowadays command-line applications are still useful, esp. when it goes to automatic or semi-automatic task.

There are several GUI's available for exiftool, did you try them:

Best Regards
 
There are many more "date/time" fields, have a look with
Code:
exiftool -list -Time:All

I think even nowadays command-line applications are still useful, esp. when it goes to automatic or semi-automatic task.

There are several GUI's available for exiftool, did you try them:

Best Regards
Wernfred,

Thank you.

I have tried EXIFTool GUI but I found problems with the way it changes dates and times. I needed to change both Windows file dates and EXIF dates and I simply could not do what I needed with EXIFTool GUI. So I need to experiment with different command-line options with just EXIFTool. I will create cmd files for the command-line parameters that I need to use. I'm working on these experiments when I have the time.

I looked at GUI2 but I found it very complex.

Phil
 
So far, I've managed to get part way there by running the following command:

exiftool -r -overwrite_original_in_place -XMP-crs:all= -XMP-idimager:all= -XMP-ics:all= -XMP-xmpMM:all= -XMP-microsoft:all= Pictures/Lightroom/LRIMAGES/

That was across hundreds of folders and sub-folders with over 24,000 files, DNG, JPEG, TIFF and PSD. It was then necessary to have LR re-read the metadata to remove it from the LR catalog.

Next, I duplicated all the images to be safe, and imported them into a new Media Pro catalog. I deleted all the flat keywords shown under the Hierarchical Keywords tab, and all the entries under the People and Events tabs (NOT the Keywords tab!) - I use hierarchical keywords for those and also to identify all my Lightroom Collections.

I then ran the built-in MP script, IPTC Date from Capture Date. Within the Date Finder, I was then able to delete all the unused date entries.

Finally, Export Annotations to update the files. So far so good, but there's still a lot more to do.
 
Last edited:
So far, I've managed to get part way there by running the following command:

exiftool -r -overwrite_original_in_place -XMP-crs:all= -XMP-idimager:all= -XMP-ics:all= -XMP-xmpMM:all= -XMP-microsoft:all= Pictures/Lightroom/LRIMAGES/

That was across hundreds of folders and sub-folders with over 24,000 files, DNG, JPEG, TIFF and PSD. It was then necessary to have LR re-read the metadata to remove it from the LR catalog.
Mike,

Were you able to confirm that the LR catalog was updated based on changes to the metadata in DNG, JPG, and TIFF files? When I made changes to the metadata in XMP files outside of LR, and tried to update the LR catalog, it appeared that LR did not detect those XMP changes.

Next, I duplicated all the images to be safe, and imported them into a new Media Pro catalog. I deleted all the flat keywords shown under the Hierarchical Keywords tab, and all the entries under the People and Events tabs (NOT the Keywords tab!) - I use hierarchical keywords for those and also to identify all my Lightroom Collections.

I then ran the built-in MP script, IPTC Date from Capture Date. Within the Date Finder, I was then able to delete all the unused date entries.

Finally, Export Annotations to update the files. So far so good, but there's still a lot more to do.
You are way ahead of where I thought you would be. :thumbsup: Did you use exiftool to inspect metadata?

I'm not surprised that you gave up on Photo Supreme. So did I but after doing just test imports and trying to synchronize the catalog with LR's catalog. There were just so many issues with Photo Supreme. :mad: Complete waste of time and money. Did you evaluate iMatch?

If you do settle on Media Pro, do you plan to keep on using Lightroom for RAW processing? Would you try to synchronize the catalogs in both applications?

Have you seen this review of Media Pro? Phase One Media Pro SE Review | The Good, The Bad, The Ugly Check out the "Bad" and the "Ugly."

Phil Burton
 
Hi Phil, and thanks for your input.

Yes I was able to confirm LR and PM were able to replace the metadata. However, there is an anomaly - isn't there always? Despite using only hierarchical keywords, LR insists on exporting flat IPTC keywords as well, whereas PM doesn't. PM uses the the designated XML fields only - which means they will re-import into both LR and PM without duplication.

Like you, I found PSu frustrating. His latest updated version only brings a fresher looking GUI, but nothing new underneath. I'm on macOS, so iMatch is not possible for me.

I may or may not settle on MP with or without C1, but right now MP is helping me remove the crud. I intend to cancel the Adobe subscription though. LR will continue working, but without the development module.

I've read that review and many others. I conclude that none of them compete with LR. I also conclude LR needs many bug fixes further development.

Here's where I am right now. Not knowing exacting what XML/IPTC LR or PM use, I've taken the BRUTE FORCE approach - using MP and my part processed copy of 24,000 images.

1) make sure the MP catalog has only the metadata I need.
2) saving that metadata back to the images.
3) deleting all XMP and IPTC from all the image files by:
exiftool -r -overwrite_original_in_place -XMP:all= -IPTC:all= MPIMAGES/
4) saving the MP catalog metadata back to the image files.

Needless to say, this takes many hours, but stage 4) is running as I type.
 
4) saving the MP catalog metadata back to the image files.

Needless to say, this takes many hours, but stage 4) is running as I type.

Do check afterwards if all metadata is saved into the image files. When i migrated from MP to Lightroom it was a very tedious process. Some of my batches did not have the metadata transfered.
 
Roelof. I'm beginning to think your earlier comment "why bother" was good advice, but I'll battle along a little longer and see where it gets me.
Mike,

Given that your metadata includes many different namespaces, it is possible that Media Pro will ignore certain names, e.g. IdImager. That said, I have read reports that CaptureOne will actually corrupt some fields when it writes metadata out to XMP files. There are a number of such reports on the IdImager website. Don't know if that also applies to Media Pro. Whatever you do, run tests on copies of your data before committing irreversible actions on your live data.

Phil
 
With the clean XMP/IPTC free images inside the existing MP catalog (but without running Export Annotations), I ran the Capture One trial and imported the MP catalog. The result was hundreds of flat keywords appearing within the Hierarchical tab alongside my hierarchical keywords. I consider this to be unacceptable. But I've written off CO for a multitude of reasons. Mostly because it doesn't recognise dates prior to 1970-01-01, and its not possible to add existing keywords directly for the HK list. As a DAM, its unusable. MP is far superior, but it can't export images or print at full size. I've also found the Phase One technical support team to technically uniformed and dismissive.

I'm forced to conclude LR is the only viable option at present (the least worst), but I'm disgusted with Adobe. They're happy to leave things as they are and rely on third party hackers to fill in the gaps, but not provide a full and proper API. There appears to be be no hope of a better DAM hitting the market any time soon and all existing products are too old to be maintained. What happened to the database expertise of the 70s? With SCO Unix and 3GL, even I could do better than todays crowd.

However, this lengthy exercise hasn't been entirely wasted. I've learned a little more and discovered a few quirks. I even discovered a missing image, hidden within the LR folders, that LR had consistantly ignored. Oh, and I managed to flush out most of the old crud, leaving only unnecessary duplicate fields which Adobe products add - thus violating 1NF by ignoring the first rule of database design!

Many thanks to those that have participated in this thread.
 
I'm forced to conclude LR is the only viable option at present (the least worst), but I'm disgusted with Adobe. They're happy to leave things as they are and rely on third party hackers to fill in the gaps, but not provide a full and proper API. There appears to be be no hope of a better DAM hitting the market any time soon and all existing products are too old to be maintained. What happened to the database expertise of the 70s? With SCO Unix and 3GL, even I could do better than todays crowd.
Mike,

So you will not be cancelling your Adobe subscription?

I don't think that database expertise is lost. it's just "following the money" to commercial applications for corporations. Oracle, for example, is very popular, but no one in their right mind would ever consider the Oracle database for consumer use. I'm not sure that I agree that the existing products are all too old to be maintained. Media Pro is a commercial application that is being updated.

For the most part, I don't regard the Adobe approach of supporting third-party apps to supplement its own development. Those third-party developers who do plug-ins and presets are often individuals or small companies that are more nimble and sometimes "closer to the ground" than a large, bureaucratic company like Adobe, and clearly more willing to take risks. In turn, Adobe is allowing these developers to share in the total market revenue for Lightroom. Photoshop has an amazing number of plug-ins and presets.

Converting a product into a "platform" is a well-respected strategy in many software fields and users like that approach. I personally worked on a firewall security product that was a platform, and then for a smaller vendor that created a plug-in for that same firewall. It's good business overall.

I have read comments by the late Rob Cole and others that Adobe needs to improve its API, so I agree with you on that point.

However, this lengthy exercise hasn't been entirely wasted. I've learned a little more and discovered a few quirks. I even discovered a missing image, hidden within the LR folders, that LR had consistantly ignored. Oh, and I managed to flush out most of the old crud, leaving only unnecessary duplicate fields which Adobe products add - thus violating 1NF by ignoring the first rule of database design!

Many thanks to those that have participated in this thread.

I think we have all benefited by your explorations here.
 
So you will not be cancelling your Adobe subscription?
I guess not.
Media Pro is a commercial application that is being updated.
Still only 32bit. The consensus is this will not change. The next macOS release will 64bit only.
Adobe is allowing these developers to share in the total market revenue
Something I would call plain lazy and a gateway for the likes of Luminar, DxO, etc.

I was an early Aperture user, but when Lightroom was rushed to market I got caught in the hype. If I and many others hadn't jumped ship, maybe Apple would not have done what they did and Adobe would have had a serious competitor.
 
Roelof. I'm beginning to think your earlier comment "why bother" was good advice, but I'll battle along a little longer and see where it gets me.

Still, I am curious about what your journey will lead to. I do find it an very interesting topic!
 
I guess not.

Something I would call plain lazy and a gateway for the likes of Luminar, DxO, etc.

Mike,

I have to respectfully but firmly disagree with you, as a software product management professional. The reason that DOS became so popular so fast was that it was an open platform, sol that the hardware manufacturer didn't have to write all application software themselves. Earlier generations of computer systems had "closed" operating systems, so that the manufacturers had to write all or most of the applications themselves. I ran the "third party vendor" program for a PC competitor wannabe called Fortune Systems in the 1980s. Our failure to have a good enough reason to attract software developers was one element in the company's demise.

A plug-in/preset approach based on an API and published standards is a "leverage" model. Don't be surprised if Luminar or its ilk adopt a similar approach at some point, with a published API, conversion and development tools, and hopefully, a better development environment than lua. Luminar is not just competing with Adobe. They are competing with the entire ecosystem of plug-in and preset authors. Good luck with that!

Phil Burton
 
It's 32 months and I'm back on the same subject. Why? Because Camera Bits have released Photo Mechanic Plus - the Plus refers to their new catalogue.

I'll need to wait until they say it's fully tested to run on Big Sur, so I'm monitoring their forum and pondering what will need to done if I decide to migrate.
 
Status
Not open for further replies.
Back
Top