How to use XMP sidecar files for JPEGs in Lightroom

[Eleni]

Hello, I am a new Lightroom user who has recently migrated from Aperture.

I am working on a referenced catalogue of nearly 150K images. The file types I am working with are RAW, RAW+JPEG, JPEG, TIFF, PNG, PSD and Videos. I am using a mac pro (late 2013), OS X El Capitan and Lightroom/Photoshop CC.



My workflow includes a lot of metadata editing and a few adjustments. For reasons of back-up efficiency and data safety I will like to find a way to store and sync all the changes I make on any file within Lightroom, on an XMP sidecar file.

Essentially I will like to find a way to get Lightroom to edit and store metadata on all files in the way that it treats the RAW files.

Now, i know that there is no way to do this in Lightroom at the moment, which is why I am bringing this issue to the community so that we can discuss workarounds.



Essentially I need to work out how to do three things:



1. Export metadata from JPEGs, TIFFs,PNGs and PSDs (videos would be pushing it I imagine) in XMP sidecar files

2. Get Lightroom to read the metadata from the XMP file instead of the JPEG

3. Automatically sync any modification of the metadata on these files.





What I got so far,

1. Found an export plugin which exports JPEGs with an xmp sidecar file. No option to only export the xmp or automatically export in next to the files in question, but still good enough. The plugin in question is:

XMP Extractor Post-Process Action – Capture Monkey

Please let me know if you know of any others



Sidenote / question:

4. I have discovered with disappointment that when I changed my catalogue settings to automatically write changes on XMP, all my metadata were written on the files that are not RAW. So, xmp were created for all RAW images, but any file that is not RAW, now has the metadata written on it. I also found this to be irreversible. I understand that it is a similar function as an XMP, but written internally in the file but this is something I didn't want and didn't get the chance to chose. This feels a bit strange for Lightroom, so please, do correct me if I have done or perceived something wrong here.



I know I have a few questions there, which is why I numbered all separate issues! Thank you all in advance

 

[Johan Elzenga]

Sorry, Lightroom ignores XMP sidecar files for all file types except proprietary raw.

 

[johnbeardy]

Remember that sidecars are only really a workaround for Adobe's reluctance to write back to undocumented proprietary files.

Second, xmp metadata doesn't contain all your work. In fact, Lightroom writes more of your work to xmp than Aperture did, but it's not all.

You need to backup your catalogue, which contains 100% of your work, and review whether you need to repeatedly backup JPEGs to which xmp has been written.

 

[Eleni]

Hi guys, thanks for your replies

@JohanElzenga I also discovered this recently, I was wondering if it is possible to bypass this even if I need to get the help of a developer and create an import plugin or something

@johnbeardy I am still learning about sidecars and xmps. I don't need to rely on them for back-up purposes. I have a few catalogue back-ups for this peace of mind. I just don't want to write the metadata on the files. I constantly change my files. In the occasion of updating JPGEs files, my back-up would have to copy the whole image just to update a keyword change. Also issues like this one make things additionally worrying Lightroom 6: Invalid XMP metadata written to exported JPEGs, tripping up reading software | Photoshop Family Customer Community
Perhaps switching to full DNG is the solution to all this, but don't know enough to trust this format. It is definitely an issue I am still contemplating. Any input and feedback would be very welcomed.


Any suggestions on my initial issues would be greatly appreciated. I've seen posts of people from 2007 about the same exact issue and struggling to find the solutions. But there must be something out there!

 

[sizzlingbadger]

I think if you make the jpgs read only then LR will create an xmp file because it can't write to the jpg. It did this in earlier versions but I haven't tested it in years.

 

[johnbeardy]

I think if you make the jpgs read only then LR will create an xmp file because it can't write to the jpg. It did this in earlier versions but I haven't tested it in years.

I could be wrong, but I don't think that has ever been true. It did certainly happen with Bridge and DNGs, maybe JPEGs too.

 

[johnbeardy]

The simple answer is to back up the catalogue, at least daily, and backup the original photos when they are created - not each time some metadata is written to them. With these two safeguards, you have 100% of your work and 100% of your photos.

It doesn't do any harm if you also save metadata to the files, and it helps if you use those originals with other apps, but you might want to fine tune your backup procedures so it doesn't repeatedly backup the files.

 

[PhilBurton]

Hello, I am a new Lightroom user who has recently migrated from Aperture.

My workflow includes a lot of metadata editing and a few adjustments. For reasons of back-up efficiency and data safety I will like to find a way to store and sync all the changes I make on any file within Lightroom, on an XMP sidecar file.

Essentially I will like to find a way to get Lightroom to edit and store metadata on all files in the way that it treats the RAW files.

Eleni,

I think the issue is that tools other than Lightroom support XMP only for RAW files. For non-RAW files, the XMP file would be ignored. So if you ever want to use a different RAW editor, or Photoshop, you need to accept the limitations on XMP usage.

Sidenote / question:

4. I have discovered with disappointment that when I changed my catalogue settings to automatically write changes on XMP, all my metadata were written on the files that are not RAW. So, xmp were created for all RAW images, but any file that is not RAW, now has the metadata written on it. I also found this to be irreversible. I understand that it is a similar function as an XMP, but written internally in the file but this is something I didn't want and didn't get the chance to chose. This feels a bit strange for Lightroom, so please, do correct me if I have done or perceived something wrong here.

I agree with your concern about backup frequency, but consider this. If you change a JPG, you would want it backed up, even if the metadata did not change.

Phil Burton

 

[Johan Elzenga]

If you ever want to use another raw editor, XMP files from Lightroom are useless anyway (with the exception of Adobe Camera RAW). Because different raw editors use different raw engines, the settings from one editor do not carry over to the other.

 

[clee01l]

My workflow includes a lot of metadata editing and a few adjustments. For reasons of back-up efficiency and data safety I will like to find a way to store and sync all the changes I make on any file within Lightroom, on an XMP sidecar file.

If your interest is in efficiency, don't waste time with XMPs, John has already pointed out that ALL of your metadata is stored in the catalog file. If you are interested in data safety, back up your original image files once AND your LR catalog files frequently. Your backup scheme needs to go no further than this WRT Lightroom and your image data.
XMP sidecar files are a waste of disk space. They have no beneficial purpose outside of sharing Adobe generated metadata within the Adobe family of products

 

[Eleni]

Thanks so much for all the advice.
I think what I may have gotten wrong is how much and what type of information is actually contained in the xmp files. I was looking for an external file to hold all metadata which can be shared and synced between platforms...
When I say all metadata I am referring to keywords,ratings, and any user added metadata. EXIF are embedded within the files and so there's no problem with them.
I was also looking to keep my files intact and have this metadata separate so I can sync and change between platforms easily.
So, in a non destructive platform like lightroom, where are the keywords,gps,ratings etc saved in a RAW + xmp file and where in JPEGs?

 

[johnbeardy]

Lightroom is similar to Aperture - all your work, adjustments, ratings, keywords etc is stored inside the catalogue file.

I was looking for an external file to hold all metadata which can be shared and synced between platforms...

What do you mean here? Different computers, operating sytems?

 

[Johan Elzenga]

Keywords and ratings ARE in the XMP file if you write XMP files (and yes, Aperture can do that), but as said before: Lightroom does not read or write XMP files for any file type except proprietary raw files. If you export a JPEG+XMP from Aperture and import this into Lightroom, Lightroom will ignore the XMP and so it won't carry over the keywords and ratings. The only way to carry over keywords and ratings is by writing them in the EXIF data of the JPEG itself (and Aperture can do that too).

 

[Eleni]

@johnbeardy I mean online platforms to organise my images which utilise user added metadata. I am getting help to develop plug ins to keep these two synced (lightroom/online), so I am trying to figure out where is the best place for the metadata to exist. It seems to me that syncing would be easier through a sidecar file. I was hoping XMPs would do that.

As far as the edits go, it is very likely that they won't be recognised through an XMP, but I was going to create JPEG versions of those. I was hoping to keep the metadata separate, but it looks very complicated.

 

[johnbeardy]

Why not just export the JPEGs to the online platform? Export or via a Publish plugin. Any web platform should be able to read the metadata from the JPEG.

Remember that xmp files are only a hack for proprietary, undocumented files. They are not the norm, and most photo file formats are designed to contain their own metadata (which is in xmp format).

 

[Eleni]

The trick here is that I will be syncing between the online platform and LR, in which case updating a light text file is quicker that updating a JPEG, where you have to copy the whole thing.

@johnbeardy you confused me there slightly. You say that xmp are not the norm but most file formats contain their own metadata in xmp format? If they do, wouldn't that make it some kind of norm? Especially when it's under public patent license?

Is there a norm to sidecar files for people who for one reason or another wish to store and sync metadata that way? Or is such a stupid idea?

 

[johnbeardy]

I italicised "file" and "format", Eleni. The xmp language or syntax is the norm but sidecar "files" are the hack.

 

[clee01l]

The trick here is that I will be syncing between the online platform and LR, in which case updating a light text file is quicker that updating a JPEG, where you have to copy the whole thing.

What online platform uses Adobe XMP files for anything? AFAIK, amp files on a website are still a wasted effort

@johnbeardy you confused me there slightly. You say that xmp are not the norm but most file formats contain their own metadata in xmp format? If they do, wouldn't that make it some kind of norm? Especially when it's under public patent license?

Each file type has specifications for data blocks that make published formats interpretable by any program needing to read then What is commonly called the EXIF is really comprised of several data blocks. The EXIF block contains metadata about shooting specifications (shutter speed etc, the IPTC sections contains information related to publishing as defined by the International Press Communications Council(IPTC) Another defined data block is the Extensible Metadata Platform (XMP) an ISO standard, created by Adobe. JEPG, TIFF, DNG and other Published and open standards include a data block for IPTC and XMP in the file header section. Proprietary file formats like CR2, NEFs are closed and do not have a specification for an XMP data block. For this reason, Adobe (for Adobe's family of programs) created a special file to hold the XMP metadata that Adobe generates OR IPTC metadata IF the proprietary file does not contain an IPTC section or can not be updated by an Adobe app.

Is there a norm to sidecar files for people who for one reason or another wish to store and sync metadata that way? Or is such a stupid idea?

A sidecar file can be any file that has information related to a master file. This can be a JPEG file that was not included as a thumbnail in the original proprietary file, an Audio file (if not embeddable in the original) or an XMP metadata file. The program reading the master file needs to be sidecar file aware and prepared to use the information contained in the sidecar. As others have pointed out, Adobe does not expect or use an XMP sidecar file associated with a JPEG because the XMP section is included in the JPEG specification. Is it a stupid idea to create XMP sidecar files. They are necessary for only proprietary Master files to exchange this metadata between Adobe programs. It is not a stupid idea to create XMP sidecar files for proprietary file formats if a person is not smart enough to backup and protect same the information with LR catalog file backups.

 

[Eleni]

What online platform uses Adobe XMP files for anything? AFAIK, amp files on a website are still a wasted effort

A fully customised one which is looking to sync directly from a lightroom catalogue.

@clee01l thank you for taking the time to explain everything. My information and knowledge is obviously bitty at the moment but I am slowly putting things together.