XMP Automatic Write

[gideon.liberman]

Lightroom Version Number

14.5.1

Operating System

1. Windows 11

I have implemented the Automatic XMP Automatic Write to enhance the safety features provided for rebuilding a corrupted catalog.

Subsequently, it initiated an extensive file update process, which took many hours.

I would appreciate your clarification on several points that are unclear:

When importing, I routinely convert all my RAW files to DNG. The inclusion of the XMP file within the DNG package has increased the developed DNG file size by 2-10MB (for Sony A7RV files). Could that volume increase be related to the number of Develop activities? Is this typical for XMP file sizes?

I have noticed that undeveloped JPG files are also being re-synced to the catalog, including both image and metadata. Could you explain why this occurs?

Since I use OneDrive, which operates via synchronization, all my images are being re-uploaded to the cloud. Am I correct in assuming that with every adjustment made in Develop, these RAW files will require re-syncing each time?

Similarly, as I also use i-Drive for backup purposes, will all my images be uploaded as new versions with every future adjustment in Develop? Is my understanding correct?

Considering the implications for OneDrive Sync and i-Drive Backup, I am questioning whether converting to DNG is advisable. While using independent XMP sidecar files results in smaller individual uploads, it does increase the total number of files. What would you recommend under these circumstances?

 

[Zenon]

I'm not going to answer your question directly. Regarding XMP files. If my drive failed then I'd also lose the XMP files. I back up my catalogue regularly which is also on the OS. The OS is backed up to an external drive so everything is there. I could even back that up to more drives or the cloud. Personally I have no need for XMP files for my workflow so I consider them clutter. Just thought I'd throw that in.

 

[Replytoken]

My recommendation are probably not what you want to hear, but I would make (and occasionally test) backups of your catalogs instead of creating XMP files, and also disconnect OneDrive (OD). To start with the latter, OD is generally the source of many issues here in the forum, and there are better options for backing up your files that do not cause the issues that OD often does.

Regarding DNG, what was your intent when you decided to convert your files to the DNG format? For years I converted my files to DNG, in addition to keeping the raw versions, and then decided to just work off of the raw files. The DNG wrapper offered error checking, but I did not routinely check the files, so the value of converting was even smaller. And the LR instructions that reside in your XMP file, or inside of your DNG file, are only readable by a few select programs.

Even if you continue to convert, I would strongly suggest turning off OD to avoid any possible issues it can create. And if you want to learn more about what OD can do, I recommend searching the forum and reading through the results.

Good luck,

--Ken

 

[Johan Elzenga]

When importing, I routinely convert all my RAW files to DNG. The inclusion of the XMP file within the DNG package has increased the developed DNG file size by 2-10MB (for Sony A7RV files). Could that volume increase be related to the number of Develop activities? Is this typical for XMP file sizes?

Not so much the number of develop activities, but the kind. If you create AI masks, or use AI Remove, then the XMP increases quite a lot. If you use Denoise or Super Resolution then it will explode.

I agree with others that I would not recommend using XMP as a kind of backup system. Make a catalog backup each time you do something in the catalog, so set that option to ‘Every time Lightroom quits’. There are several things in the catalog that are not saved to XMP, so XMP is a poor substitute for a catalog backup. Some people do both, arguing that a crash that corrupts the catalog will still make you lose all edits you made since the last backup, while writing to XMP would only make you lose the very last edit you were working on when the crash happened. I’m willing to take that small risk.

 

[John R Ellis]

Note that LR 14.5 introduced a severe performance bug with Catalog Settings > Metadata > Automatically Write Changes Into XMP:
https://community.adobe.com/t5/ligh...ise-on-large-numbers-of-photos/idi-p/15393188

If you use Denoise, even on a fraction of your photos, the size of your catalog could explode by two orders of magnitude and LR's interactive performance can get very poor.

I've turned off that option for my catalog. There have been other issues reported about it in the last couple of years, and it's not clear that Adobe is really supporting it any more. We'll see if LR 15 fixes the problems.

 

[Califdan]

I have implemented the Automatic XMP Automatic Write to enhance the safety features provided for rebuilding a corrupted catalog.

Last resort recovery option which you should never have to resort to if you have a solid backup strategy that you test from time to time

When importing, I routinely convert all my RAW files to DNG. The inclusion of the XMP file within the DNG package has increased the developed DNG file size by 2-10MB (for Sony A7RV files). Could that volume increase be related to the number of Develop activities? Is this typical for XMP file sizes?

XMP 'data' (not 'file') is embedded in DNG files but I got your meaning. Yes the complexity and quantity of the edits can significantly raise the size of the file

I have noticed that undeveloped JPG files are also being re-synced to the catalog, including both image and metadata. Could you explain why this occurs?

I don't know what you mean. Re-synced from LR/Cloud?

Since I use OneDrive, which operates via synchronization, all my images are being re-uploaded to the cloud. Am I correct in assuming that with every adjustment made in Develop, these RAW files will require re-syncing each time?

First DO NOT HAVE YOUR LRC CATALOG UNDER CONTROL OF ONE DRIVE!

For images, ANY change to the image file such as any metadata change or any develop module edit when that XMP data is saved to disk will trigger OneDrive to re-sync the file to the cloud as well as any other backup tools you use. In the case of very large files such as DNG, TIFF, AND PSD files which embed the XMP data inside the file, Cloud sync can quickly be overwhelmed if you save XMP automatically as some simple change like changing the star rating, or the spelling of a keyword can trigger a massive number of these large images file to need to be re-synced.

Similarly, as I also use i-Drive for backup purposes, will all my images be uploaded as new versions with every future adjustment in Develop? Is my understanding correct?

Same thing. When I was converting my RAW files to DNG this is what forced me to abandon auto save of XMP. This also influenced my decision to stop converting my RAW's to DNG's on import - but I'm still stuck with those DNG's from that period as I no longer have the RAW files.

Considering the implications for OneDrive Sync and i-Drive Backup, I am questioning whether converting to DNG is advisable. While using independent XMP sidecar files results in smaller individual uploads, it does increase the total number of files. What would you recommend under these circumstances?

If you stick with DNG's - turn off the Auto Save of XMP's. If you stop converting ARW's to DNG's, once you've gotten to a point where the DNG's are rarely touched, you MAY consider turning the auto save of XMP data back on, but there are long standing bugs in LrC that mark images as having been changed even when they haven't. If that stays in the catalog no problem, but the Auto-Save of XMP data uses that flag to trigger the write.

So, best is to stop the XMP auto saves and assure your catalog and image backups are in proper working order and you test it once or twice a year.

 

[gideon.liberman]

I have identified the issues caused by XMP Automatic Write, which resulted in catalog and file ballooning, and have since disabled this feature. I appreciate having consulted the forum for guidance.

Subsequently, I reverted to an earlier backup of the catalog taken prior to implementing XMP Automatic Write. I am currently addressing conflicts arising from using an older catalog with a more recent cloud version, primarily metadata discrepancies and missing newer files.

To enhance my workflow, I am setting up OneDrive on a dedicated data disk that stores my photos and catalog backups, while the catalog itself remains on the system disk. I am also establishing a comprehensive backup routine with i-Drive for both system and data disks.

Thank you all for your valuable advice, which has helped prevent a potential ongoing issue.

 

[Califdan]

Subsequently, I reverted to an earlier backup of the catalog taken prior to implementing XMP Automatic Write. I am currently addressing conflicts arising from using an older catalog with a more recent cloud version, primarily metadata discrepancies and missing newer files.

You may want to look at these two articles on my website. Case 3 in article 025 seems to be your situaiton. Depending on what specific 'conflicts' you are experiences, article 026 my provide some guidance

https://www.danhartfordphoto.com/lightroom-articles-blogs-index

 

[gideon.liberman]

@Califdan
Most of the conflicts are Metadata which I try to Sync their folders + Scan for Metadata updates.
in case of using a former catalog, is it advisable to copy both .lrcat & .ltcat-data?
will copying both reduce the metadata conflicts?

 

[Califdan]

Most of the conflicts are Metadata which I try to Sync their folders + Scan for Metadata updates.
in case of using a former catalog, is it advisable to copy both .lrcat & .ltcat-data?

the .lrcat and .lrcat-data are the only two catalog related files that LrC cannot rebuild if somehint happens to them. The rest are for performance purposes and they go misisng, LrC can recreate the data in them as needed.

will copying both reduce the metadata conflicts?

No. The conflict is between the metadata stored in/with the image files and the metadata stored in the catalog. Restoring catalog files will not correct this unless the metadata in the restored catalog files just happens to match that stored in/with the images and if it does, it is probably

Trying to keep metadata (XMP data) matched between the two, while laudable, is not always possible. For a significant number of people (including myself) keeping the XMP consistent cannot be done for on a larg number (most) images in the catalog as soon as the data is made consisted, it becomes inconsistent (within a few seconds). This is a long standing bug going back many years which Adobe has yet to remedy. But, even if you can keep the XMP data stored in/with the image files consistent with what is in the catalog it's not really all that useful unless you have ignored all our warnings about backing up the catalog. The reason is that the catalog contains a lot of information that does not find it's way to the XMP data on disk. Among that information is: all information concerning collections, virtual copies, snapshots, edit history and some metadatas that is not part of the XMP standard. You will save a lot of time and effort if you devote your energy to assuring you have good backups of both your images and your catalog rather than trying to keep the metadata consistent.

 

[gideon.liberman]

I had my Adobe system working flawlessly: it synced correctly, there were no metadata conflicts, and my catalog size was small. However, after attending the last Lr virtual summit and following a single recommendation to enable Auto XMP, things changed.

When I activated Automatic XMP writing, I quickly noticed that my Catalog .lrcat file doubled in size, picture files ballooned, and after seeking advice here, I turned it off. I restored a backup catalog that was four days old. I had only two uploads of 11 photos total after that backup.

Right after starting up LrC, I saw two photos uploading and about 8,500 photos downloading, with approximately 3,500 ongoing metadata conflicts. The .lrcat and lrcat-data files also grew noticeably larger. I'm baffled by how this chaos happened and question whether the Catalog Backup Megaproject is justified, since it's causing such significant misalignment.

What a system!

 

[Califdan]

We need to get straightened out on the use of the word "Sync". LrC uses "Sync" or "Synchronize" in reference to 4 completely different things. I think the two you are referring to are "Cloud Sync" which passes info back and forth between LrC and Adobe Cloud and "XMP Sync" wich copies metadata between the LrC catalog and image files on disk (or their sidecars). So far I thought we were talking about XMP Sync but your last post seems to also talk about 8,500 images using Cloud Sync. While these two can trigger one another, they are very different processes.

I had my Adobe system working flawlessly: it synced correctly, there were no metadata conflicts, and my catalog size was small.

Just to be sure, by metadata conflicts you mean the little icon in the upper right corner of images in the grid

However, after attending the last Lr virtual summit and following a single recommendation to enable Auto XMP, things changed.

When I activated Automatic XMP writing, I quickly noticed that my Catalog .lrcat file doubled in size

That is very odd and I know of now rational explaination on how enableing LrC to automatically save metadata to disk could double the size of your ".lrcat" file. I can imagine it enlarging one of the other catalog helper files as it may have queued up tens of thousands of images for that action. I suppose it is possible that it keeps such a queue of images needed its XMP data saved in the '.lrcat' file.

picture files ballooned

As Develop Module edits are included in this XMP data, I can see where image files on disk that store such metadata inside the image file (e.g. DNG, TIFF, JPG, PSD) could get much larger depending on how much editing had been done

, and after seeking advice here, I turned it off. I restored a backup catalog that was four days old. I had only two uploads of 11 photos total after that backup.

Restoring the catalog would not affect the enlargement of the image files but it probably would have dealt with the ".lrcat" file size issue.

Right after starting up LrC, I saw two photos uploading and about 8,500 photos downloading, with approximately 3,500 ongoing metadata conflicts.

This is where I think you switeced to talking about "Cloud Sync". Is that correct? When you switch which catalog is syncing to LR/Cloud it needs to reconcile what is synced. If the newly synced catalog is a blood relative of the previously synced catalog - which it would be if it was from a backup - this goes pretty quickly as even though the sync status box shows large numbers of images queued up to sync each way, it's really only transmitting small bits of info for each image.

The .lrcat and lrcat-data files also grew noticeably larger. I'm baffled by how this chaos happened

the .lrcat-data file mostly contains masking and AI edit data. it's not clear why this would get bigger unless you applied some change to a large number of images either in LrC or LR/Cloud.

and question whether the Catalog Backup Mega project is justified, since it's causing such significant misalignment.

I'm not clear what "Catalog Backup Mega project" referrs to. But, if you are referring to using saved XMP data as part of your protection strategy it is only marginally useful.

If you have the data, you can restore both your catalog files and image files to pre-problem point in time and start over from there. But, at the same time, you may also want to clear LR/Cloud and start fresh there too but that is another ball of wax (see: https://www.danhartfordphoto.com/blog/2025/2/lr027-clear-lr-cloud-and-stat-over-with-syncing )

 

[gideon.liberman]

Just to be sure, by metadata conflicts you mean the little icon in the upper right corner of images in the grid

Correct

That is very odd and I know of now rational explaination on how enableing LrC to automatically save metadata to disk could double the size of your ".lrcat" file. I can imagine it enlarging one of the other catalog helper files as it may have queued up tens of thousands of images for that action. I suppose it is possible that it keeps such a queue of images needed its XMP data saved in the '.lrcat' file.

my Catalog enlarged from 1.9GB to 3.5, I also read a similar reporting in this forum

As Develop Module edits are included in this XMP data, I can see where image files on disk that store such metadata inside the image file (e.g. DNG, TIFF, JPG, PSD) could get much larger depending on how much editing had been done

as I use DNG format, the files ballooned by ~10-20MB related to the develop process extent.

Restoring the catalog would not affect the enlargement of the image files but it probably would have dealt with the ".lrcat" file size issue.

that is my intention

This is where I think you switeced to talking about "Cloud Sync". Is that correct?

Correct.

blood relative of the previously synced catalog

Correct

this goes pretty quickly

half the way after few hours!

.lrcat-data file

this is now inflating by ~500GB, only small amount of develop activities (except the XMP switch - on/off which affected
all the DNG population)

"Catalog Backup Mega project"

I have invested in in 3 deferent types of backup/sync process, to local/cloud/off-line disks destinations, Dayli/weekly, all the live Catalog files/backup files

If you have the data, you can restore both your catalog files and image files to pre-problem point in time and start over from there

I have all the needed data, but despite the blood relative backup which was only 11 photos before switching the XMP - the parallax of this catalog vs the cloud is irrational to the number of changes, and frustrating vis-a-vis the Backup infrastructure that was prepared for this D Day

you may also want to clear LR/Cloud and start fresh

I have read it, it looks intimidating, I still hope to straighten the discrepancies Catalog vs the Cloud
second thought - if i was using proprietary files thus separated XMP file, I could simply delete the XMP files!

 

[John R Ellis]

LR 15.0 fixed the severe performance but with Denoise, Super Resolution, and Automatically Write Changes Into XMP. If your catalog's .lrcat file (the SQL database file) ballooned in size, see here for how to recover that space:
https://community.adobe.com/t5/ligh...large-numbers-of-photos/idc-p/15567524#M64295

Note that the .lrcat-data file remains large in LR 15.0, since it is the primary storage for the voluminous Denoise and Super Resolution data.