Writing XMP and ACR files - to overwrite or not to overwrite...

[mstrathmore]

Lightroom Version Number

15.1

Operating System

1. macOS 15 Sequoia

Folks I know that writing XMP has been discussed before - and seemed to be fairly valueless from the perspective of restoring photos in the event of a catalog loss. However I saw that Lightroom now also writes an .ACR file if there are "complicated" edits, such as those involving AI. This appears to make it a better proposition.

Previously, I had turned on:

but that almost immediately corrupted my catalog and I spent almost two months recovering from that (thanks to help from folks on this forum) - so, that's an absolute hard no for me to try again.

Today I tried a "low risk" approach by clicking "Save Metadata" for a single folder containing two or three photos. It worked, and created .XMP and .ACR files for those photos. These were duly automatically uploaded from my NAS, where I store my photos, to BackBlaze - one of my backup destinations.

All seemed well.

I then tried a folder that had a number of DNG files, created back when Lightroom did Denoise by creating a DNG; all my edits, therefore, are on the DNG, not on the RAW. On Clicking "Save Metadata" I immediately get presented with: "One or more of these photos has been changed in an external application. Should Lightroom overwrite the settings on disk?" My only options are "Cancel" or "Overwrite" - Cancel stops the process (and therefore sidecars aren't written) whilst Overwrite happily goes ahead and creates XMP & ACR files.

My question: Does "Overwrite" mean I am destroying the edits I have made to the DNG, by writing the (unedited) RAW information over it?

Ultimately, my desired outcome is that I have a RAW plus sidecar files (or a DNG, in those cases where I have DNGs purely because Lightroom created them during Denoise) which gives me a fighting chance to recover my edits should I lose my catalog.

Some may probably think I am paranoid about catalog loss and backup (I am...) but I can tell you from personal experience that when Lightroom starts crashing every time you open it, and Adobe support "helpfully" helps you create a blank catalog, then imports all your photos into it, with two months of ensuing chaos, your blood runs pretty cold!

Cheers,

Mark.
PS. sorry for being so busy on the forums: can you tell work hasn't ramped up yet for the year?

 

[John R Ellis]

Does "Overwrite" mean I am destroying the edits I have made to the DNG, by writing the (unedited) RAW information over it?

Selecting Overwrite will replace all metadata in the DNG -- Develop settings, keywords, captions, capture date, and all other fields shown in the Metadata panel -- with what's currently in the catalog.

The design intent of the Overwrite warning is to handle this use case: You've edited the file in another application but you haven't done Read Metadata From File to bring the changed metadata back into the catalog. LR doesn't know which version of the metadata to keep, that in the catalog or that in the file, so it warns you.

Unfortunately, LR often gets the metadata status of files wrong and the Overwrite warning is spurious. If you're confident you haven't edited the file in an external app, just select Overwrite, and you'll be sure that what's in the file is the same as what's in the catalog.

 

[John R Ellis]

Also, I'm in the minority of people who use .xmp / .acr sidecars as part of my backup strategy: Hourly Time Machine backups going back 6 months on two external disks, continuous Backblaze backups to the cloud, and sidecars if everything else screws up (or if I accidentally deleted from the catalog a file I've edited in the last hour, which I seem to do once a year despite a reject/delete-rejected workflow).

 

[mstrathmore]

Selecting Overwrite will replace all metadata in the DNG -- Develop settings, keywords, captions, capture date, and all other fields shown in the Metadata panel -- with what's currently in the catalog.

The design intent of the Overwrite warning is to handle this use case: You've edited the file in another application but you haven't done Read Metadata From File to bring the changed metadata back into the catalog. LR doesn't know which version of the metadata to keep, that in the catalog or that in the file, so it warns you.

Unfortunately, LR often gets the metadata status of files wrong and the Overwrite warning is spurious. If you're confident you haven't edited the file in an external app, just select Overwrite, and you'll be sure that what's in the file is the same as what's in the catalog.

Thanks John, this is very useful. I have an app called "DIM5" that renames my photos automatically, and then I use Geotag Photos Pro to geotag them, but that happens prior to import. I literally have no other apps other than LightRoom and Photoshop which edit the actual photo. I rarely "Edit in..." Photoshop from Lightroom if I want to expand a canvas or something like that, but 99.9% of the time the only editing software that sees my photos is Lightroom.

Based on what you said, I think it looks safe to choose "overwrite" and that's also the only way to get the sidecar file too (like you I also believe in using the sidecar as part of my backup, though in my case Time Machine only protects my catalog (which also goes to OneDrive, Synology NAS, Backblaze and periodically, to offsite HDD).

Cheers,

Mark.

 

[Jim Wilde]

I then tried a folder that had a number of DNG files, created back when Lightroom did Denoise by creating a DNG; all my edits, therefore, are on the DNG, not on the RAW. On Clicking "Save Metadata" I immediately get presented with: "One or more of these photos has been changed in an external application. Should Lightroom overwrite the settings on disk?" My only options are "Cancel" or "Overwrite" - Cancel stops the process (and therefore sidecars aren't written) whilst Overwrite happily goes ahead and creates XMP & ACR files.

Did you verify that XMP/ACR sidecar files are created for those DNGs? That shouldn't happen, all the "sidecar" data should be stored within the DNG file itself. Same with other non-proprietary Raw files, it should be only proprietary raw files that will get XMP/ACR sidecar files when saving metadata to file.

 

[mstrathmore]

Did you verify that XMP/ACR sidecar files are created for those DNGs? That shouldn't happen, all the "sidecar" data should be stored within the DNG file itself. Same with other non-proprietary Raw files, it should be only proprietary raw files that will get XMP/ACR sidecar files when saving metadata to file.

Hi Jim,

Yes, it looks like only the RAWs get an XMP (and an ACR if there are advanced edits). Here's an example of one file that I denoised - any subsequent editing is obviously only done on the .dng file.

To be honest I am not sure how to interpret this.

• If all of the information in the sidecars is now stored within the DNG as a result of me clicking "Save metadata" then I'm a happy camper because I have a good backup that contains all my edits.

• If, however, the dng doesn't contain the editing information, then the XMP and ACR sidecars are not really useful since they only apply to the RAW file (which is essentially unedited), and I need to figure out how to update the DNG to contain the sidecar data.

Cheers,

Mark.

 

[Jim Wilde]

• If all of the information in the sidecars is now stored within the DNG as a result of me clicking "Save metadata" then I'm a happy camper because I have a good backup that contains all my edits.

It does this.
Easy to test: save Metadata to XMP/ACR for an edited DNG file, then create a temporary catalog and import that DNG file into it. Observe the edited (and other metadata) state.

Note that saved XMP (for all file types) does not contain all metadata (which leads many users to think that DNG is not a valid backup), things like edit history, collection membership, virtual copies are not included. But if you have a corrupt catalog with no valid backups you'd be delighted if you have all your images files containing saved XMP/ACR data. Of course it's not a complete substitute for a rock-solid backup regime, but in an emergency it would be very welcome.

 

[clee01l]

Did you verify that XMP/ACR sidecar files are created for those DNGs? That shouldn't happen, all the "sidecar" data should be stored within the DNG file itself. Same with other non-proprietary Raw files, it should be only proprietary raw files that will get XMP/ACR sidecar files when saving metadata to file.

DNG files should store the XMP data internally. What happens if there is also ACR data for a DNG file? Is it stored internally or as a sidecar?

 

[Jim Wilde]

DNG files should store the XMP data internally. What happens if there is also ACR data for a DNG file? Is it stored internally or as a sidecar?

Internally.

 

[PhilBurton]

but, but, but. If you use a DNG, assuming you back up up your originals (and I sure hope you do!!!) every time you change keywords or do DEVELOP operations, those changes will be change the DNG and force a backup of say 20 - 60 MB. using up a lot of backup storagage. On the other hand, if you use a native RAW file, and export XMP and ACR files and you back up all your system files, you will need to back up just those files.

If you don't do those exports, then you will need to just back up your catalog file, which I hope you do after every Lightroom Classic session.

 

[mstrathmore]

It does this.
Easy to test: save Metadata to XMP/ACR for an edited DNG file, then create a temporary catalog and import that DNG file into it. Observe the edited (and other metadata) state.

Note that saved XMP (for all file types) does not contain all metadata (which leads many users to think that DNG is not a valid backup), things like edit history, collection membership, virtual copies are not included. But if you have a corrupt catalog with no valid backups you'd be delighted if you have all your images files containing saved XMP/ACR data. Of course it's not a complete substitute for a rock-solid backup regime, but in an emergency it would be very welcome.

Hey Jim, that's awesome - thank you!

I totally agree with you: as much as it would be nice to have the full history and collections etc, what I'm really most interested in is a bullet proof copy of the final, edited, product.

Thanks again and take care.

Mark.

 

[mstrathmore]

but, but, but. If you use a DNG, assuming you back up up your originals (and I sure hope you do!!!) every time you change keywords or do DEVELOP operations, those changes will be change the DNG and force a backup of say 20 - 60 MB. using up a lot of backup storagage. On the other hand, if you use a native RAW file, and export XMP and ACR files and you back up all your system files, you will need to back up just those files.

If you don't do those exports, then you will need to just back up your catalog file, which I hope you do after every Lightroom Classic session.

Hi Phil,

I'm backed up to the wazoo, to be honest, so I think I'm covered:

Raws are stored on a Synology NAS, the Photos folder on the NAS backs up any changes in real time to BackBlaze. I also periodically back up the entire thing to a separate hard drive that's kept offsite. The Catalog is stored on my Mac, and is backed up to OneDrive - to a folder that replicates to my NAS, which then runs a Cloud backup job to BackBlaze. Similarly, that is also kept on a separate offsite hard drive.

The weak link in all of this is the catalog. I've encountered the situation where all of the above didn't protect me, because Lightroom somehow took umbrage to my catalog and every backup.

It's for this reason that I've been pursuing a means of backing up the "final edited" copy of each photo, that is viewable in finished form including all edits, outside of the Adobe ecosystem and independent of the catalog.

Cheers,

Mark.

 

[jrobnz]

Be careful with your catalog in One Drive. In general that is not recommended since it has caused many people to end up with corrupt catalogs. Possibly it was even the cause of yours.

 

[Gnits]

I had a personal horror story in relation to One Drive .... triggered by purchasing and travelling with a new Windows laptop, which started synching massive amounts of data (unknown to me, incl catalogs, raw files , personal files, etc) between my main desktop at home, my old laptop, my new laptop and a travelling MacAir. I had travelled to a remote island off the West coast of Ireland, with the new Win laptop. The island had very thin and unreliable internet connectivity.... not helped by the fact that OneDrive on may laptop was consuming 99% of it for 99% of the time. I discovered the problem a few days later when my laptop started filling up. It took me several days to unravel the mess and understand the cause and I know to this day I did not fix everything. The issue was that no version of the truth was complete or fully synchronised and I deleted stuff thinking I was just tidying up a new Win laptop.

The real problem was I did not understand at the time the combination of One Drive and Adobe Lightroom Classic default behaviour. This may be an extreme case .. but it happened to me.

The problem a lot run into is that Catalogs may be synched between a local drive and One Drive Cloud... but who controls or knows when a synch is complete or what direction the synch is taking place... leading to corrupt catalog (or other files .. eg preferences).

 

[mstrathmore]

Be careful with your catalog in One Drive. In general that is not recommended since it has caused many people to end up with corrupt catalogs. Possibly it was even the cause of yours.

Hi, 100% I hear you: after reading this and other forums I had the same view of OneDrive, so I have to clarify what I said above a little.

I only use OneDrive for the catalog backup - my catalog lives on my Mac HDD. So basically what happens is when I shut down LR, it backs up to my OneDrive folder on my Mac, which, once fully uploaded, triggers a sync between OneDrive and my NAS. Once the files are on the NAS, it then syncs up to Backblaze.

 

[mstrathmore]

I had a personal horror story in relation to One Drive .... triggered by purchasing and travelling with a new Windows laptop, which started synching massive amounts of data (unknown to me, incl catalogs, raw files , personal files, etc) between my main desktop at home, my old laptop, my new laptop and a travelling MacAir. I had travelled to a remote island off the West coast of Ireland, with the new Win laptop. The island had very thin and unreliable internet connectivity.... not helped by the fact that OneDrive on may laptop was consuming 99% of it for 99% of the time. I discovered the problem a few days later when my laptop started filling up. It took me several days to unravel the mess and understand the cause and I know to this day I did not fix everything. The issue was that no version of the truth was complete or fully synchronised and I deleted stuff thinking I was just tidying up a new Win laptop.

The real problem was I did not understand at the time the combination of One Drive and Adobe Lightroom Classic default behaviour. This may be an extreme case .. but it happened to me.

The problem a lot run into is that Catalogs may be synched between a local drive and One Drive Cloud... but who controls or knows when a synch is complete or what direction the synch is taking place... leading to corrupt catalog (or other files .. eg preferences).

Hi, I don't trust OneDrive that much, to be honest - but for me it's the "least worst" of the cloud storage solutions. Because it was the first service I used, back in the SkyDrive days, and because it's so integrated into Office which I have to use for work, and OneNote which contains my life, I settled on it.

Based on a lot of good advice on this forum (I think some of it actually came from you!) I stay far away from using it as the place where I store the live catalog - as I mentioned to jrobnz above, I only store my backup there - hopefully eliminating any issues that would result from Lr trying to access the file whilst the file is still uploading (which would be the case with the live catalog). The main problem with using OneDrive (or any cloud storage solution for that matter) to store the live catalog is that, almost irrespective of you internet speed, a 6+ Gb file is going to take time to access. So either Onedrive locks the file while syncing - which causes problems - or, if it doesn't, Lr will continue to write to a file that only really partially exists - which also will cause problems. Either way, OneDrive for the live catalog is a bad idea (IMHO).

I think it's ok for the backup though, unless you think otherwise?

 

[clee01l]

because it's so integrated into Office which I have to use for work, and OneNote which contains my life, I settled on it.

Really? I run OneNote as a standalone , OneDrive is not an app that runs on my computer. It is not that integrated into Office any more than my Documents and Pictures folders are integrated into iCloud. OneNote syncs to my One Drive cloud storage but only while OneNote is running and only because I can't figure out how to save N otebooksto iCloud. I hate Outlook and its bloat. I will not install it on my Mac. Preferring Spark as my email client on all of my devices. I've been using OneNote for over 11 years initially Importing from EverNote.

Excel and Word save to my iCloud Storage.

 

[Gnits]

I stopped using OneDrive and then uninstalled it completely when I ran into issues with a Win 11 upgrade and was trying to slim down all the background processes and unwanted junk introduced by Win 11 update. That took me weeks to do… and probably need to revisit that again (and maybe OneDrive is back… I need to check.



Instead, when I want to synch anything… I use Dropbox. Dropbox is turned off 99% of the time, only active when I have a specific task to do.. When I want to synch something (say to another of my computers or to share files with someone or to invite someone to share files with me…. I copy the files to a temp folder in Dropbox.. then turn on Dropbox. On the destination drive I copy the files to their new destination(ie outside of Dropbox). I then delete the temp Dropbox folder. So I only use Dropbox Cloud Synch services for specific device to device synch. At any stage I can lose the complete contents of my dropbox folders, because they only ever contain temp copies. I really like the Dropbox Request Files feature.

If I check my Dropbox folder it always has a tiny footprint (in terms of disk size)… and the only content that survives more than a day or two are documents I need for travel, such as copies of travel insurance, tickets, schedules, hotel reservations, etc. Ie.. available to me my phone, iPad or laptop in an emergency.

I also did not like how Win 11 handled the Task Bar… and knew before I moved to Win 11 that would be the case. So I built a tiny Windows App with a series of buttons, when clicked launch my most used apps, web URLs, etc. The Lightroom Queen Forums is an example of one such button. I have a 32 inch screen, so plenty of real estate to place my personal dashboard in a corner.. In my little app.. I have a big box which displays the current date and time in a large font. Visible to me regardless of whether I am wearing glasses or not. Two vip large buttons are a) “Turn On Dropbox” …It turns on Dropbox service annd displays the Dropbox folders in Explorer and b) Shutdown My Computer. The Dropbox button is only tapped when I want to synch using Dropbox . The shutdown button does a background check that certain vip apps are no longer running (such as Lightroom) and warns me if they are still active. If none of my vip apps are running it starts a 30 sec countdown and then does a full PC shutdown. When I startup again, my Win device only launches apps and services of my choosing… and background services, such as Dropbox do not get started.

This radically simplifies my computer life. I have a lot of devices, laptops and various servers… but controlling what is happening on my main desktop is the key to a simple workflow and min stress computing.

I use Macrium Reflect to backup my system drive, data drive, catalogs and library of images to a spinning internal backup drive. I do regular Beyond Compare synchs of my backup data to a remote NAS drive . So I am using real local apps, configured as I wish to automatically manage my backup strategy. I do not have any Cloud Synch Services as a component of my backup workflow. At the moment, I do not have a backup to a cloud service such as Backblaze, Amazon S3, etc… as I am comfortable with my current level of resilience… but may change my view if I move house or do a major upgrade of my own computer hardware setup.

We evolve our own workflows that ultimately best suit our real world life and accumulated experience. I am using these snippets as examples of various approaches and not pushing them as the only or best option for everyone.