• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom 6 questions. All the information you need is in Adobe Lightroom 6 - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. When you upgrade to subscription, there's also a Lightroom Classic version available.

Develop module .xmp sidecar files - which LR setting generates these?

Birdbrain186

Member
Joined
Apr 4, 2018
Messages
36
Location
Birmingham UK
Lightroom Experience
Intermediate
Lightroom Version
6.x
Lightroom Version
Lightroom v 6.14 (Perpetual licence)
Operating System
  1. Windows 10
I have recently found that .xmp "sidecar" files are no longer being generated for .CR2 files that I have edited.

I think (but am not 100% certain) these have always in the past been generated on importing the .CR2 files rather than at the time of carrying out the edits.

I do know that I can subsequently generate the .xmp files by using either Develop / "Save metadata to photo" or Library / Metadata / Save Metadata to file .

  • However, I am unclear what I must have done to stop the .xmp files being automatically generated, so would be grateful if anyone can please tell me.

Regards

Mike
 

Birdbrain186

Member
Joined
Apr 4, 2018
Messages
36
Location
Birmingham UK
Lightroom Experience
Intermediate
Lightroom Version
6.x
I have recently found that .xmp "sidecar" files are no longer being generated for .CR2 files that I have edited.

I think (but am not 100% certain) these have always in the past been generated on importing the .CR2 files rather than at the time of carrying out the edits.

I do know that I can subsequently generate the .xmp files by using either Develop / "Save metadata to photo" or Library / Metadata / Save Metadata to file .

  • However, I am unclear what I must have done to stop the .xmp files being automatically generated, so would be grateful if anyone can please tell me.

Regards

Mike
P.S. I suspected it might be due to my having recently unticked the Metadata setting "Automatically write changes into XMP".
However I have found that re-ticking that setting still does not cause .xmp files to be generated (neither on import nor on making edits)
 
Joined
Feb 1, 2010
Messages
13,296
Location
West Sussex, UK
Lightroom Experience
Advanced
Lightroom Version
Classic
Yes, that is the option which controls the automatic creation of XMP sidecars, They are only created when something is changed, i.e. metadata or edits, so unless you apply changes during import you would not expect to see the sidecar files created at that stage.

If you have now re-checked that option again, you may find it will take some time to "catch up" as Lightroom should now start updating the files with all metadata/edit changes made since the option was unchecked.
 

Birdbrain186

Member
Joined
Apr 4, 2018
Messages
36
Location
Birmingham UK
Lightroom Experience
Intermediate
Lightroom Version
6.x
Thanks for your prompt response and clarifying that for me Jim.

I had wondered if it might just be a delay due to the catalogue "catching up" after the change.

I'm now wondering though if it might be better for me to uncheck it again and "live with" having to write the .xmp data after each batch of pictures processed.

Reason I say this is because the only reason I'd unchecked the option was (as per helpful advice from your fellow Guru Hal) in order to avoid my previous issue of Lightroom changing the "creation date" of pictures imported from my iPhone when I add keywords to them.
(This only relates to photos received via WhatsApp that (apparently) have no prior "creation date" until LR generates one on import; but which date is then changed to the subsequent "current" date & time of whenever I do edits / keywording - which "wrong" dates are then very confusing!!)

I don't suppose by any chance you know any alternative way I might be able to avoid that earlier issue without stopping the sidecar files automatically being generated?

Many thanks

Mike
 
Joined
Jun 20, 2009
Messages
18,370
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service

I'm now wondering though if it might be better for me to uncheck it again and "live with" having to write the .xmp data after each batch of pictures processed.

It would help to understand why it is important to “write the .xmp data after each batch of pictures processed.” What do you mean by this?
The metadata is maintained inside the Lightroom Catalog and the exported JPEG will have the XMP section included in the header. Only RAW files because they are proprietary need a separate file for the XMP. RAW files are not much use in any app that can not process the RAW data and turn it into RGB pixels.


Sent from my iPad using Tapatalk
 

Birdbrain186

Member
Joined
Apr 4, 2018
Messages
36
Location
Birmingham UK
Lightroom Experience
Intermediate
Lightroom Version
6.x
Most of my photos are RAW files which I then export as .jpgs after editing & keywording etc.

Until recently I have always had the "Automatically write changes into XMP" ticked so have always had (and have never before questioned the need for) both .cr2 & .xmp files kept on a different drive from the processed .jpgs in an archive - solely for backup purposes.

I have always thought that the .xmp files were necessary to avoid the need to re-process all edits / keywording on the RAW files in the event of any loss of processed .jpgs, or corruption of the LR catalog.

However, if I'm following you correctly, I am now coming to the realisation that as long I keep my catalog backups on the same drive as the original RAW files (ie separate from my main .jpg archive) that this would probably be adequate safeguard for even a catastrophic corruption / failure of the main archive HD, and if I did that, then there would then be no need to write the metadata to ,xmp files.

Thanks Cletus for your comments which have helped me gain a better understanding, and in the light of which I can now again turn off the "Automatically write changes into XMP" option, which should then avoid any "creation date" issues with the "WhatsApp" sourced photos from my iPhone.
 
Joined
Jun 20, 2009
Messages
18,370
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Yes the automatic “Write Changes to XMP” is completely unnecessary. All metadata including some not saved in an XMP file is always stored in the catalog file. The Original RAW file contains shot related XMP (Capture date, F-Stop, Shutter speed etc.) And this is never changed by Lightroom.

If you have to resort to reimporting to capture the Develop settings again in a Lightroom Catalog, you have screwed up royally. You want frequent catalog backups and can quickly revert to a very recent backup if you need to recover from a catalog corruption. Sometimes you may need to recover from a “stupid user mistake” and backup catalogs are critical to recover. I have never had a corrupt catalog file in the 12 or so years that I have been using Lightroom. I have had to merge a 6 month old backup into my master catalog to recover from a “stupid user mistake”

I can’t stress how important it is to use a system backup app to automatically back up your critical files (catalog and images). Making copies is not backup. What Adobe calls a catalog backup is nothing more than a snapshot copy. This is why you want a system backup running at all times to protect your critical data in the eventual disk failure that will happen to every disk.


Sent from my iPad using Tapatalk
 

Birdbrain186

Member
Joined
Apr 4, 2018
Messages
36
Location
Birmingham UK
Lightroom Experience
Intermediate
Lightroom Version
6.x
Thanks again.

At present I don't have any system backup in place.

Excuse my ignorance but could you expand a little please on why having duplicate copies of all .jpg files / original RAW files and regular catalog backups kept on a separate drive would not be adequate to enable retrieval after a disk failure (other than if both drives were to happen to fail simultaneously)
 

PhilBurton

Lightroom enthusiast (but still learning)
Premium Classic Member
Joined
Nov 16, 2015
Messages
2,578
Location
California, USA
Lightroom Experience
Intermediate
Lightroom Version
Classic
Thanks again.

At present I don't have any system backup in place.

Excuse my ignorance but could you expand a little please on why having duplicate copies of all .jpg files / original RAW files and regular catalog backups kept on a separate drive would not be adequate to enable retrieval after a disk failure (other than if both drives were to happen to fail simultaneously)
@Birdbrain186

MacOS has a continuous backup utility. I'm not sure that Windows still does. Microsoft may have removed it with Windows 10.

To expand on Cletus' point, making copies at some regular interval means that between backups, your data is not backed up. That's a potential risk. Only you can decide how bad that risk is. For me, when I'm doing an intense Lightroom session, especially with lots of edits, I quit Lightroom about once an hour, so that I can get a backup of the catalog. If you can tolerate more risk, then just quit Lightroom at the end of the day.

Of course the backup of the catalog is on a separate physical drive from the catalog itself. I also backup ALL data file changes daily to a separate physical drive.

Phil Burton
 
Joined
Jun 20, 2009
Messages
18,370
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Thanks again.

At present I don't have any system backup in place.

Excuse my ignorance but could you expand a little please on why having duplicate copies of all .jpg files / original RAW files and regular catalog backups kept on a separate drive would not be adequate to enable retrieval after a disk failure (other than if both drives were to happen to fail simultaneously)
You might be able to recover from another copy and you might not,
  • One type of failure is called "bit rot". Parts of a file can degrade over time due to the flakey (literally) quality of the disk media. So you make a copy and then the copy has bit rot and is not usable for recovery.
  • You make frequent copies of your critical files each writing over the previous backup copy. At some point bit rot creeps in and you are then making copies of the corrupt originals.
  • You have a word document the you frequently edit. It gets copied to your backup copy each time it changes. Later you discover that a previous version of that file contained critical information, The version containing that critical data was the file as it existed 6 month ago. You only have a backup of the file as it exists today, The backup won't help you there. If that file is your Lightroom Catalog file and you just wiped out the keyword accidentally on 2000 images how are you going to recover?
A system backup app will track version changes to files and keep copies of files that have been deleted. MacOS comes with a system backup app called TimeMachine. It runs constantly in the background backing up all of your critical user data so that you can recover from an inevitable disk crash, accidentally deleted file, bit rot, older versions etc. There are several 3rd party Backup apps that work in Windows. Acronis is a Multiplatform app that I use as an additional MacOS backup. Theres is a windows version that I would recommend. I use both Acronis and Time Machine both app runs in background as background apps detecting and changes to individual files and creating incremental backups. These run unattended 24X7 as long as my computer is running.

Besides Acronis there are other equally good System Backup apps for Windows besides Acronis. There are even some that backup to the cloud (BackBlaze) comes to mind but restoring from the cloud can be slow and tedious and expensive if the company managing you cloud backup proves a physical drive for resonation. Also Cloud backup solution have been known to suddenly with little warning cease business.
 

Birdbrain186

Member
Joined
Apr 4, 2018
Messages
36
Location
Birmingham UK
Lightroom Experience
Intermediate
Lightroom Version
6.x
Thanks both for the further detailed information.

Personally I don't want to (or feel I need to) start using any cloud storage, due to some of the possible shortcoming Cletus mentioned.

But I will investigate system backup apps further based on your recommendations. I suppose a downside is needing even more spare storage space for the backups taken by the software (I only just bought a new 6TB External HD as space on my PCs internal HDs was getting quite sparse)

Fortunately I do already regularly back up my LR catalog and have kept all LR backups from start on my current PC - 160GB worth, going back over 3 years - a fair portion of which I think could now safely be deleted so freeing up some more HD space!

In the meantime I will now start backing up even more frequently - but also start deleting older backups. I'm thinking keeping say the last 12 months will be plenty for my purposes.

Also as an additional precaution I've now moved the location of my LR backups from an internal HD on my PC (LR catalog being on separate C Drive) onto a separate external HD.
 
Joined
May 17, 2017
Messages
39
Lightroom Experience
Intermediate
Lightroom Version
Classic
Acronis has now added some cybersecurity, which a lot of its users aren't happy with, and it is now a subscription production. The Acronis users that don't like subscription plans have started to look elsewhere. Two other backup software options are Macrium Reflect and EaseUS. All three of these products will let you determine how long to keep backups and will delete the old ones, so you don't have to take the time to do the deletions. I have used Acronis since 2009 and was happy with it until the past two years. I'm now migrating to Macrium Reflect.

I have used Backblaze for 3-4 years and have been happy with it. It gives me peace of mind that my data is offsite in case there is a fire, flood or robbery at my house.
 

Birdbrain186

Member
Joined
Apr 4, 2018
Messages
36
Location
Birmingham UK
Lightroom Experience
Intermediate
Lightroom Version
6.x
Cheers for that Gary. No I don't fancy using want any subscription based product myself. But neither do I want a cloud based backup really either. Clearly I will have to give the matter some more thought!
 
Top