• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom Classic questions. All the information you need is in Adobe Lightroom Classic - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. And better still, the eBooks are updated for every release, so it's always up to date.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

more sidecar and catalog questions

Status
Not open for further replies.

Stumped

Active Member
Premium Classic Member
Joined
Aug 31, 2019
Messages
163
Location
Louisiana Gulf Coast
Lightroom Experience
Beginner
Lightroom Version
Classic
Lightroom Version Number
9.2.1
Operating System
  1. macOS 10.15 Catalina
So there is a lot out there, and here, on the above. Ive been reading, re-reading and re-re-reading to the point where im possibly more confused than I was before I started. It began from another thread when I mentioned .dng and .cr2 files, convert or not to - another ball of wax itself.... So, if I may, lay out what I understand to be correct for confirmation and some questions. I save all original files externally from local hard drive outside of LR. I currently shoot raw .cr2 files and had been importing them to LR as such and have the "Automatically write changes into XMP" checked in the catalog Metadata settings. QUESTION - Doing such with that setting checked gives me .xmp sidecar files, right?

I have also, after reading, started to import .cr2 files as .dng files, saving the original .cr2 externally and have the same "Automatically write changes into XMP" checked in the catalog Metadata settings. QUESTION - Because i now have that setting checked w/ a .dng file, LR writes to the XMP section of the file and not a xmp sidecar file?

Ive read that metadata refers to develop settings, tags, labels, crops, etc. that are primarily stored in the catalog file. QUESTION - This is the default whether the above setting is checked or not checked, right? Where can I see that / those files?

To make sure I fully understand this and possibly redundant - jpeg, png, tiff and psd and dng files - allow for writing to the xmp section of the file by having the above option checked and proprietary .cr2 and HEIC will get xmp sidecar files when checked?

Some examples of other proprietary raw files besides .cr2, HEIC ?

Confirm: Ive read that all changes to pictures are always recorded to the catalog - yes or no? If yes, when recorded to the catalog, does this imply to a xmp sidecar file or just the catalog?

If saving to a sidecar file, will that picture be able to be opened in PS as is, w/o further changes?

Confused - Under the Catalog Settings window, Metadata, "Automatically write changes into XMP", is this what writes the sidecar file or is this referring to the "into xmp" section of the file?

Are .xmp sidecar files hidden by default? Im on a Mac.

So now that ive probably asked the same question 18 different ways - all this reading as mentioned has led to confusion for me. Additionally some places are written by those much more experienced and I think sometimes some things are taken for granted and maybe not explained very well and im just trying to make sure I fully understand this before getting too deep into it.

Additionally, from the other thread I briefly made reference to dng vs cr2 --- this led to convert or not to - a whole other argument - file check vs no file check - in hind site, at this point, probably a non-issue for me. Still trying to learn that just because i read it doesn't mean I have to try it. Maybe just concentrate on basic workflow.

Finally, this is what id like to accomplish, and my understand, please correct if not - as mentioned, on a Mac and I use time capsule and store originals on external drive. Its my understand(please confirm), that if I write to the xmp portion of a file, those changes are picked up as file changes by time capsule and I get a new back up copy with every change made, right? This seems like a waste of space compared to if I were just writing to a sidecar file and that was the file being backed up and not the whole image file.

What settings should I set to use sidecar files and not back up the picture with every edit made?

Finally, If worried about space, and if I were to keep the settings as is and seeing as I save originals external, Im safe to exclude this LR (picture only directory) from Apple backups and still be safe? I am backing up my catalog in LR AND time capsule.

Thanks all for any help here in getting me straight!

Attached is what I have now.
 

Attachments

  • Screen Shot 2020-05-21 at 2.42.50 PM.png
    Screen Shot 2020-05-21 at 2.42.50 PM.png
    282.4 KB · Views: 298
I currently shoot raw .cr2 files and had been importing them to LR as such and have the "Automatically write changes into XMP" checked in the catalog Metadata settings. QUESTION - Doing such with that setting checked gives me .xmp sidecar files, right?
Yes

QUESTION - Because i now have that setting checked w/ a .dng file, LR writes to the XMP section of the file and not a xmp sidecar file?
Yes

Ive read that metadata refers to develop settings, tags, labels, crops, etc. that are primarily stored in the catalog file. QUESTION - This is the default whether the above setting is checked or not checked, right? Where can I see that / those files?
This data is saved into the Lightroom database (SQLite)
 
Last edited:
Confused - Under the Catalog Settings window, Metadata, "Automatically write changes into XMP", is this what writes the sidecar file or is this referring to the "into xmp" section of the file?
Both! It writes XMP files for photo's wich i can not write into the XMP section of the file itself

Additionally, from the other thread I briefly made reference to dng vs cr2 --- this led to convert or not to - a whole other argument - file check vs no file check - in hind site, at this point, probably a non-issue for me. Still trying to learn that just because i read it doesn't mean I have to try it. Maybe just concentrate on basic workflow.
Preserving the integrity of your files is probalby 'basic workflow' i think? In that case, the file validation option of DNG can be very valuable.

What settings should I set to use sidecar files and not back up the picture with every edit made?
If you use sidecar files they are the only files that are changed. Only changed files are in the backup.
When you have backups of your files and from you catalog (.cat file) then there is no need for XMP however.

inally, If worried about space,
Why? Storage is very cheap these days
 
Last edited:
When you have backups of your files and from you catalog (.cat file) then there is no need for XMP however.
I agree with all your comments, including this one - there is no need. However, as an additional step (in event of catalog failure and inability to repair) or for reading by some other programs outside of Lr, they can have their place.
 
Roelof, totally agree with you! And great point-by-point breakdown of answers too!
 
Both! It writes XMP files for photo's wich i can not write into the XMP section of the file itself


Preserving the integrity of your files is probalby 'basic workflow' i think? In that case, the file validation option of DNG can be very valuable.


If you use sidecar files they are the only files that are changed. Only changed files are in the backup.
When you have backups of your files and from you catalog (.cat file) then there is no need for XMP however.

To clarify - so if I have the box check w/ .dng file, it writes to the XMP section of the file and if I have it checked w/ .cr2 files I get a .xmp sidecar file?

yes

when you refer to "XMP", do you mean XMP Metadata section of file or .xmp sidecar? Nomenclature ? Because I have .cr2 files mixed with .dng files in the same catalog I do not want to be backing up each .dng when I make a small change. Thats why I wanted to know if I could unselect that directory from the time machine back up.
 
Last edited:
I agree with all your comments, including this one - there is no need. However, as an additional step (in event of catalog failure and inability to repair) or for reading by some other programs outside of Lr, they can have their place.
Please consider two things. Catalog failure is best dealt with by making regular catalog backups, not by writing changes to XMP. Too many catalog stuff is not written to XMP. And writing changes to XMP will trigger your image backup application if you use DNG. The entire DNG file will be copied again.
 
Sorry for the confusion - I didnt mean to say, if I did, that anything was not necessary. I'm just trying to figure out the above and what happens when, so that I have a full understanding of all the settings. As far as I can see now, as a new user, seems redundant if I have an original unmolested file in offsite storage to keep making backups when I might just be making simple edit (by writing into the XMP section of a file). So I'm seeing what I need to have set to only have the sidecar file, that the changes be made there and that would be the file that would be included in the backup by Time Machine and how to accomplish that and any additional provisions, if necessary(another question), that would need to be done to have the file to be be able to be used by PS or another edit program while using sidecar file.....
 
As far as I can see now, as a new user, seems redundant if I have an original unmolested file in offsite storage to keep making backups when I might just be making simple edit (by writing into the XMP section of a file)
1. Writing to XMP is not necasary for a dam backup
2. If you decide to write into XMP then only the XMP sidecarfiles are changed so in the backup. When the file has a XMP section, there wil not be written a sidecar file. So, in these situations the whole file wil be in the backup
3. There is no option in Lightroom to force XMP sidecar files on files with a XMP section (as DNG)
 
1. Writing to XMP is not necasary for a dam backup
Ok, you are a "Power User" with thousands of posts and I am new trying to understand this - there's no need to curse. I never said writing to XMP is necessary for a backup.
2. If you decide to write into XMP then only the XMP sidecarfiles are changed so in the backup. When the file has a XMP section, there wil not be written a sidecar file. So, in these situations the whole file wil be in the backup
Tks
3. There is no option in Lightroom to force XMP sidecar files on files with a XMP section (as DNG)
Tks
 
If saving to a sidecar file, will that picture be able to be opened in PS as is - showing edits, w/o further changes?

Or do the changes have to written to the file and then opened in PS?

What settings should I set to use sidecar files and not back up the picture with every edit made?
 
If saving to a sidecar file, will that picture be able to be opened in PS as is - showing edits, w/o further changes?
Saving to XMP is irrelevant if you open Ps directly from Lr - see page 336 of your Classic book for the options.
 
Saving to XMP is irrelevant if you open Ps directly from Lr - see page 336 of your Classic book for the options.
And while reading that page, you could skip forward a few pages to page 345 to the section called "Saving Metadata to the Files"....there you'll find the answers to most of the questions you've been asking here. If there's anything that you don't understand then tell us what that is.

What settings should I set to use sidecar files and not back up the picture with every edit made?
It's not a question of settings, it's a question of which file-type you are using....sidecar files will only be created when you save XMP to proprietary raw files (CR2/3, NEF, ARW, etc.). For ALL other file-types in your catalog saving XMP will update the actual files, and so would cause your backup software to back up those updated files. All this is covered in your book as specified above.
 
Sorry! English is not my native language and i meant to write 'sound'...
Got it - tks for the help - sometimes that spell check can get you - dam is close to sound......

Saving to XMP is irrelevant if you open Ps directly from Lr - see page 336 of your Classic book for the options.
ok. tks.

And while reading that page, you could skip forward a few pages to page 345 to the section called "Saving Metadata to the Files"....there you'll find the answers to most of the questions you've been asking here. If there's anything that you don't understand then tell us what that is.
So that's why I put all this up--- my very 1st sentence of the post says it all, "So there is a lot out there, and here, on the above. Ive been reading, re-reading and re-re-reading to the point where im possibly more confused than I was before I started." Ive read a lot in Victoria's book and Martin Short's, specifically pages 626-634 and more.... I was simply confused and trying to understand the details. For instance, Should I write to XMP? section on 346... Is very helpful and everyone's situation is different, but its not clear at all which are Pros and which are Cons... Also, on that page at the end, another confusion to me and another question I asked but no answer I think - Which of Lightroom's data isn't stored in XMP - nomenclature - when its said, "stored in XMP", this will always refer to the XMP section of the file like the next section How do I write to XMP, also referencing the XMP section of the file? Compared to when "xmp sidecar" is used will always refer to the sidecar file?

So, to confirm...... Catalog Settings, Metadata, ☑️ Include Develop metadata settings inside JPEG, TIFF, and PSD files when checked will always write to file XMP and when not checked to XMP sidecar. Same with the next setting, Automatically write changes into XMP - when checked the XMP section of the file will get updated. If unchecked, one gets an XMP sidecar file?
 
So, to confirm...... Catalog Settings, Metadata, ☑ Include Develop metadata settings inside JPEG, TIFF, and PSD files when checked will always write to file XMP and when not checked to XMP sidecar. Same with the next setting, Automatically write changes into XMP - when checked the XMP section of the file will get updated. If unchecked, one gets an XMP sidecar file?
No, you still don't seem to get this. The only option it to write to XMP or not to write to XMP (the default is NOT to write to XMP as all that data, plus more, is always stored in the catalog database). If, however, you do choose to write to XMP then how that is done is determined (as I said earlier) by the type of files that you are dealing with.....for proprietary Raw files, the XMP data is stored in XMP sidecar files, but for all other file-types the XMP data is stored in the XMP section of the file header. That is automatically determined by Lightroom, the user cannot control how the XMP data is stored in the files.
The other option (Include Develop Settings in metadata....) only determines if the develop settings are additionally included with all the other metadata when writing to XMP.....if you don't write to XMP, having this option checked is irrelevant,
 
For instance, Should I write to XMP? section on 346... Is very helpful and everyone's situation is different, but its not clear at all which are Pros and which are Cons...
They are listed as Pros and Cons - the first three can be considered potential pros, the five that follow (that start 'on the downside) are cons.

Which of Lightroom's data isn't stored in XMP - nomenclature - when its said, "stored in XMP", this will always refer to the XMP section of the file like the next section How do I write to XMP, also referencing the XMP section of the file? Compared to when "xmp sidecar" is used will always refer to the sidecar file?
Page 346, the paragraph immediately before 'Should I write to XMP?' makes it clear from most file types this will be in the file header, but for proprietary files it is written to a separate sidecar file.

So, to confirm...... Catalog Settings, Metadata, ☑ Include Develop metadata settings inside JPEG, TIFF, and PSD files when checked will always write to file XMP and when not checked to XMP sidecar. Same with the next setting, Automatically write changes into XMP - when checked the XMP section of the file will get updated. If unchecked, one gets an XMP sidecar file?
No, nothing is written to any type of file unless you tell it to - by Ctrl/Cmd+S or ticking the Automatically write changes into XMP
 
No, you still don't seem to get this. The only option it to write to XMP or not to write to XMP (the default is NOT to write to XMP as all that data, plus more, is always stored in the catalog database). If, however, you do choose to write to XMP then how that is done is determined (as I said earlier) by the type of files that you are dealing with.....for proprietary Raw files, the XMP data is stored in XMP sidecar files, but for all other file-types the XMP data is stored in the XMP section of the file header. That is automatically determined by Lightroom, the user cannot control how the XMP data is stored in the files.
The other option (Include Develop Settings in metadata....) only determines if the develop settings are additionally included with all the other metadata when writing to XMP.....if you don't write to XMP, having this option checked is irrelevant,
Thanks Jim - I think im getting it. Sorry to be so dense here, but it may actually be a medicine thing too..... So.... To beat a dead horse more, write to XMP refers more to a format/platform(as stated in the book) than location (unless a sidecar file)and format? As I quoted earlier, all info is written to the catalog and you just confirmed - tks.

Which are prioprietary formats and which are not?
 
Yes, they are proprietary (i.e. raw files which are specific to each different camera manufacturer). Non-proprietary files are "open-standard" formats such as Jpeg, Tiff, PSD, DNG.
 
Im sorry - those are proprietary or are not?
ANY RAW file that is not in the open DNG format is considered proprietary to the camera manufacturer. Writing into an XMP block in those RAW file leaders would destroy the format integrity of the RAW file format and therefore Lightroom will not write to those source files. DNGs, JPEGs, TIFFs etc are documented public formats and have rules to writing into the XMP section for the file header blocks.
 
Last edited:
Status
Not open for further replies.
Back
Top