"Metadata" includes...

[mholt]

A bit new to this whole photo processing workflow where metadata is so important... and was wondering what "metadata" consists of. Yeah, I know it's "data about data" and includes things like when the picture was taken, by what camera, the specs, etc, but in Lightroom - does it also include the adjustments made (as in - I can move a few sliders on an image then move on, and this goes into a metadata sidecar file)?

I wonder where the adjustments in Lightroom are stored because I recently formatted my computer, backed up my raw photos (PEF format), and restored them into Lightroom, including the catalog, yet all the adjustments were lost. For some reason, I didn't have any XMP files...

I am considering going to DNG and been researching it a bit. I know that it depends on what types of backups you make... regardless, however: I wouldn't need XMP sidecar files... (which I now have, I think, as I've seen them in Explorer) and the adjustment data would be stored within the file.

I'm worried that the filesize is half the size of my 1' MB PEF files, though. When you convert to DNG, (i tried it on one photo), you have options to:

- Medium size JPG preview
- Preserve RAW image (instead of Convert to Linear - I assume we want to preserve)
- Compressed (lossless)
- Embed original RAW file (which I won't do, in order to save space - otherwise a DNG file is more like 14.5 MB)

I'm worried about the "Compressed" option. I know that reduces file size, but do I lose data, even though it is "lossless"?

Also, what about my old JPGs before I had a camera that shot in raw?

Thanks for any responses. My main concern is that if I need to restore all my photos from scratch, I want to save my adjustments and other photo information (like copyright/when/camera/lens, etc - the metadata that I know) so I don't have to redo it all again.

 

[johnbeardy]

The meaning of metadata in this context is changing - it can include both descriptive metadata and adjustment metadata. The term for the latter hasn't settled down yet.

Lightroom adjustments are stored in its database, an lrcat file - you need to back this up. You can also put most adjustments into xmp files if you want, though since the xmp doesn't include all the Lightroom data, it's best to regard xmp files as for interaction with other programs - eg Bridge - and not as backup.

As for DNG, you don't lose anything if Adobe says lossless. Elsewhere you sometimes see qualifiers like the word "virtually", but not with Adobe. The reduction in file size is because your PEF files aren't as efficiently compressed as DNG is (NEF compress a lot, Canon files less so). I'd put a full size preview in there, but your other choices are good. Some, though not many, discard the raw file entirely. Others keep them as an extra backup - this is what I do, and I've been happy with that choice.


John

 

[mholt]

John, thank you very much for the helpful reply. I will reconfigure my conversion options to include a full-size preview.

The way that I do backups - the way that is easiest for me - is to have Windows Vista do daily backups with its Backup utility to a secondary hard drive within my computer. I also do manual backups ("Just in case," I say) by dragging important folders from my primary drive to the secondary one, plus my external backup-backup hard drive.

So I think what I'll do regarding photos is this:

1. Backup catalog daily using Lightroom
2. Daily backups with Windows Vista's default utility to an internal secondary drive.
3. Manual (drag-drop) backups of important files including photos & catalog, periodically to internal secondary drive and external hard drive.


I am seriously considering converting all my photos to DNG (after long-term archiving my PEFs and old JPGs) next week. If I do this, all adjustments made in Lightroom (including descriptive metadata) will be embedded into the DNG and thus, if I need to restore them, I can restore the DNG and all the adjustments will be restored with it? So I won't lose my work?

 

[johnbeardy]

Avoid backing Lightroom's previews folder - it can grow big, unnecessarily swell the backup size, and an LR catalogue will rebuild it automatically anyway.

My suggestion is not to switch over immediately but to adopt a DNG workflow for a period - long enough to see if it works for you but not so long that it becomes painful if you decide not to proceed. When I switched (with CS2's introduction), I chose a month but decided after 2 weeks. I initially edit a shoot as NEFs, add initial metadata and adjustments, dump the rejects, save XMP files, switch to DNG, back them up, and back up and archive the NEFs.

The DNGs will contain all the adjustments. What they won't contain is virtual copies, stacks, history steps, collections. So the backup is the catalogue daily, and the DNGs once upon creation.

John

 

[mholt]

Avoid backing Lightroom's previews folder - it can grow big, unnecessarily swell the backup size, and an LR catalogue will rebuild it automatically anyway.

Gotcha.

My suggestion is not to switch over immediately but to adopt a DNG workflow for a period - long enough to see if it works for you but not so long that it becomes painful if you decide not to proceed.

Good idea, I may try this with a few unimportant photo shoots (just some random nature ones out in my backyard maybe).

The DNGs will contain all the adjustments. What they won't contain is virtual copies, stacks, history steps, collections. So the backup is the catalogue daily, and the DNGs once upon creation.

Since I don't use Photoshop, I'm assuming I don't need to worry about:

"virtual copies, stacks, history steps, collections"

I don't really even know what those are (except history steps).

So I guess I just need to back up the DNGs and the Lightroom catalog to keep all the work I've done to the photos in Lightroom. I'll proceed to do that (since, at this rate, I will be converting - but I'll do a trial).

Thank you!

 

[Brad Snyder]

Since I don't use Photoshop, I'm assuming I don't need to worry about:

"virtual copies, stacks, history steps, collections"

I don't really even know what those are (except history steps).

Matthew, these are quite handy productivity tools which exist in LR, not PS. You might want to skim the Wiki on these subjects, and then ask back here if you have any more questions. (Scroll thru the table of contents, or use the search box.)

LR Wiki

 

[mholt]

Interesting. I had never heard of these before. I assumed they were Photoshop-related things. I will look into these.

Edit 1: I know what collections are... silly me.

Edit 2: As I group some of my photos into stacks, and in the middle of a recent photo session, there is ONE picture - in the middle of a shoot - that was taken "the next day" from all the others, before and after it. All the others were May 24, yet this one was somehow May 25. I didn't modify the metadata - heck, I don't know how yet. Apparently I cannot stack pictures that are not in the same folder. (I organize by date taken.) This means that one photo in the middle of the photo shoot is in a different folder from the others.

Edit 3: Weird... I think I figured out how to change some of the relevant metadata here: Metadata->Edit Capture Time, right? In the "Original Time" textbox it says May 24 - the day it SHOULD be. But in the "Metadata" viewer on the right side in "Library", it says "Date Time Original" to be the 24. It says "Date Time Digitized" to be the 24th. But it says "Date Time" to be the 25th. And "Metadata Date" is the 27th. (But that's the date the metadata was last modified, I guess.) So why is it in the "May 25" folder? I did nothing with the photo on the 25th. It was Sunday - my day off. ^_^

 

[mholt]

In the mean time, I did a little research of my own of DNG files & Lightroom. I tried to simulate various scenarios of backup and restoration of photos and I think I understand it much better now. I also feel much more comfortable with Lightroom. I did these on a one-photo scale and just converted a RAW file of mine to DNG and put it through hell again and again and again. I have detailed steps recorded on my computer of what I did.

Basically, the option "Automatically write XMP data to file" is very magical and important and vital and - did I say important? - if one does not want to press Ctrl+S during the development cycle!

The most fundamental thing I learned is that adjustments made in Lightroom are stored as metadata. This, along with descriptive metadata, is handled almost completely separately from the actual photo file. It is saved seperately and used differently. While photo files are "Exported", metadata is "Saved" and "Updated" and "Read".

It's like a new paradigm for me.

I also did a little digging through an LRCAT file in the command prompt using the "edit" command. Windows Notepad would have surely locked up upon loading it. It's big ... ... but I did find some interesting things:

• Lightroom uses an SqlLite database to store its catalog of photos, adjustments, and metadata. I found many SQL queries that created tables, plus a bunch of scrambled binary data that was unreadable.
• I found - in a seemingly random place - a string of text, "Birds on a Line" - the name of one of my photos, yet one that I was pretty sure I didn't title in Lightroom (only in Fireworks for a project) - a search in Lightroom's "Find" box yielded ' results. Hm.
• One thing that I was curious about was whether Lightroom's catalog stored absolute or relative file paths to the photo's locations. It could be either, I suppose, but was worried because, what if:
  • You upgrade your OS? It could happen!
  • You change your Windows username at some point between backup and restoration? (Especially potent if you do a reformat, like I do annually.)
  Unless there is a feature in Lightroom I do not know about, the two above scenarios will likely break the backed-up catalog. I store my photos in the "Pictures" folder offered by Vista. I mean, that's why it's there, right? For me right now on Vista, the file path is:
  
  
  C:\Users\MyUsername\Pictures
  
  
  For me on Windows XP, the path was (I think):
  
  
  C:\Documents and Settings\MyUsername\My Documents\My Pictures
  
  
  See the problem? The paths to photos stored by Lightroom look like this. ANY change to the photo root directories and all the photos will go missing from the catalog and have to be re-imported with the risk of losing metadata if not stored in XMP or DNG files.
  
  This is not Lightroom's fault, however it would be very nice if one could specify a "root" directory for all - or at least most - of the photos so that if that changes, the catalog can be updated and will not break.

This is what I've concluded overall (correct me if wrong):

1. Lightroom photo and metadata adjustments/modifications are stored in the catalog - no matter your settings or file format.
2. Lightroom adjustment and descriptive metadata is stored in .XMP files, if using RAW formats and if you save it to disk manually by pressing Ctrl+S or checking the option "Automatically write XMP data to file".
3. Lightroom adjustment and descriptive metadata is updated in .DNG files if you save it manually with Ctrl+S or you checked the option "Automatically write XMP data to file".
4. If folder structure or OS preferences are likely to change, it is wisest to not rely on a catalog file alone to retain adjustments - it is best to save them to .XMP or embed them in .DNG files REGULARLY or AUTOMATICALLY and back those up as well.

Whew. That was actually kind of fun. Though basic to what you all know of Lightroom, I gained a lot from this thread and from a little fiddling.

So is this basically correct? :roll: It would be embarrassing if I was way off track.

 

[johnbeardy]

Basically yes, though I'd take issue with "If folder structure or OS preferences are likely to change, it is wisest to not rely on a catalog file alone to retain adjustments - it is best to save them to .XMP or embed them in .DNG files REGULARLY or AUTOMATICALLY and back those up as well."

It is wisest to move folders in LR or to learn how to fix folder structure mismatches in LR. There are both relative and absolute addresses, but humans can always do things to break them.

John

 

[Brad Snyder]

Interesting. I had never heard of these before. I assumed they were Photoshop-related things. I will look into these.

you'll be glad you did.

Edit 1: I know what collections are... silly me.

Edit 2: As I group some of my photos into stacks, and in the middle of a recent photo session, there is ONE picture - in the middle of a shoot - that was taken "the next day" from all the others, before and after it. All the others were May 24, yet this one was somehow May 25. I didn't modify the metadata - heck, I don't know how yet. Apparently I cannot stack pictures that are not in the same folder. (I organize by date taken.) This means that one photo in the middle of the photo shoot is in a different folder from the others.

[COLOR=#ff'''']Yes, stacks only work in individual folders, and the folder must be the lowest level. I believe there are changes in 2.'
[/COLOR]
Edit 3: Weird... I think I figured out how to change some of the relevant metadata here: Metadata->Edit Capture Time, right? In the "Original Time" textbox it says May 24 - the day it SHOULD be. But in the "Metadata" viewer on the right side in "Library", it says "Date Time Original" to be the 24. It says "Date Time Digitized" to be the 24th. But it says "Date Time" to be the 25th. And "Metadata Date" is the 27th. (But that's the date the metadata was last modified, I guess.) So why is it in the "May 25" folder? I did nothing with the photo on the 25th. It was Sunday - my day off. ^_^

[COLOR=#ff'''']There are known bugs in the capture time calculations.[/COLOR]

 

[Brad Snyder]

Basically, the option "Automatically write XMP data to file" is very magical and important and vital and - did I say important? - if one does not want to press Ctrl+S during the development cycle!

Yes, I leave this enabled, altho' some folks complain of a performance hit.

The most fundamental thing I learned is that adjustments made in Lightroom are stored as metadata. This, along with descriptive metadata, is handled almost completely separately from the actual photo file. It is saved seperately and used differently. While photo files are "Exported", metadata is "Saved" and "Updated" and "Read".

It's like a new paradigm for me.


[COLOR=#ff'''']Ahhh, another LR convert. This is where the magic lies![/COLOR]

I also did a little digging through an LRCAT file in the command prompt using the "edit" command. Windows Notepad would have surely locked up upon loading it. It's big ... ... but I did find some interesting things:

• Lightroom uses an SqlLite database to store its catalog of photos, adjustments, and metadata. I found many SQL queries that created tables, plus a bunch of scrambled binary data that was unreadable.
• I found - in a seemingly random place - a string of text, "Birds on a Line" - the name of one of my photos, yet one that I was pretty sure I didn't title in Lightroom (only in Fireworks for a project) - a search in Lightroom's "Find" box yielded ' results. Hm.
• One thing that I was curious about was whether Lightroom's catalog stored absolute or relative file paths to the photo's locations. It could be either, I suppose, but was worried because, what if:
  • You upgrade your OS? It could happen!
  • You change your Windows username at some point between backup and restoration? (Especially potent if you do a reformat, like I do annually.)
  Unless there is a feature in Lightroom I do not know about, the two above scenarios will likely break the backed-up catalog. I store my photos in the "Pictures" folder offered by Vista. I mean, that's why it's there, right? For me right now on Vista, the file path is:
  
  
  C:\Users\MyUsername\Pictures
  
  
  For me on Windows XP, the path was (I think):
  
  
  C:\Documents and Settings\MyUsername\My Documents\My Pictures
  
  
  See the problem? The paths to photos stored by Lightroom look like this. ANY change to the photo root directories and all the photos will go missing from the catalog and have to be re-imported with the risk of losing metadata if not stored in XMP or DNG files.
  
  [COLOR=#ff'''']Fortunately LR offers a function to "locate" the missing files. If you haven't changed the relationship between the files and folders, i.e. moved/relocated a folder tree as a whole, LR will straighten it out.[/COLOR]
  
  This is not Lightroom's fault, however it would be very nice if one could specify a "root" directory for all - or at least most - of the photos so that if that changes, the catalog can be updated and will not break.

This is what I've concluded overall (correct me if wrong):

1. Lightroom photo and metadata adjustments/modifications are stored in the catalog - no matter your settings or file format.
   
   [COLOR=#ff'''']true[/COLOR]
2. Lightroom adjustment and descriptive metadata is stored in .XMP files, if using RAW formats and if you save it to disk manually by pressing Ctrl+S or checking the option "Automatically write XMP data to file".
   
   [COLOR=#ff'''']true, optionally, you can save the metadata to RGB types (jpg, etc) as well[/COLOR]
   
   
3. Lightroom adjustment and descriptive metadata is updated in .DNG files if you save it manually with Ctrl+S or you checked the option "Automatically write XMP data to file".
   
   [COLOR=#ff'''']other file types as well, if option is enabled[/COLOR]
   
   
4. If folder structure or OS preferences are likely to change, it is wisest to not rely on a catalog file alone to retain adjustments - it is best to save them to .XMP or embed them in .DNG files REGULARLY or AUTOMATICALLY and back those up as well.
   
   [COLOR=#ff'''']This is subject to much debate amongst LR users. Personal preference plays a large role. [/COLOR]

Whew. That was actually kind of fun. Though basic to what you all know of Lightroom, I gained a lot from this thread and from a little fiddling.

So is this basically correct? :roll: It would be embarrassing if I was way off track.

...........