• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom Classic questions. All the information you need is in Adobe Lightroom Classic - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. And better still, the eBooks are updated for every release, so it's always up to date.
  • 17 August 2022 It's Lightroom update time again and there's a couple of minor improvements as well as new cameras, lenses and bug fixes! See this blog post for Lightroom Classic and this blog post for the Lightroom Cloud Ecosystem changes.

DNG Confusion from an Experienced User

Status
Not open for further replies.
Joined
Jul 11, 2011
Messages
321
Location
San Antonio, TX
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Lightroom Version
Latest Classic
Operating System
  1. Windows 11
I have used LR almost daily for 12 years and am accustomed to working with Fuji, Canon and Nikon raw files which all have sidecars that I write all edits and metadata to (besides just to the catalog). I have shot the Leica Q2 a lot lately and just shot it every day for 7 weeks in Rome and Sicily. Leica uses DNG for their raw files.

I have a folder on my base PC with the 2,400 edited DNG raw files shot on our recent Sicily trip (imported to LR from the laptop when I got home). As always, I had “Automatically write changes to XMP.” Since DNG files have no sidecar XMP, those changes are written directly to the DNG file itself.

There are advantages and disadvantages to that. I hate it because I sync to external drives as backup using GoodSync, and any edit in LR causes the whole DNG file to resync on a GoodSync job later on when I backup my big 6TB raw image folder to several other single 8TB drives. So instead of just a tiny sidecar file syncing, the whole DNG file has to overwrite the older version of the file on the GoodSync job. That is annoying and slow, but not my question – just background.

A week after returning from Italy, I decided to skim through the images again in LR on the big 32- inch 4k mini-LED IPS pro monitor. You see things on that fantastic monitor that you don’t see on the 4k 15-inch laptop on the road. So I did some more fine tuning edits. On the next GoodSync job, it said “No changes.” But there were changes. I edited 400 of those files in LR that day and all 400 should have been picked up by GoodSync when I ran a sync job. But it said no change. My changes were not being written to the DNG, despite having checked “automatically write changes to XMP.” So I went into the Photo menu to “Update DNG Previews and Metadata.” I selected all 2,400 DNG files and clicked on that, thus starting the longest LR project in history. What was happening? All the metadata changes were written in (but should have been already there). It was a lot slower than generating 1:1 previews at import. I have very fast gaming PC with 64 ram, M.2 SSD for Adobe and the system and 3080 GPU.

I create 1:1 previews on import, and that goes pretty fast. My CPU maxes out and my fans sound like a jet talking off. But when I use “Update DNG Previews and Metadata” the update moves at a snail’s pace and the CPU only goes to about 20% usage. What the heck is going on with that “update DNG Preview & Metadata”?

So here are my questions if anyone knows the answers:

- What is the difference between “Automatically write changes to XMP” and “Update DNG Previews and Metadata” when working with DNG files?

- Does “Update DNG Previews and Metadata” redo the 1:1 preview file stored in the .lrcat file that I generate at import or is it updating the embedded jpeg in the raw DNG file? When is best to do this? Is there a way to make it automatic or do you have to select the images and run the update?

- After an editing session, should I always highlight all the files and click on “Update DNG Previews and Metadata”? When is it best to do that?

- What is the difference in the Metadata that writes to the file in those two menu items? “Automatically write changes to XMP” writes all edits and metadata changes to the file. When does that writing to the DNG file happen? Instantly after edits or at some specified time on exit? Then, metadata is also written to the file if you execute “Update DNG Previews and Metadata.”

I read all the help files and I don’t think Adobe makes this clear. I might have missed something.
 
Joined
Jun 20, 2009
Messages
19,330
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
But I still say - The only thing I dislike about shooting a Leica Q2 is the fact that Leica uses DNG files (which means no sidecar files), and I do not like that.
No one needs sidecar file or XML stored in the DNG header block. Lightroom stores everything it needs in the catalog. The only reason there is a sidcar XML file is because you choose to store SOME of the metadata that Lightroom copies or generates with the original proprietary RAW file. Only Adobe products can understand and use the Edit metadata that Lightroom Generates. You can think of the Lightroom Classic Catalog file as one huge all inclusive XML file. As long as you have a backup catalog file and the original image file , you never need any XML found in the DNG or sidecar file.
 
Joined
Jun 20, 2009
Messages
19,330
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Cletus,
I've been dutyfully converting all my RAWs to DNG thinking an open source file format has a better chance of surviving to posterity than a proprietary format. Now you tell me that DNG is just a container. Also, there are DNGs and DNGs depending on what camera produces them and to top it all off, a DNG is not even necessarily RAW. Have I been deluding myself? It makes one wonder then what is the purpose of an open source so called "file format" anyway?
The DNG file format is an open source container. It can contain raw photo site data. TIFF and JPEG are also open source containers. They only contain by their definition RGB data. The purpose of the DNG open source file format is flexibility. TIFF, JPEG, NEF, CR2 are rigid structures and do not permit flexibility. As I pointed out DNG is based upon the TIFF/EP6 standard. NEF and CR2 are also based upon the same TIFF/EP6 standard. Their definitions require that they only contain RAW photo site data and there is no provision for an XML data block. Each of these companies have their own RAW processing software (usually contracted to the Japanese Image software company SILKYPIX). Any DNG RAW file can be processed by the RAW processing software from SILKYPIX.
A RAW DNG created by Leica or Ricoh or Pentax (now owned by Ricoh) produce the same DNG RAW files.
If you think Adobe might be around longer than Nikon or Canon you might be right, but while Asahi Pentax Corporation no longer produces digital cameras, its Pentax line was bought and is carried on by Ricoh. The PEF RAW format is still widely supported and of course so are the Pentax RAW DNGs.
The only Digital Camera that I know of and is no longer produced or supported by its manufacturer are this NX models from Samsung. The Proprietary RAW SRW files are still supported by Adobe and everyone else. And of course so are the RAW DNGs generated by those obsolete cameras. It is clear to me that someone will be around to support every proprietary RAW file format long after Adobe, Canon, Nikon and Ricoh are gone. If a company like Canon or Nikon developed some whiz bang feature to perhaps correct the focus of OOF images, that process will use the proprietary data stored in the parts of the Proprietary metadata. This new future whiz bang process won't be available to be used on older images that have been converted to RAW DNGs. Convert your proprietary RAW files if you want, but don't discard the original proprietary RAW file formatted files.
 
Joined
Jul 11, 2011
Messages
321
Location
San Antonio, TX
Lightroom Experience
Power User
Lightroom Version
Cloud Service
No one needs sidecar file or XML stored in the DNG header block. Lightroom stores everything it needs in the catalog. The only reason there is a sidcar XML file is because you choose to store SOME of the metadata that Lightroom copies or generates with the original proprietary RAW file. Only Adobe products can understand and use the Edit metadata that Lightroom Generates. You can think of the Lightroom Classic Catalog file as one huge all inclusive XML file. As long as you have a backup catalog file and the original image file , you never need any XML found in the DNG or sidecar file.
Disagree Clete. I need the sidecar files and I prefer them greatly. So don't say no one. There are many benefits to sidecar files, which is why LR has them.
I think Leica made a mistake using DNG, and I'm not alone. I don't like DNG because they don't have sidecar files.
 
Joined
Jul 11, 2011
Messages
321
Location
San Antonio, TX
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Cletus,
I've been dutyfully converting all my RAWs to DNG thinking an open source file format has a better chance of surviving to posterity than a proprietary format. Now you tell me that DNG is just a container. Also, there are DNGs and DNGs depending on what camera produces them and to top it all off, a DNG is not even necessarily RAW. Have I been deluding myself? It makes one wonder then what is the purpose of an open source so called "file format" anyway?
Bob, you do not need to covert to DNG for that or any other reason. This is a big topic of conversation on a lot of the camera forums. I have told people for years not to convert to DNG just because you are afraid that your Canon, Nikon, Sony or Fuji raw files won't be read in future generations of software or the company (and thus the raw format) dies.
I could go on about this because it has all been argued before and the list is long about why not to do it. Just trust me. You don't need to do it. You are gaining nothing by doing it. In fact, you are gaining disadvantage by doing it.
 
Joined
Jun 20, 2009
Messages
19,330
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
Disagree Clete. I need the sidecar files and I prefer them greatly. So don't say no one. There are many benefits to sidecar files, which is why LR has them.
I think Leica made a mistake using DNG, and I'm not alone. I don't like DNG because they don't have sidecar files.

IIRC, you said that you imported in a non conventional fashion from one computer to another which makes the XML data useful when you import those images to the second (master Catalog) It is only this unconventional way that you use Lightroom Classic that makes you think that you need sidecars. If you used the conventional method of merging Lightroom catalogs you would not nee the XML data because it would be imported directly from the source catalog to the master catalog file.

At last check I believe there were 12 camera manufacturers that selected using the open source DNG file format for their RAW image files SOOC. If more camera manufacturers adopted the Open Source DNG there would be less problem getting RAW data from new cameras since image processing apps would not need to reverse engineer the proprietary file formats. The people that think Leica made a mistake choosing DNG as their RAW file format simply do not understand RAW files and the flexibility of the DNG file format. What you do not like about the DNG format is its flexibility over an inflexible Proprietary RAW file format.


Sent from my iPad using Tapatalk
 
Joined
Jul 11, 2011
Messages
321
Location
San Antonio, TX
Lightroom Experience
Power User
Lightroom Version
Cloud Service
IIRC, you said that you imported in a non conventional fashion from one computer to another which makes the XML data useful when you import those images to the second (master Catalog) It is only this unconventional way that you use Lightroom Classic that makes you think that you need sidecars. If you used the conventional method of merging Lightroom catalogs you would not nee the XML data because it would be imported directly from the source catalog to the master catalog file.

At last check I believe there were 12 camera manufacturers that selected using the open source DNG file format for their RAW image files SOOC. If more camera manufacturers adopted the Open Source DNG there would be less problem getting RAW data from new cameras since image processing apps would not need to reverse engineer the proprietary file formats. The people that think Leica made a mistake choosing DNG as their RAW file format simply do not understand RAW files and the flexibility of the DNG file format. What you do not like about the DNG format is its flexibility over an inflexible Proprietary RAW file format.


Sent from my iPad using Tapatalk
I hear you Clete, and that is a legitimate point. In fact, I am this week going to do some test runs on importing from the laptop to my PC in the correct Adobe-intended way (and as Victoria and the Gurus teach) of exportin g as a catalog vs just copying the raw files with their sidecars into my PC's data disk and then importing them to the cat.

But there are some advantages to using the proprietary formats of raw with those camera systems they are intended for (especially with Fuji RAF files). Plus, there are advantages to having those sidecar files stored with the raw in case you or someone else needs them outside of the confines of LR. There is no harm in having an updated sidecar file to accompany your non-DNG raw file even if you don't use them often. Plus, with sidecar files, there is no writing to the raw file and it remains untouched on edits, which makes sequential backups and folder syncs much faster, smoother and easier after further edits in LR to DNG raw files.

But none of this is probably a big deal for most LR users. I can tell you from the camera forums and many discussions with pro photographers that I'm not the only photographer who feels this way. I've talked to a lot of pro photographers out there who don't like messing with DNG files and would certainly never convert to them from the raw formats they use. But then again, there are throngs of guys (mostly Leica shooters) who love DNG files and swear by that format.
But to be fair, the best Fuji photographer in the World (in my opinion) converts his RAF files to DNG.

It's a popular topic on the camera forums. I think it is good for LR users to be aware of the behavior of DNG and sidecar files in LR and the advantages and disadvantages of converting to DNG their work-flows (if they choose to do so).
 

PhilBurton

Lightroom enthusiast (but still learning)
Premium Classic Member
Joined
Nov 16, 2015
Messages
2,777
Location
California, USA
Lightroom Experience
Intermediate
Lightroom Version
Classic
A DNG created /converted from a proprietary raw image file remains as raw.

A proprietary raw (eg. NEF CR3) might contain a structure like this-
IMAGE DATA + METADATA D,A,F,G,B,E,C.
The DNG converter simply re-arranges the metadata to be more compatible with a wider range of software-
IMAGE DATA + METADATA A,B,C,D,E,F,G.
Notice the difference?

OTOH, yes there is other software that places info into a DNG 'wrapper' that does not necessarily contain sensor 'raw' Image Data.

And to get very technical there is the PDF-
https://helpx.adobe.com/content/dam/help/en/photoshop/pdf/dng_spec_1_6_0_0.pdf
Since I am a Nikon user with NEF files, I was curious to see if a DNG back to NEF convertor exists. I could find only this one, with a 2012 release, and the source blog no longer exists.

https://leicarumors.com/2012/05/23/new-version-of-dng-to-nef-file-converter-released.aspx/
 

PhilBurton

Lightroom enthusiast (but still learning)
Premium Classic Member
Joined
Nov 16, 2015
Messages
2,777
Location
California, USA
Lightroom Experience
Intermediate
Lightroom Version
Classic
At last check I believe there were 12 camera manufacturers that selected using the open source DNG file format for their RAW image files SOOC. If more camera manufacturers adopted the Open Source DNG there would be less problem getting RAW data from new cameras since image processing apps would not need to reverse engineer the proprietary file formats. The people that think Leica made a mistake choosing DNG as their RAW file format simply do not understand RAW files and the flexibility of the DNG file format. What you do not like about the DNG format is its flexibility over an inflexible Proprietary RAW file format.


Sent from my iPad using Tapatalk
@clee01l and @GregJ

I don't know any of those manufacturers except for Leica. However, the "big three" Nikon, Canon, Sony, have not adopted DNG. That is a shame, but is apparently a competitive issue in the minds of those manufacturers. (I happen to disagree very strongly, but none of them has yet approached me as a product planning consultant. :rofl:)

Around 2006, I became aware of something called OpenRAW, whose purpose was to convince manufacturers to adopt an "open" format. They wrote letters. The one they got back from Sony was a masterfully written non-answer, committing Sony to exactly nothing. Regrettable, but as a general rule, hardware companies don't understand software and software companies don't understand hardware.

For all the fear-mongering about proprietary formats, support for Nikon/Canon/Sony is not likely to disappear anytime soon from mainstream photo software. On the other hand, a truly niche manufacturer like Leica might get little or no support for their own proprietary format, and they avoid the expense of creating and maintaining their proprietary format. All I'm describing here is typical software industry dynamics, and is not specific to photo software.

Phil Burton
 
Joined
Jun 20, 2009
Messages
19,330
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
I don't know any of those manufacturers except for Leica.

Phil Burton
Apple, Pentax, Ricoh, Samsung, GoPro, DJI Drone cameras, DxO, Google (and most other manufacturers that make phone cameras), and Hasselblad are the ones that come to mind along with Leica. Ricoh owns Pentax and Samsung has ceased production of DSLR cameras. Even Sony has a DNG format camera on the Adobe List https://helpx.adobe.com/camera-raw/kb/camera-raw-plug-supported-cameras.html


Sent from my iPad using Tapatalk
 

hfogel-aol

New Member
Joined
Apr 8, 2019
Messages
2
The DNG file format is a container file that describes its own contents. It is quite flexible. Converting proprietary RAW files to a DNG on import creates a DNG file that has RAW data in one of the main data blocks. The Ricoh camera creates a RAW file and stores the RAW photo site Data in one of the main data Blocks. RAW data is photo site values stored in a regular order in a data block whose contents have been delineated in the DNG file Header. This is the same whether the file is a Nikon NEF, Canon CR2 or a Ricoh, Pentax, or Leica DNG. In camera processors create the RGB view as seen on the camera back and also stored in another data block as a JPEG thumbnail. This is the image that you first see when you open the RAW file in any image viewer. The RAW data is not in image form as the photo site values are just numbers measuring the intensity of light at the photo site and not RGB pixels .
What you have termed Post Production is I surmise a processed image . Raw files no matter their origin have to be demosaic'd and converted to RGB pixels before any photos application can read them . In Lightroom, this is done using Adobe Camera RAW. Once the raw Photosite values has been converted to RGB pixels it can be manipulated by an image processor like Lightroom. Since Lightroom is classed as a non destructive editor, none of the changes it makes are ever applied to the original RAW file . To preserve the Lightroom processed image, the Lightroom process "Export" is used to create a derivative image. Most often the file format used to store this derivative RGB image is either a JPEG or TIFF. Other file formats are options in the Export process. One of these is the DNG format. When selected, the processed RGB image data is copied to a DNG file type container and the main data block contains these RGB pixel values. You could just as easily create a TIFF file on export and the main data block would contain the same RGB pixels. The reason is that the DNG file format is based upon the TIFF/EP6 standard just as the TIFF file specification. TIFF, DNG and PSDs are public file formats created and managed by Adobe.
To summarize, RAW files preserve and store the original photo site values. They can be stored in a proprietary file format or an open public file format. DNG is the open Public file format used. Not all DNG files contain RAW photo site values. The DNG file format is a container file that describes its own contents. If it describes the main data block as containing raw Photo site values, the DNG is a RAW file. If it describes the main data block as containing RGB Pixels, then the file is nor considered RAW and is subject to the same contraints as a JPEG, TIFF pro other file format containing RGB pixel data.
Hi ,
Thanks for the note. I should explain, when I first worked with the DNG format, as it was entering the standards approval process, I offhandedly mentioned Peter Krogh that I was going to be reviewing a new Ricoh camera, that was it was rumored to use DNG as it's native RAW format, and assumed that it was the same as a converted DNG from a Camera Raw. He stopped me, and referred to those converted files as Post-Processed DNGs, which contained all the other features that format contains.

A native DNG, it turns out, is mostly an open source RAW file, and doesn't have the features of what we consider a post-processed DNG. And I've discovered that Adobe treats native DNGs, direct from the camera differently. For example, I've found that despite having the "Automatically Save to XMP" setting that Lightroom for example doesn't necessarily update the DNG's metadata, JPEG preview, etc. I recently sent Peter a DNG that had been adjusted in LR, and on his end, the DNG didn't show the edits. On my end, opened in Photoshop, the edits were all there. But this can be due to shared caches, etc.. so it's a bit misleading.

One option that I've been experimenting with is the option to select a group of DNGs and run the Update DNG feature, which updates all the post-processed DNGs. My question has to do with the exact make-up of a native camera produced DNG. And how Lightroom treats it. I remember a conference at the LOC, and there was a discussion about converting all files to DNGs, including native camera produced DNGs. I've run tests with their help converting JPEGs, TIFFs, etc.. to DNG, thus able to use the Validate DNGs function, as well as protect vulnerable files from destructive editing. Peter sent me a note about the following change to the DNG format.

"Okay, I just looked at the EXIF and it shows a "NewRawImageDigest" value. This is something that Adobe is putting in to the file to be more universally computable. There is probably no original raw image hash - only this new one. Here it is in the spec:

https://www.awaresystems.be/imaging/tiff/tifftags/newrawimagedigest.html "

This still doesn't answer my question of how a native DNG is handled by Adobe. And if there is truly a significant difference, which I'm sure there is, between an native vs. post version, how best to convert, and not make life really confused. You can convert to post versions on export. but that becomes a bit of a mess, not the least of which seems to be the need to reimport the files. So, you can see my conundrum!
 
Joined
Jun 20, 2009
Messages
19,330
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
This still doesn't answer my question of how a native DNG is handled by Adobe. And if there is truly a significant difference, which I'm sure there is, between an native vs. post version, how best to convert, and not make life really confused. You can convert to post versions on export. but that becomes a bit of a mess, not the least of which seems to be the need to reimport the files. So, you can see my conundrum!
. I’m not sure what you are calling a Post version. If you create an HDR in LrC, it is a DNG file that contains no RAW data block only processed RGB pixel data blocks.
If you take a CR2 RAW file and convert it on Import to DNG it will contain EXIF, ITPC Header data that is copied from the EXIF and ITPC blocks in the CR2 header. The CR2 file will contain one or more camera processed JPG data (RGB Pixels) in Thumbnail data blocks. Depending upon the Proprietary RAW file format one of those JPEG data blocks could even be a Full size JPEG. Adobe copies each of these header and data block for the proprietary RAW file into a DNG formatted file.

When a Camera company chooses to Use DNG for the Raw file format they still create EXIF and ITPC header blocks a RAW data block and one or more JPEG thumbnails of sizes specified in the DNG descriptor block.

If you begin with a RAW DNG and edit the RGB data, Lightroom does not automatically update and JPEG data blocks. Like any other RAW file it is not a destructive editor. It does not other wise touch the original file with one exception. That one exception is if you write XMP back to the file. Because the headers block of a DNG is extensible, there is a possibility to add an XML block to the DNG header. Something that you can not do with a proprietary RAW file format.

And there are processes that can update the JPEG thumbnails in the Original file. However, this goes against that policy of being a non-destructive editor.


Sent from my iPad using Tapatalk
 
Status
Not open for further replies.
Top