Import When a DNG is not a DNG

[VR in Oz]

Operating System:
Exact Lightroom Version (Help menu > System Info):
Lightroom Classic version: 7.1 [ 1148620 ]
License: Creative Cloud
Language setting: en
Operating system: Windows 8.1 - Home Premium Edition
Version: 6.3.9600
Application architecture: x64
System architecture: x64
Logical processor count: 8
Processor speed: 3.5 GHz
Built-in memory: 32660.6 MB
Real memory available to Lightroom: 32660.6 MB
Real memory used by Lightroom: 1897.4 MB (5.8%)
Virtual memory used by Lightroom: 1994.9 MB

I don't know whether my recent discovery is widely known (haven't seen any reports in the usual forums), so I thought I'd report here.

Having chosen to shoot using the DNG format offered by Pentax rather than proprietary Raw (PEF), I was delighted when Lightroom offered hashing as an early warning for file corruption. I used "Copy" for the Import option. On running "Validate DNG files", all of the files were reported as "not validated", as if they were not being recognised as DNGs.

So I tried importing with "Copy as DNG", thinking that the hash key would be created in the process. Lightroom reported "non-raw files were not converted to DNG. Not all of the files being imported were raw camera files. Only raw camera files were converted to DNG". And "validate DNG files" gave the same result as above.

Prompted by the excellent MissingFAQ, I used the DNG option for Metadata in Library to look at each of the files. I had set Lightroom Preferences to embed Fast Load Data as soon as this facility became available. To my horror, not one of the files in "Pentax DNG" format has embedded Fast Load Data - no wonder Library viewing has been slow!

All is not lost however, applying "Convert Photo to DNG" actually does what is says: the files were all reported as being converted, DNG view shows that Fast Load Data is embedded, and "Validate DNG files" reports that there are no invalid DNG files (whew!)

So the Pentax option of shooting in DNG format is quite misleading, wouldn't you say?

 

[Jim Wilde]

So the Pentax option of shooting in DNG format is quite misleading, wouldn't you say?

Possibly not. The validation hash may not be part of the DNG spec (it was certainly not in the original spec), as it was introduced in a later LR version. So Pentax may still be fully compliant with the spec.

And similarly with "Fast Load Data", that was also a later addition by Lightroom/ACR, so again may not be in the (original) DNG spec. By the way, the fast load data is not used in the Library module, so any slowness you observe in Library is unrelated to the presence or not of that data.

Might be worth asking Pentax what they think about it?

 

[Hal P Anderson]

"Validate DNG files" reports that there are no invalid DNG files (whew!)

Note that converting to DNG created (and stored in the catalogue) a hash value for the image data in your DNG files. Validate DNG files creates that hash value again and compares against the value in the catalogue. Validating will only show files that became corrupted after import. Which all means that you still have to "eyeball" all your imports to check that they aren't corrupted.

 

[johnbeardy]

If I remember correctly, updating the DNG metadata and preview will create the hash value for images captured as earlier-specification DNGs. So you get the validation benefit from that point on, not from the point of import.

 

[Roelof Moorlag]

If I remember correctly, updating the DNG metadata and preview will create the hash value for images captured as earlier-specification DNGs. So you get the validation benefit from that point on,

This does work on DNG's made with the Lightroom App on my iPhone. They come in natively without an hash but after 'updating the DNG metadata and preview' the validation works again.

I have tried "Convert Photo to DNG" also and that does the job too but it's a lot slower.

 

[PhilBurton]

Possibly not. The validation hash may not be part of the DNG spec (it was certainly not in the original spec), as it was introduced in a later LR version. So Pentax may still be fully compliant with the spec.

And similarly with "Fast Load Data", that was also a later addition by Lightroom/ACR, so again may not be in the (original) DNG spec. By the way, the fast load data is not used in the Library module, so any slowness you observe in Library is unrelated to the presence or not of that data.

Might be worth asking Pentax what they think about it?

Considering that Adobe originated the DNG standard, it is sort of incumbent on them to update the standard according to how they implement DNG support in their own products.

This is a totally unforced error by Adobe.

 

[johnbeardy]

They have updated the standard - Pentax have applied an earlier version of it.

 

[PhilBurton]

They have updated the standard - Pentax have applied an earlier version of it.

Then shame on me for blithely assumed that Pentax would have made sure to apply the current version of the software. Adobe is completely absolved here.

EDIT: I meant STANDARD, not software.

Phil