DNG - why one should keep your raw anyway

Status
Not open for further replies.

Linwood Ferguson

Linwood Ferguson
Lightroom Guru
Joined
Jan 18, 2009
Messages
2,587
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
I noticed this little tidbit in the latest release:
  • Fixed issue where some images converted to DNG from the Hasselblad H6 contained a color cast.
Often I see people converting RAW to DNG, and then getting rid of their raw, thinking DNG has everything in it that the raw did.

This is just a reminder that a DNG is a PROCESSED image. It may not be AS processed as one converted to JPG or TIFF, but it is still processed to get it in the standard format Adobe promotes.

And if the conversion process ever has an issue, and you get rid of your RAW, it may or may not be possible to fix the issue, especially if it was subtle and you did not notice for a while.

Embedded RAW in DNG is a possibility of course, but again you depend on the DNG conversion always working properly.

This is just a reminder that it is a piece of software, like any other, that occasionally does not work properly.

If your images are important, keep your raw files!

FWIW.

Linwood
 
I noticed this little tidbit in the latest release:
  • Fixed issue where some images converted to DNG from the Hasselblad H6 contained a color cast.
Often I see people converting RAW to DNG, and then getting rid of their raw, thinking DNG has everything in it that the raw did.

This is just a reminder that a DNG is a PROCESSED image. It may not be AS processed as one converted to JPG or TIFF, but it is still processed to get it in the standard format Adobe promotes.

And if the conversion process ever has an issue, and you get rid of your RAW, it may or may not be possible to fix the issue, especially if it was subtle and you did not notice for a while.

Embedded RAW in DNG is a possibility of course, but again you depend on the DNG conversion always working properly.

This is just a reminder that it is a piece of software, like any other, that occasionally does not work properly.

If your images are important, keep your raw files!

FWIW.

Linwood
Or you simply don't convert to DNG, despite the urgings from Adobe.

Full disclosure: My RAW files are NEF, and I'm fine with that. I do incremental backups, and keep backups for years. I back up to a 4 TB internal drive in my desktop (which has the room for this drive). Every year, I put in a new 4 TB drive. The old one goes into an anti-static wrapper and into a drawer.)

Phil

Phil
 
Or you simply don't convert to DNG, despite the urgings from Adobe.

Well sure, that's my solution as well, but a lot of people around here are swayed by the DNG arguments and this was just a reminder that keeping the NEF's anyway is a good idea if you use DNG.
 
I think only a fool discards the raw files. Consider them as another backup, in another format, and space is cheap.

Another big reason is that there is not just one type of DNG - there are lossy DNGs too. If you convert to DNG and subsequently discover that you accidentally chose a lossy option, you will need the raw files to recreate real DNGs.

Currently a related issue is that lossy DNG is also being used as a workaround for some dissatisfaction with Adobe's handling of Fuji XTrans files. A 3rd party app, XTransformer, demosaics these files and returns lossy DNGs to Adobe - some prefer the rendering, and processing speed is noticeably faster. Again, when Adobe next improve their Fuji handling, you would need access to the raw files.

There's little point discarding them.
 
I've confirmed with Adobe that the DNG files were fine - it was just rendering incorrectly in Lightroom.
 
I think only a fool discards the raw files.
I know I'm a fool for LOTS of reasons; but ya didn't have to tell another reason :rofl:

We all have different reasons for taking and keeping photos -- for some unedited jpg off the card or mobile phone pics is all they need while at the other end are those who want to count, preserve and edit every last pixel from the best gear their money can buy.

But I'm getting to a stage when thinking ahead is not so far or as important as it used to be

For me; dng means I can see the image in other programs; but I'm now mostly using the Lumix jpg files :eek2:and only use the raw/dng file I need to use because the jpg is not good enough --- haven't really needed a raw file for awhile (LR 5 can not read lumix fz300 raw files so I need to convert them to dng)

Personally; I feel more people should give more thought to their actually photography than worrying about gear and file types used. Took me far too long to wake up to that small fact :(
 
I've confirmed with Adobe that the DNG files were fine - it was just rendering incorrectly in Lightroom.

That would go along with what I recently experienced in my C1 testings.. For my Canon G7X files converted to DNG, they were not displaying any histograms or allowing proper use of its color editor - yet the native raws did. C1 eventually got the issue fixed. The DNG's were fine - the processor wasn't.
-------------

Outside of that - like Ian says, it depends on individual needs/desires. If one were any kind of pro, one would likely be able to afford more storage space on a whim. For those who are not pros and maybe can't afford more storage space on a whim - then DNG allows them to save some serious space.. Not only in the conversion to DNG itself (generally), but also ridding of the locked-in ~3-10+ MB full-size useless jpg previews that are contained in most native raws. That can add up quickly. (For those wondering, it's ~3MB for Canon S100 raws (12-bit,12mp) -- and ~10+MB for Pentax K-3 raws (14bit,24mp).)

If one is not a pro and can't afford more space on a whim.. And one ties that much importance to a picture.. Then I think a re-examining of one's life priorities could be in order. :coffee:

One is likely to very quickly notice if major issues are happening with DNG conversions. And in the worst possible case scenario for this current bug, even if something was indeed wrong with the DNG itself, then all you would have to do is simply change/correct the color cast.. Even if you had to take it to PS to do so.

Although, unlike Ian, I haven't quite been able to let myself take it to the jpg level. For me, at most, I might take it to the lossy DNG level for less than ~.5% of my images. The only jpgs I've made (and kept just the reduced resolution high-compression jpgs (from processed raws)) are of this apartment building's office signs.. Like office hours and whatnot.. No need for anything fancy there. However I also no longer take family snapshots anymore.

Again, if one is pro, it might be a different story about keeping native raws. But I'm not. And it's safe to say that if one can afford a Hasselblad, then they are not your average enthusiast. Sometimes I think people try too hard to convince themselves of reasons not to convert to DNG. :p :whistling:
 
Last edited:
Personally; I feel more people should give more thought to their actually photography than worrying about gear and file types used. Took me far too long to wake up to that small fact :(

A friend who always asks me for advice puts it differently - doing things properly means more time enjoying my pictures.

Sorry if the "fool" comment is a bit strong. It costs little to keep these files, and another backup never hurts.
 
To my way of thinking, the most important reason for using DNG is futureproofing. Whether or not a scintilla of information is lost in the process is purely academic to all but the most dedicated pixel peeper. Just like the discussions in bygone years about how to store our negatives - vertically, horizontally, in paper, in polyester etc etc - when in fact something unexpected (digital) came along and practically condemned the lot to trash, no matter how perfectly we had archived them. Oh, I know we can scan them, but how many of us have scanned more than a fraction of our film?
 
Last edited:
If one were any kind of pro, one would likely be able to afford more storage space on a whim. For those who are not pros and maybe can't afford more storage space on a whim - then DNG allows them to save some serious space.. Not only in the conversion to DNG itself (generally), but also ridding of the locked-in ~3-10+ MB full-size useless jpg previews that are contained in most native raws. That can add up quickly. (For those wondering, it's ~3MB for Canon S100 raws (12-bit,12mp) -- and ~10+MB for Pentax K-3 raws (14bit,24mp).)

I guess it depends on your perspective for how much is "Serious space" and how much is a "whim" worth.

I converted a raw to DNG with no fast-load data and no preview. The file went from 40megs to 30 megs, or 25%.

With a full size preview, it went from 40 meg to 34 meg, or 15%.

A HGST (probably best disk drive maker at present) Desktar 3TB drive costs $70. A 2TB version is $50. Let's hand wave and say the 3TB is what you need for the original files and 2TB for your smaller files. That's $20.

With my 36mpx D800 images the 3TB will hold 75,000 images. More modest cameras much more, but that comes out to 3750 images per dollar saved.

To be fair, you should have at least 2-3 backups, so let's divide by almost 4 and call it 1000 images per dollar of disk space you pay extra to keep the raw instead of using DNG.

Try taking all your camera gear and adding it up, and divide by the number of images you kept from it. Bet the number is orders of magnitude more. ;)

One is likely to very quickly notice if major issues are happening with DNG conversions. And in the worst possible case scenario for this current bug, even if something was indeed wrong with the DNG itself, then all you would have to do is simply change/correct the color cast.. Even if you had to take it to PS to do so.

Since apparently Adobe has not (yet) introduced a known bug in the conversion itself, we are both speculating, but I see no reason whatsoever to assume a bug introduced will be "major" enough you will notice immediately. What if it did the math wrong and you lost 2 stops of dynamic range in the shadows. Would you notice? Or just think the shadows were pretty deep that day? A year later some scientifically inclined photographer does some experimentation, finds the issue, Adobe mulls it over for a while and releases a fix.

Maybe the answer is you do not care -- that if it did not jump out at you as a problem at initial post processing, it's not worth worrying about.

Maybe this sort of scenario will never happen.

We could go back over Adobe's history of introducing bugs, from serious to minor -- but it's software. Software ALWAYS has bugs. Anyone who has done it seriously for a long time knows many (maybe most, certainly the most dangerous) data corrupting bugs are those which are subtle and not noticed quickly.

I do see the rationale of "it it wasn't bad enough to notice, it is not bad enough to care about". I do not agree with it, since if I later found out I would be really upset, but you may be a more calm person. :)

Sometimes I think people try too hard to convince themselves of reasons not to convert to DNG.

First of all I started this thread not to convince people not to convert, but to save your raws even if you do. But inevitably this subject comes up and I have one response:

I do not need to convince myself NOT to convert, because no one has convinced me of a reason TO convert.

To my way of thinking, the most important reason for using DNG is futureproofing.

And I'm sorry, but that is demonstrably bunk. It implies that at some time in the future you will lose the ability to process your raw files (but not DNG). Now that may happen.

It also implies all software will support all current DNG formats for now and forever, which in terms of technology there's very little historical precedent. Heck, my version 1 XQD cards won't work in my Version 2 reader even, from the same company, and that happened in about 3 years. And just where do I buy a Betamax VCR?

But most importantly it also implies it will happen unexpectedly and with no way to deal with it at the time, and that just plain will not happen. If tomorrow Adobe announced they were dropping support for D70 conversions (my oldest raw), I can select them all, and do a mass DNG conversion over night. Done. No loss of edits, no issues, no real work (on my part, and who cares if my computer works hard one night).

And when it does start happening, the internet will ring with people lamenting the end of the world with dropping support -- unless you live in a cave with no news or internet, you will get a head's up. And if you do live in such a cave, you won't be installing the updates anyway, so your software will still work.

Please do not take this the wrong way -- I am not saying people are making a mistake to use DNG. It's a valid decision, and there are some good reasons. I just happen to think that "future proof" is Adobe propaganda and not one of those reasons.

But this is all about preservation of your work. It's a lot like backups. I back up; some people do not. To me it is illustrative of them not caring enough about their images to bother (I find it hard to believe anyone could actually say "I never knew I should back up" in today's age). That's their choice, their value judgement.

Whether you decide to keep your raw images as a hedge against future problems is similarly your choice, and a value judgement. I just thought it was one worthy of a bit of exploration.
 
A friend who always asks me for advice puts it differently - doing things properly means more time enjoying my pictures.

Sorry if the "fool" comment is a bit strong. It costs little to keep these files, and another backup never hurts.
And every time some piece of software touches that RAW file, that is another opportunity for bugs and or incomplete features. At least with a non-destructive editor like Lightroom, these issues are reversible, if only over time. A DNG conversion, without the RAW backup, is irreversible.

Phil
 
And every time some piece of software touches that RAW file, that is another opportunity for bugs and or incomplete features. At least with a non-destructive editor like Lightroom, these issues are reversible, if only over time.

Ironically, that's the utmost reason I started converting to DNG.. That image data hash. Sure, there are other ways to verify the integrity of images.. Like par sets and whatnot. But to me they're all a PITA. In LR, it's as simple as choosing "Library->Validate DNG Files".

The DNG tiling, I might call 2nd on my list.. To make use of multiple cores when decoding DNG data. This factor isn't mentioned often enough, in my opinion.

Size might be 3rd. But coming from someone that way overextended themselves to feed their addiction... I mean.. Hobby. :cool: ..And is about destitute, in the middle of a [so far] 8 year disability case (with no income).. ANY amount for new drives is too much right now. Even then, size is still lower on my personal list, as I'm not able to get out taking pictures as much as I'd like to - so my library is quite small. And fortunately, I bought a 4TB drive when I still had some money - to go along with a 2TB that I already had. So for me, the reduced size mainly means I get to use fewer BD-R/BD-RE50's for full backups. And when the [infrequent] chance comes that I make many worthwhile pictures, I'll break out the 2 drives for backup as well.

Very low on my personal list might be future-proofing.. For many of the reasons Linwood mentions. But it is a side benefit. Hopefully DNG will finally get ISO ratified one of these years/decades and become like JPG and Adobe's TIFF format.

-------

Yes, it did take me a while to get over that 'never touch or throw away the originals no matter what' feeling that only comes natural. However when I went to go edit some legacy jpgs one day, I got some 'corrupt file' warnings. Fortunately I was able to recover them from an ancient backup on CD -- the corruption had propagated through all my other normal post-LR backups. It was at THAT point I started getting over that feeling, and converted even all my legacy jpgs to DNG. My only gripe is that layered TIFF's can't retain their layers in DNG - but I'm not good with Photoshop anyways, so there's few of those. ... It was also because of the validation, that I slowly got over the 'never write anything to the original' and started 'writing metadata to files' - to add yet another backup layer. And before I do a full backup, I now always do the validation.

As far as worrying about the original image data not converting properly... It really isn't hard to copy over the data from the Adobe TIFF formats that most native raws are made of anyways. And my Pentax cams write DNG natively to begin with. .... But I guess it IS Adobe we're dealing with here. :laugh: Despite that fact.. If it don't act broke - then don't worry about it. I don't take my images that seriously to worry about such things.


(And I know John is firmly in that 'pro' category - so I can understand his stance. :) )
 
Last edited:
The validation issue is one of the good ones for DNG, though I still maintain they gave it short shrift by including ONLY the image data, and none of the other structures.

To me it's a much bigger issue, since you can't similarly validate TIFF's, and any editing outside of Lightroom is generally going to produce TIFF's (or PSD/PSB, same issue) as the file you want to persist. Adobe has really been remiss in not building in an automated validation for ALL managed files -- image files of all types (that Lightroom supports) as well as sidecars. It's really not hard at all, they have almost all the structures needed in the catalog already (well, for images, not sidecars, but that's trivial to add).

But with DNG's at least you get to validate some of them.
 
I too wish they did 2 separate hashes: image, and metadata. And for the life of me, I can't understand why they have yet to do hashes for all files. They're already all referenced in a database - so they shouldn't be moving around behind LR's back if one is doing things properly. I don't see how it would be so hard, except for layered tiffs that might change contents outside of LR. .... But such is the norm for Adobe. :banghead:

I guess the theory is that Image data is most important - metadata can be rebuilt and/or isn't as vital. However if you write to files, and store those in verified rotating backups, along with the many catalog backups one should be making (at every exit) - then you should be quite well covered for such a metadata scenario. And people should be keeping some OLD backups, for when they come across either metadata corruption or Stupid User Mistakes.
Here, for my catalog backups - I store them in a directory that is synced to a free Google Drive account.. And I'm also about to set up a sync to an old and slower Transcend SD card. Both Gdrive and the SD card are 15/16 GB- so it'll work out quite well. On occasion, I'll also manually put a catalog backup or two onto some other old SD cards (as in the 1GB variety) - as well as a few ancient IDE hard drives with USB adapters. So I'm hopefully well covered for such situations.
 
please remember my thoughts are about happy snappers; pro work is a different ball game although I do find it hard to understand some of MBs "needed" today . And to verify; I still use raw capture also, although I don't seem to need them or see much difference between a good jpg file and raw file for my uses. It's also so easy now to blend a couple of bracketed files together to match exposures if need be.

Sorry if the "fool" comment is a bit strong. It costs little to keep these files, and another backup never hurts.
No offence taken John; made me laugh actually :) . "costs" always has different meanings to all.

I could point to all my old personal and work negatives gather dust -- am I really likely to use them again? The better ones were printed at the time and those photos are now mostly in a large storage tin and boxes. It has often been said the best digital storage is actually good old fashion printed photos -- and they so inexperience these days; even ones printed at pro labs! 25 years ago we sold minilab 8x10-12 inch pics for $10-13 --- 6x4 postcard were a $1.10!! A 36 exposure film and printing was something $25>> . Yep; a think before clicking needed and you can basically forget this rapid burst stuff of today.
At least today it's so much easier to keep track of all the files

The other big drama IMO, or even biggest drama is the number of photos we take because it's considered FREE!! But how many of all these super high resolution RAW/DNG files actually get printed and printed large enough to justify the need of raw files apart from a cover for bad photography

sorry; I got carried away again because I'm very much about showing others we don't need it all and the $$$$ to enjoy photography and even be good at photography .
 
I guess the theory is that Image data is most important - metadata can be rebuilt and/or isn't as vital. However if you write to files, and store those in verified rotating backups, along with the many catalog backups one should be making (at every exit) - then you should be quite well covered for such a metadata scenario. And people should be keeping some OLD backups, for when they come across either metadata corruption or Stupid User Mistakes.

The danger in all these situations are errors that occur silently. NOTICING when you have corrupt data is a difficult situation, unless you have some over-arching process.

I've got a routine that checks all files. It is a pain because it is not tied into Photoshop, so when I edit in Photoshop, it reports that as a difference and I just have to remember. It's not too hard, as I do not do the initial checksum until after I finish post processing for a shoot (which I do within a day or so).

If it was integrated with lightroom (and if you use edit-in from lightroom to make the changes), Lightroom could notice and update the appropriate checksums, to reduce false positives.

It just seems that data integrity is not a priority for Adobe, or for that matter almost any home computer company. If you think about it, known technology to deal with this is absent from Mac (isn't it?) and from Windows except for some work on ReFS which is way immature and not available for most users anyway. BTRFS and especially ZFS are mature, solid mechanisms to avoid bit rot. Yet Microsoft and Apple crank out new feature filled releases without actually addressing reliability of storage using existing (and free as far as I know, as I think both are open source) techniques.

The more I think about it, perhaps it is a good thing to NOT have lightroom do integrity checks itself, since if they introduced a bug in Lightroom that corrupted data, it seems very lightly the corrupt data would be re-checksummed and saved. For that matter it's almost certain that would happen in DNG's also. Hmmm... maybe I was on the right track in the first place - do it outside.
 
The danger in all these situations are errors that occur silently. NOTICING when you have corrupt data is a difficult situation, unless you have some over-arching process.

It just seems that data integrity is not a priority for Adobe, or for that matter almost any home computer company. If you think about it, known technology to deal with this is absent from Mac (isn't it?) and from Windows except for some work on ReFS which is way immature and not available for most users anyway.

Ferguson,

How right you are. There is a memory technology called ECC. Server computers all use ECC memory, but not consumer/home computers. If I wanted to use ECC memory in one of my computer builds, I would probably have to spend an extra $200 or more just for the motherboard, and lots more for the memory. IF consumer/home computers used ECC, volumes would increase and the price differential would come way down.

Phil
 
I could point to all my old personal and work negatives gather dust -- am I really likely to use them again? The better ones were printed at the time and those photos are now mostly in a large storage tin and boxes. It has often been said the best digital storage is actually good old fashion printed photos -- and they so inexperience these days; even ones printed at pro labs! 25 years ago we sold minilab 8x10-12 inch pics for $10-13 --- 6x4 postcard were a $1.10!! A 36 exposure film and printing was something $25>> . Yep; a think before clicking needed and you can basically forget this rapid burst stuff of today.
At least today it's so much easier to keep track of all the files

For me, digital made it so that I actually look at my photos. I have a lot of old-fashioned print-from-film photos, but they've been gathering dust for a couple decades. I even ended up throwing out about a dozen ancient undeveloped film rolls due to the developing costs. (The 'important' ones would have surely gotten developed.) I think about scanning all the ones that did get developed, but then I think of all the work that would need to be done - on top figuring out all the color management and other issues that relate to scanning. So the digital photos are basically the only photos that exist in my mind.. I have yet to find a reason to print any photos. I bought a photo inkjet many years ago along with some photo paper, thinking to make some prints - but I have not even printed one single solitary photo. So to me it's either digital - or it don't exist :stop:.

The other big drama IMO, or even biggest drama is the number of photos we take because it's considered FREE!! But how many of all these super high resolution RAW/DNG files actually get printed and printed large enough to justify the need of raw files apart from a cover for bad photography

And also because it's 'free', we end up taking so many just slightly different shots of the same thing.. And after getting them into the computer, can never decide on exactly which ones should be called the keepers, and the others 'trash'. ... What.. Just me?? ... That's why I ended up using all the color labels that were suggested in The DAM Book: neutral, outakes, and outakes-of-outakes (which I renamed from Krogh's "Trash"). Then on top of that I use, so far, up to 2 (or maybe 3?) stars - but as you can image, it gets just a wee bit arbitrary at this point.. But I didn't want the stars field to go to waste for the so-called 'negative' color labels. :D
 
Status
Not open for further replies.
Back
Top