Can LRC's DNG files use internal compression?

[dfkotz]

Lightroom Version Number

Lightroom Classic version: 11.4.1 [ 202206241800-b406ce4c ]

Operating System

1. macOS 12 Monterey

I just switched to a Canon R5, which produces CR3 raw files. I've read a lot about the difference between its RAW (uncompressed) and CRAW (compressed) formats, and decided to use CRAW; the raw files are about half the size. (I'm not here to debate that choice today.)

My LR workflow always converts raw to DNG during import. (I'm not here to debate that choice today.)

In the past, on my Nikon cameras, the DNG files were a bit smaller than the raw NEF files - because neither were compressed. But with my Canon I find the DNG files are (on average) double the size of the Canon CRAW files. As comfirmed in a prior forum thread, CRAW Files doubling on conversion to DNG, LRC is decompressing the CRAW before storing the image in the DNG file.

Question: is there some way to set LRC so it will compress the image inside the DNG file? It would halve my use of disk space! If DNG were to use lossless compression on the data blocks, it would reduce disk usage, possibly reduce load time, and be useful for all cameras (not just Canon CRAW).
(FYI, I have "Embed Original Raw File" OFF, and "Embed Fast Load Data" ON.)

 

[johnrellis]

DNG provides two compression options: Lossless (the default) and lossy. The lossless compression gives modest space savings compared to the original raws, while lossy can save more than 50%. I did a quickie experiment with two .NEFs, one with a fair amount of detail throughout the image, the other with a large amount of flat sky:

I've never used DNG lossy compression, but a google search shows lots of reports that it's "excellent" with much less noticeable losses than with JPEGs.

Since you've already decided to use lossy compression (CRAW), it might be that CRAW > DNG Lossy gives acceptable image quality compared to CRAW > DNG Lossless. Also compare with CR3 > DNG Lossy, which would avoid a second compression step but at the expense of dealing with larger CR3 files on your camera card and during import.

Intuitively, we might expect that CRAW > DNG Lossy would be visually inferior to CRAW or CR3 > DNG Lossy, but I wouldn't jump to that conclusion without doing some careful experiments. Years ago, I tested editing high-quality JPEGs and couldn't detect any visually noticeable degradation in image quality after 5 edit/recompress steps.

 

[dfkotz]

DNG provides two compression options: Lossless (the default) and lossy. The lossless compression gives modest space savings compared to the original raws, while lossy can save more than 50%. I did a quickie experiment with two .NEFs, one with a fair amount of detail throughout the image, the other with a large amount of flat sky:

View attachment 19054

I've never used DNG lossy compression, but a google search shows lots of reports that it's "excellent" with much less noticeable losses than with JPEGs.

Since you've already decided to use lossy compression (CRAW), it might be that CRAW > DNG Lossy gives acceptable image quality compared to CRAW > DNG Lossless. Also compare with CR3 > DNG Lossy, which would avoid a second compression step but at the expense of dealing with larger CR3 files on your camera card and during import.

Intuitively, we might expect that CRAW > DNG Lossy would be visually inferior to CRAW or CR3 > DNG Lossy, but I wouldn't jump to that conclusion without doing some careful experiments. Years ago, I tested editing high-quality JPEGs and couldn't detect any visually noticeable degradation in image quality after 5 edit/recompress steps.

Thank you for this detailed explanation. Your experiments are with NEF files, which are not compressed and thus it is not surprising that the DNG files are about the same size (a little smaller) - consistent with my former Nikon experience and with most web commentary about DNGs.

Canon's CRAW is already compressed, and somewhat lossy as you note. I'm not too keen on adding a second layer of lossy compression on my photo collection. I wish LR had a lossless compression option.

 

[dfkotz]

BTW, I did my own experiments, with results here. I snapped 14 random photos from the backyard, with a variety of tonal and brightness ranges. The table shows their size (in KB) as CR3 files (CRAW), and as DNG files after import to LRC. I then used the average and maximum sizes to forecast the number of photos that would fit on common memory cards and SSD drives.

For fun I also compressed (with gzip) those DNG files, and surprisingly they did not compress well... which seems to imply they are already compressed in some way??

 

[johnrellis]

I'm not too keen on adding a second layer of lossy compression on my photo collection.

That would be pretty much everyone's intuition. But the difference between CRAW and CRAW compressed to DNG Lossy may not be visually noticeable. These compression algorithms aren't like the 30-year-old JPEG algorithms that we were all trained to fear.

I found a sample R5 CRAW online, with a range of highlights and shadows, and only by deep pixel-peeping could I see any minor differences between CRAW and CRAW > DNG Lossy. But that's a sample size of 1, so not to be trusted either way.

 

[johnrellis]

I wish LR had a lossless compression option.

It does -- that's what's used when you convert any raw to DNG in LR and don't select Use Lossy Compression. That's what accounts for the approximately 15% shrinkage from NEF to DNG I observed above.

I think what you're wishing for is a DNG option that would convert the compressed CRAW data to DNG format, using the same compression algorithm as in CRAW, so there wouldn't be an uncompress/recompress step that potentially might introduce more noticeable loss.

 

[johnrellis]

For fun I also compressed (with gzip) those DNG files, and surprisingly they did not compress well... which seems to imply they are already compressed in some way??

Right, that's the default lossless compression that LR's DNG conversion always applies unless you select Lossy.

 

[dfkotz]

Thanks - good to know that LR is already using lossless compression for the DNG files.

I looked for a mechanism to use lossy compression during import... but amazingly there is no such setting. It appears that it is only possible to choose lossy compression when converting to DNG after import. I'm not sure I'd want to spend the extra time to import raw files and then later convert them to DNG.

For anyone following this thread, here's a 2018 video from Tony Northrup that does a nice job describing the advantages of lossy DNG compression... he uses it only for the lower-rated photos in his library.

 

[Johan Elzenga]

I don’t like to watch videos, but I would be interested to learn what advantage lossy compression has over lossless compress, apart from saving a bit of cheap disk space.

 

[dfkotz]

I don’t like to watch videos, but I would be interested to learn what advantage lossy compression has over lossless compress, apart from saving a bit of cheap disk space.

Saving disk space is the point. There is no other significant advantage to lossy compression. (ok, maybe a slight speed improvement when LR needs to load the file from disk, but it's not likely substantial.)

The tradeoff is disk space vs. image quality,

 

[Johan Elzenga]

That's what I thought. Because disk space is dirt cheap these days, that's a trade off I would never make.