Round-Tripping to CS3:Insane File Sizes!

[KED]

Hi everyone, especially you Gurus. I've been rocking and rolling with LR, but I've just noticed something really quite amazing that I will need to deal with sooner or later since I don't have infinite hard drive capacity. I am a CS3 idiot - essentially, I only use it to run noise reduction via Noiseware plug-in. I shoot RAW so my originals are in the 11 MB range. I had been exporting to CS3 at 16 bit, but that created 55MB files (!!!). Even dropping it back down to 8-bit doubles the file size to 22 MB +/-, and presumably upon the return to LR I now have my DNG at 11 MB PLUS a 22 MB TIFF or PSD. That will become an issue sooner or later and these are not, by any means, all museum-quality photos. What am I doing wrong?

Let me acknowledge that I had a comparable post months ago related to imports, but that was because (thanks to your help) I was unnecessarily embedding the RAW image along with the DNG. Hard to imagine how this could be related.

 

[rcannonp]

You're not doing anything wrong. Those file sizes sound about right for a TIFF or PSD from something like a 1'MP camera.

 

[KED]

You're not doing anything wrong. Those file sizes sound about right for a TIFF or PSD from something like a 1'MP camera.

But what's to double up from the RAW image? Not to mention 4X'ing at 16-bit? I admit that I have no clue about the technical details, but I have 75' GB internal and and another 75' GB external, and that could get exhausted pretty quickly with only moderately-important images.

 

[Brad Snyder]

The raw image is not exactly pixels. It's the raw data from the sensor mosaic, and to a certain extent, it's losslessly compressed in the camera as part of the file format. Typically in popular prosumer cameras the sensor buckets are recording at 12-bits. In the RAW rendering, each RGB pixel is determined by a matrix calculation of neighboring sensor buckets(which are in effect mono-chromatic). LR does this rendering in 16 bits. Your camera probably does it in 8-bits for JPG output. Unfortunately for the purposes of this explanation, a lot of this varies by manufacturer, and is typically proprietary.

When you convert this RAW rendering to an RGB uncompressed format, you'll need 3 bytes per pixel for 8-bit images, and 6 bytes per pixel for 16-bit images.

1' megapixels = 3' MB or 6' MB in an uncompressed raster RGB image.

Of course at this stage you can apply compression. I believe TIFF offers a lossless compression mode.

It's just the physics of the thing. High rez files chew up a lot of diskspace. One of the key advantages of LR style processing is that the changes you make are not "baked-into" the image. They're stored basically as a set of relatively simple text instructions that tell the rendering engine how to interpret the RAW sensor data. This takes a lot less room than actual pixels.

I guess, in the meantime I'll ask, at what point in your workflow do you actually need noise-reduction? Is there some less aggressive approach you could take, if, for example you're not printing every one? Would LR's NR be sufficient for web output? Just suggestions, to help conserve your space, clearly you know better what your needs are.

 

[KED]

The raw image is not exactly pixels. It's the raw data from the sensor mosaic, and to a certain extent, it's losslessly compressed in the camera as part of the file format. Typically in popular prosumer cameras the sensor buckets are recording at 12-bits. In the RAW rendering, each RGB pixel is determined by a matrix calculation of neighboring sensor buckets(which are in effect mono-chromatic). LR does this rendering in 16 bits. Your camera probably does it in 8-bits for JPG output. Unfortunately for the purposes of this explanation, a lot of this varies by manufacturer, and is typically proprietary.

When you convert this RAW rendering to an RGB uncompressed format, you'll need 3 bytes per pixel for 8-bit images, and 6 bytes per pixel for 16-bit images.

1' megapixels = 3' MB or 6' MB in an uncompressed raster RGB image.

Of course at this stage you can apply compression. I believe TIFF offers a lossless compression mode.

It's just the physics of the thing. High rez files chew up a lot of diskspace. One of the key advantages of LR style processing is that the changes you make are not "baked-into" the image. They're stored basically as a set of relatively simple text instructions that tell the rendering engine how to interpret the RAW sensor data. This takes a lot less room than actual pixels.

I guess, in the meantime I'll ask, at what point in your workflow do you actually need noise-reduction? Is there some less aggressive approach you could take, if, for example you're not printing every one? Would LR's NR be sufficient for web output? Just suggestions, to help conserve your space, clearly you know better what your needs are.

I can't argue with the physics and thank you for articulating them. At the moment I have abundant free disk space, but that disappears in a hurry at 5'MB/image. I shoot a lot of nighttime sports and other high-ISO situations, and anything that is a keeper, whether for web posting or for my own personal viewing pleasure, has to go through Noiseware. I shoot primarily with a Canon 1D mk III, supposedly among the best at high ISO/low noise, but at 16'' or 32'', Noiseware is an absolute must -- LR's NR just doesn't cut it, but that's why plug-ins exist in the first place. So the physics being what they are, I either get over my compulsive nature or I get more hard drive. Thanks for your time.

 

[Brad Snyder]

Kent,

And, as I'm sure you're aware, but for the edification of other interested readers who are following along, this creates an interesting paradox which aggravates the situation, because:

Higher ISO implies more noise, and more noise implies larger files, because none of the compression algorithms (lossless or otherwise) are able to distinguish noise from important high spatial-frequency data (meaning sharp edges and high contrast), so
the noise pixels(or sensels, sensor elements) are carefully preserved as important data.

 

[Victoria Bampton]

I either get over my compulsive nature or I get more hard drive.

One more alternative for you - if these aren't 'museum shots', as you put it, you could consider saving to 8-bit quality 12 jpeg instead. File sizes would be much smaller, and depending on how much more you need to do to these files, it's sometimes a good trade-off.

Now before we start a big debate on tiff vs. jpeg, and 8-bit vs. 16-bit, do the reading for yourself. If you were going to keep opening and resaving these files repeatedly, I probably wouldn't suggest jpegs. But on the other hand, if you've done most of your work in LR on the raw data anyway, I'd seriously consider taking that file into PS at uncompressed 16-bit, but then saving it off as a jpeg for archiving. After all, if one of those shots did need to be perfect, you could always open the raw file in LR and output as 16-bit tiff and just re-do the noise reduction anyway.

Don't take my word for it though - do a few tests and decide for yourself whether it's a compromise that's right for you.

 

[KED]

I certainly knew about the high-ISO/high-noise tradeoff, but I didn't know what special treatment was given to noise after the fact!

 

[KED]

One more alternative for you - if these aren't 'museum shots', as you put it, you could consider saving to 8-bit quality 12 jpeg instead.

Thanks Victoria, I will definitely try that.