Output File Size

DOwnloaded the trial to check out and have an initial question:

Why are the resulting DNG files almost 3x larger than the originals? When I process NEF files in Lightroom and export as DNGs, those DNG files are roughly the same size as the NEFs.

1 Like

Please note that the same is true for PureRaw.
Output files are “linear DNG” files, hence the size difference.

Copied from Adobe: Adobe Digital Negative Converter

  • Linear (demosaiced)
    The image data is stored in an interpolated (“demosaiced”) format. This option is useful if a camera’s particular mosaic pattern is not supported by a DNG reader. The default mosaic format maximizes the extent of data preserved. Mosaic image data can be converted to linear data, but the reverse is not possible.

PureRaw is a useful programme that has a lot of potential. I have also noted that DXO PureRaw produces DNG files three times the size of the original raw files.

I have understood the explanation given above that DXO Pure Raw Converts to linear DNG. However, users need to be given the option to have a smaller raw or Tiff file. IMHO this is a problem that needs to be fixed.

My CR3 files from my 90D are about 45 Mbt. If I convert these with Adobe DNG converter then the raw files with lossless compression are about 50 Mbt which is acceptable. DXO PureRaw produces raw files about 135 Mbt.

For example, DXO RawPrime could export to a tiff with lossless compression (LZW or zip). That would be better than a jpg. Or maybe the file can be exported to tiff and then back to the smaller non linear version of DNG.

Just my 2 cents worth.

Thank you for your hard work in developing this software.

Kind regards
Chris

1 Like

Somebody may correct me.

DNG files are TIFF based files. They both contain RGB values: 3 channels per pixel.
RAW files do contain R,G,B values: 1 channel per pixel.
My Nikon D750 has 6016x4016 pixels. For simplicity the sensor has the same elements.
I shoot 12 bit RAW. The RAW image will be 6016x4016x12=289923072bit or divided by 8 36240384byte. or 34.56MB.
The 16 bit DNG/TIFF takes 2 byte per channel. That means 6016x4016x2x3=144961536byte or 138.25MB. NB: 16 bits.

Lineair DNG means no corrections to the demosaiced image has been done. This doesn’t have any influence on the amount of pixels.

From Adobe https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf
Under the tag compression I read that if compression has been used, it will be a JPG compression.

So what I would think that if the DNG file is not much bigger as the raw file, then either 8 bits has been used or a JPG compression has been used or both. If so, I wonder why I had to shoot in RAW??

George

1 Like

Please see recent reply to a post by someone regarding output tif file. Thanks

Thank you George for this explanation. Its clearer after seeing the math.

However it must be theoretically possible for DXO to output to a smaller tiff in a future release. It will be helpful if users have this option. Not everyone wants a massive DNG.

Here is what I have found.

1- Original CR3 File 46 Mbt
2- Converted by Adobe DNG Converter 50 Mbt (Lossless)
3- CR3 converted to DNG by DXO PureRaw 134 Mbt
4- 134 Mbt DNG From DXO opened as 8 bit rgb PSD 85 Mbt
5- Same PSD saved as 8 bit RGB TIFF (LZW Lossless) 39 Mbt

Users could be given the option of choosing which modules of DXO PureRaw to apply. If de-mosaicing is not required, then maybe it will be possible to get a smaller DNG or Tiff.

Kind regards
Chris

Keep in mind that the Adobe DNG converter is creating DNG files with no edits while PureRAW is adding corrections for Chromatic Aberration, Lens Distortion, Vignetting, Lens Sharpness and Noise Reduction. Perhaps an increase in size by 3 times is excessive, but I would expect the output to be much larger than from the DNG Converter.

Mark

Strange things with Adobe DNG converter.
NEF file D750 6016x4016 30.5MB
Adobe DNG converter .dng file 1024x684 27.6MB, 24 bit color depth. When opened in IrfanView it shows 1024x684B. When opened in Photolab it shows 6016x4016.

Chris, have a look at the properties of your file from the Adobe DNG converter.

George

It seems that the properties are referring to the embedded jpg.
To me the only explanation for the smaller file size is that it is stored in the original, not de-mosaiced, structure with the camera parameters in a general structured form.
When I select a embedded jpg in full size, the file size is about the original size.

George

@George, I just checked Adobe DNG Converter output at different settings. This is what I got with a few images of different dimensions and size:

Each of the four blocks of images go along with the following rule:

  • Original file (white background vs blue background of DNG files)
  • DNG output with DNGC set to default conversion settings
  • DNG output with DNGC set to custom (version 1.4) setting, uncompressed
  • DNG output with DNGC set to custom (version 1.4) setting, compressed
  • DNG output with DNGC set to custom (version 1.4) setting, linear, uncompressed
  • DNG output with DNGC set to custom (version 1.4) setting, linear, compressed

DNG output files can be smaller than the original file or considerably larger.
DPL 4 (current release) accepted+processed all DNG files without hesitation.
DPR 1 (current release) accepted+processed all except the linear DNG files (expected)

DPR Output size per file is the same, no matter what type and size the file image had,
the example image 20130220-5160 turns out to have a size of 99 MB, which is close to the 94 MB of the linear compressed file written by Adobe DNG Converter but still about three times the size of the original .cr2 file.

My lessons learned: If drive space matters, don’t use DPR - or get more drive space.

What I see that when no linear RGB is involved that then just the CFA from the original RAW is copied and the parameters rearranged in a standardized order. Just a guess.
The difference in size between the original and the …-std.dng is caused by the smaller embedded jpg. When you choice a full size embedded jpg the file sizes are nearly or exactly the same.


The addition ad means adobe.

George

It’s not going to be me because your calculations are sensible.

We all end up with 3-channel per pixel output no matter what formats we go through in between. When starting with RAW formats there is only one channel per pixel. The magic of DeepPRIME is the conversion between single channel and 3 channel which, at the most basic level, will triple the output size because there is triple the information. This level of information is always used when RAW conversion is done, it’s just that we don’t usually write it to a file if the conversion and subsequent processing is done in the same application.

There are nuances to do with how many bits the input data is and how many bits the output data could be said to contain as a result of the input and the conversion process. That’s above my head, though.

So if you don’t like the size of your linear DNGs, just do what would happen if you did everything in either PhotoLab or Lightroom — delete it as soon as you’re finished. Of course that means you cannot keep your adjustments to return to them.

Although… there is a way to proxy the DNG in Lightroom. You could build Smart Previews which will act as a proxy for the DNG and if you need to re-export, you could recreate the DNG using PureRAW.

By example a sensor of 4000x6000 sensels. The camera uses a bit depth of 12. The theoretical size of the raw data will be (4000x6000x12)/8=36Mb.
When converted to an 8 bit RGB raster image that theoretical size will be (4000x6000x3x8)/8=72 Mb. and with a 16 bit color depth 144Mb.
All theoretical sizes. Camera does compress the raw too.

What I wonder about is the use of the DNG files.
When used non linear it has the same size as my NEF files when using the full embedded jpg.
When used linear it stores a linear tiff file. And linear means that no gamma correction has been used. That’s what I figured out. So why not using a normal TIFF file??

George

DNG is a container format. Within it there can be different content types, same as with TIFF. Different bit depths, different codecs (e.g. JPEG previews versus compressed but lossless original data), and possibly multiple entities (also previews etc).

From Wikipedia:

As a “normal TIFF file” is quite similar to a “normal DNG file”, the question at hand is what do we want put in it?

Personally I wouldn’t see any value in outputting 8-bit data as at the point of demosaicking I have yet to make my exposure and colour adjustments which is why I shoot RAW in the first place — to have the bit depth available to do that to maximum effect.

DNG can be (and generally is) compressed too. If it was not, every frame from a given camera would be identical in file size. I just looked at one (busy) month’s images* and they range in size from 21.8 MB (a too-dark night exposure with little texture) through to 42.9 MB (with most of the frame containing a fine netting material). The average size of the 774 images is 33.2 MB.

My sensor resolution is 6060 x 4013 at 14 bits, which is approximately 42.56 MB of uncompressed data, suggesting that largest file I found is actually suffering slightly at the hands of compression, but my average frame is compressed by 22%.

When converted to a full RGB frame of 6016 x 4000 at 14 bits, it comes to a theoretical 126.34 MB uncompressed. Put that in a 16 bit file (TIFF seems to only ever come in 8 or 16 bit varieties) and we’re at 144.38 MB.

I took that smallest DNG and converted it with PhotoLab using “Denoise and optical correction only” (equivalent to PureRAW) and it came out at 77.2 MB. The largest became 115.9 MB. When I exported them to 16 bit TIFF they both came out at 144.4 MB. I note that PhotoLab’s choices for TIFF include “8 bit” and “8 bit compressed” and “16 bit” so it stands to reason there is no compression on the 16 bit TIFF, as borne out by the sizes which very closely match my theoretical number.

If the 16 bit TIFF could be compressed, I’m guessing we’d arrive at the similar numbers to the DNGs, though it would depend on the compression algorithm in use (I’ve seen at least two used in TIFF files).

So I get back to my original point. TIFF or DNG makes little difference. Theoretically storing only 14 bits could be a significant saving and it does seem like the TIFF 6.0 standard would support this (and by extension TIFF/EP and DNG) but I can find no reference to this actually being used and support would likely be almost non-existent in other software.

*Note that my DNG files are camera-native, so they are the original RAW files.

I think we’re on the same line.
I just wonder what is the practical use of DNG. It’s meant as a universal raw file, replacing all the other different raw formats. It can be created with either the original RAW file or with a TIFF file. PL creates a DNG that includes a 8bit TIFF file. For me that’s to less for further editing.
When creating a DNG from a NEF file with the Adobe DNG-converter, the filesize is nearly the same when a full jpg is added. When I choose the original NEF file to be embedded, the file size just doubles about.
I just wonder what is the profit of a DNG in general and of a DNG containing linear TIFF special.

George

Why DNG? When I tested DNG vs. TIFF, I found that

  • DNG files were roughly half the size of the TIFF
  • DNG files allow for temparature/tint WB, which TIFF doesn’t offer
  • Metadata can be written into both files, which can be a plus compared to sidecar files

DNG containing linear TIFF?

  • A DNG file contains image info in raw (r,g,b,g) format or linear (rgb,rgb,rgb,rgb) format
  • The term “linear” as in linear DNG as used by DxO means that (rgb,rgb,rgb,rgb) is used

Images can look different too:

Upper row: original RAW, linear TIFF and linear DNG, both made from the original RAW
Lower row: Same as above with DxO Smart Lighting

In that case I think you compare 8 bit and 16 bit or RAW DNG with linear DNG. PL uses 8 bit for its linear DNG.

Maybe when you use a RAW DNG. But I don’t see why something can be done on linear DNG and not on TIFF. They contain both a RGB raster image. Exposure compensation is als done mostly on the RAW data.

What I learned a linear DNG image is a RGB raster image without gamma correction. That’s why linear. But whatever it means, it is a RGB rasterimage.

George