Dealing with file size in dams

Hi I use dx0 pureraw to correct my nikon z7 photos which are 45mbs. When I run pure raw the files end up being 150mb each. This is isnt a moan about file size as the quality is well with it. Just a question how do people manage the huge file size convert them to another format or deal with it and get more SSDs. Cheers

I don’t store processed images, but export them just before I need to.

Hello @Nxdibbles, welcome to the forum. DNG is famous for being extraordinary big. Like @platypus I only export the RAWs when I need them - but never DNG, only either JPG or TIF. I don’t know PureRAW, I guess it’s the only way to use the RAWs with PureRaw edits in Lightroom? And my Z 7 files are alreday in the 55-60 MB region (it’s 45 MP, maybe you confused that?) I was just wondering why your RAWs are so small… Are you saving the NEF in camera lossless compressed (which is thenway I set my Nikons) or uncompressed? I guess, you set the camera to uncompressed and then export DNG explains the 150 MB as DNG is a container.

Up to you to get more SSDs, but you might consider: Do you keep the genuine NEF? If DNG grows the NEF 2×, then one image needs 3× the storage. Or, instead storing 3000 NEF you only can store 1000 NEF + the DNG exports.

Yea sorry mean to say 45mp and uncompressed which your right comes 55-60mb. Yes I keep both nefs and dngs. I use capture one with the dngs and keep the raws on an external drive. I always cull, keyword, pureraw then edit in capture one. I maybe be doing it wrong and should use pure raw the at the end before I export

I also use Capture One and DxO PhotoLab Elite. The latter only if I need DeepPrime noise reduction. After 1 1/2 years of learning DxO PL, it’s good to have both possibilities. Everything else I do with C1 from import to export as I need a RAW converter with acceptable image management. That, and some lesser issues, unfortunately exclude DxO products.

I admit I’m too stingy with diskspace to use NEF and double size DNGs for the same image. In one way I don’t trust DNG enough to throw away the genuine NEF. It was meant as a universal RAW format and few manufacturers like Leica, Sigma, Pentax/Ricoh use it as native RAW, but the others stick to their genuine formats. I simply dislike the massive size of DNG.

A Sigma dp 0 quattro can produce both Sigma Foveon RAWs or DNG. DNG has no better quality but uses 2 1/2 × more diskspace - and that’s an Adobe thing, be it Photoshop, Illustrator, InDesign - always a bit bigger than others. Bigger alone is not better.

I’m not sure where DNG gets this reputation for enormous file sizes.

My Pentax is set to record DNG — a decision I made many years ago when upgrading to a camera that could write them in approximately the same time as a native PEF file. A PEF and DNG are very close in file size for any given shot. I just shot a test scene:

  • PEF = 34.3 MB
  • DNG = 34.7 MB

This should come as no surprise as they hold the same amount of sensor data, which is by far the largest contributor to file size. This also explains why DNG has “no better quality” than a native RAW — because the quality is identical.

Linear DNGs exported from PureRAW or PhotoLab will approximately triple in size because they contain triple the data. These are no longer “sensor dumps” which contain, say, 24 million pixels, each of which is red, green, or blue. They contain 24 million full colour pixels as a result of the demosaic process and other corrections. Where before you had 12 bits of red or green or blue you now have all three — 36 bits per pixel, three times the data, and therefore approximately three times the file size.

DNGs that come from the camera, or DNGs which are converted processing-free from PEF/NEF/CR2 etc (such as by using Adobe DNG Converter, or Lightroom), do not carry any real penalties in size.

Linear DNGs obviously carry a massive penalty of size but they cannot be said to be “originals”. The concerns over Linear DNG size and “keeping originals” are two separate things.

PureRAW is meant to be used as a first processing step. The best potential for further processing is with DNG (jpeg is almost useless imo), and if one wants DPR’s quality output, one has to live with these files and the space they require.

In order to be able to fall back to the originals, one has to keep the ooc RAW files too. Better be prepared with a lot of storage space and for moving files between primary and secondary storage.

Plus that the RAW files are written in 12 or 14 bits blocks. So every color is written away in 12 or 14 bits blocks. As soon the image is demosaiced and saved in (linear) tif they are written away in 24 or 48 bits blocks.
So a 16 bit tiff (48bits) compared to a 12 bit raw will be even 4 time as large. Without compressions and additional data.
I use Nikon.

George

1 Like

From it’s enormous file sizes. Rhetoric questions… I’m not sure if they deserve rhetoric answers? :grin:

Nikon D7100 24 MP (lossless compressed): 28 MB
Nikon D750 24 MP (lossless compressed): 28-30 MB
Fuji X-T2 (lossless compressed) 25 MB
Leica SL2S, 24 MP DNG: 44 MB
Sigma dp0 quattro native X3D: 58 MB
Sigma dp0 quattro DNG: 110 MB (and although most photo apps can read these DNG, PL5 can’t. That’s no problem, Sigma’s adaption of DNG is not as good as their own format, so I need the Sigma RAW converter anyway)

As PEF and DNG are rather close and bigger than the 24 MP files from Nikon or Fuji I think Pentax adapted their PEF to DNG.

And frankly, what do you expect from a company (I mean Adobe) who’s renting software and disk space and makes more money the more and the bigger the files get? Adobe cares about business, not file size. This goes through all Adobe apps.

It wasn’t a rhetorical question at all. I stand by my assertion that DNG does not mean larger file sizes than native. Or, at least, it shouldn’t.

The reason my PEF and DNG are so close in size is because the same data is included in the files. The whole point of DNG was to provide a generic wrapper around the sensor data. That wrapper should describe the sensor characteristics in order to understand the data. In the end, I have ~24 million red, green, or blue values stuffed in one file or the other. By design they should be approximately the same size.

I have no idea what wild scheme Sigma have come up with, but it is not something that can be blamed on the DNG format. My suspicion with the sizes you have shown is they are trying to demosaic the data in-camera, which means it’s not really RAW but a linear DNG. Or perhaps they’re just being incredibly inefficient in recording the sensor data. I can only speculate.

I would be very interested to know what size DNG you would get if you converted an X3D file using Adobe’s DNG Converter application — it’s free and I don’t think it needs an Adobe account.

My 24 MP DNGs range from ~22-43 MB, averaging at 33.2 MB. I derived those numbers from my April 2019 photos which were mostly taken of all manner of subjects in Singapore (instead of mostly aircraft any other time). The 22 MB example was mostly dark with a couple of large blown out lights. The 43 MB example had almost the entire frame covered in very fine netting. I’m actually surprised at your quoted range being so narrow, as I find scene complexity can make a significant difference. Of this batch of photos, 90% of them fall within 29-36 MB but that’s still a much wider range.

FWIW In PL5 i just exported one of my Canon CR3 files (after editing) from my M6 MKii.(32 megapixel sensor).
RAW = 40.665 mb
DNG = 67.283 mb

@zkarj don’t get me wrong, I’d welcome a unified RAW format for all cameras, together with all metadata reliably written into one single file instead of having (in PL’s case) tow additional files.

One reason for larger files can be bigger JPG sizes included in it and camera manufacturers not willing to sacrifice processor performance to collapse JPG, or to compress RAW data.

Another reason it has not become more popular is the completely stupid name: Digital Negative Graphics - what is the negative part in this file format? It’s just another raw-format, used by small exotic companies, some of them no longer in the market of cameras.

I consider that as the usual Pentaxian DEFCON 3 - if anything doesn’t work, it simply can be based on what Pentax decided to use. Sorry, read that too often. Sigma has to find a way to bring their Foveon RAW data into a DNG made for Bayer sensors as most people, me included, don’t enjoy minutes of waiting until the data is rendered by Sigma’s own RAW converter.

I mean, they even don’t managed that PL is able to read the DNG data (like it is not able to read DNG from VueScan, btw.) therefore DNG might be not the universal RAW format Adobe would like to make people believe. And in the case of Fuji X-trans files: The uncompressed goes way beyond your 24 MP file sites, at 50 MB/file it’s eating harddrives.

When using the Adobe DNG converter and embed a full size JPG in it, the filesizes are nearly the same. Nikon includes standard a full size JPG. I believe other brands don’t.

George

Ah, I had not considered the JPEG previews.

My understanding of the intent of DNG was that it could hold any type of sensor data, along with a description of its layout.In overly simple terms, mark it as Bayer with a RGGB square, or mark it as Foveon with whatever layout it has.

Clearly that’s not how it has turned out.

I think, camera manufacturers are disregarding DNG for the same reasons which let all attempts of one single mount/interface between body and lens fail. I can also remember some people using Canon or Nikon RAW converters as these were the “only ones bringing the full capacity of the RAWs to the light”, or something like that. Same thing goes on in various camera forums. There were more film manufacturers back in the day than sensor fabs these days, but what manufacturer A does to the sensor readout, is soooo much better than the rest of the world.

I’m afraid, we photo-gear consumers are sometimes in need for every kind of identification and confuse brands with “own style”.

1 Like