Reduce DNG files size when exporting?

Or, said in other words: the original raw sensor data is (sort of) upscaled to final image resolution. That demosaicing is ‘filling in the blanks’, some sort of upscaling in very cruse terms.

But it does increase the amount of data, so it also increases the filesize.

Dng converter does - when possible - nothing but repacking the raw data. And applying a lossless compression (but most raw files also have something like that).

We come back to the two ‘modes’ a dng file can be. Raw sensor data, or ‘normal image data’.
In raw sensor data , we talk about each pixel as a measurement of light. In normal image data , we talk about a measurement of r, g and b per pixel.

Most non-tech photographers don’t realize that when we talk about megapixels, we basically talk about monochrome. A single value. A single measurement.
In case of a bayer filter, some of those pixels are for red, some for green and some for blue.
So not every pixel has information for red, green and blue together. Only one of the colors at a time.

So imagine a 24mp bayer sensor. That is a 24mp monochrome file what’s basically inside your raw file. Instead of 24mp monochrome, you can also think of it as a 6mp red , a 6mp green, another 6mp green and a 6mp blue file.
But, 6 mp.

Turning that 24mp monochrome file into a 24mp full color rgb file is creating more data. Or, turning those 4 6mp files (channels) into a single 24mp r, g and b inage is sort of (smart) upscaling of the starting data.

So, more data more storage. The dng files from dxo are technically not much more than a 24mp tiff image, with (or without) a form of lossless compression.

Can dxo do their denoising on just the raw bayer image instead of demosaicing? Maybe, sometime. But it could cause issues with the algorithms of the software you use to open the dng , like lightroom. Since they expect certain flaws of the camera to be in there, like a raw file. But those are corrected already.

If you really don’t like the size of dxo output dng, you can use adobe dng converter to convert from dng to dng. And you could opt for the LOSSY compression then. You loose data, i think they only work on lightroom, but they are a lot smaller.

I just see the dng files as a temporary step in the workflow. And i don’t keep them. I let dxo to its thing, then open and process the dngs, and when i have my output, the dngs are gone.

My output is often an uncompressed 16bit tiff file, also huge. That gets resized and processed according to output media and needs, and then that tiff file is also gone.

Processing gets better over time , or i prefer different tools or looks. If i had saved the dngs from the Optics Pro era, i could throw them all away now and start over. This will also happen, sometime, for deep prime xd files.

This is a result of the biggest difference between DNG and TIFF — TIFF is equivalent to a Linear DNG not a ‘regular’ DNG. TIFF files have full RGB pixels (generally at 8 or 16 bit depth), not bayer matrices.

If you have, say, a CR2 file, then your software needs to have knowledge of the layout of a CR2 file, both in terms of the headers and metadata, and also how the bayer matrix data is recorded. If Canon release a new camera with a new variation on the CR2 file, then your software has a potential problem.

DNG, at least as far as I understood it when I first came across it many years ago, seeks to address this problem by standardising the header and metadata layout and also providing additional metadata about the image data layout.

The theory behind this approach was any software which knew how to read DNGs would be able to read any DNG. At least in a technical sense. Whether said software had a good way of using the data, and turning it into something useful, seems to be where the issues lie.

In the case of PhotoLab, the ability to read the image data is only half the story. Its optics modules are tuned to specific sensors, so at best you might get the same features as you get with a TIFF file today — no optics modules, no ML noise reduction. Why hasn’t this been done? I don’t know, but I would suspect one factor would be the customer story could be very confusing, where some DNG files would, for example, allow DeepPRIME and some would not, possibly even from the same manufacturer with very similar cameras.

I’ve DNG converter 13.2.
I downloaded a Nikon Z9 and a Lumix 2511 raw file and tried to convert that to DNG.
It says : no images have been converted.
And : the file seems to have an unsupported format.

The DNG converter needs to support that specific camera too. What is the added value of DNG compared with a native raw file?

The current version of Adobe’s DNG Converter app is 15.2. and supports this gear.

No added value imo, but your mileage may vary. PhotoLab does not fully support DNGs, but limits itself to DNGs according to what’s written here.

Adobe’s intention was/is to create a universally accepted standard. Success has been and still is limited, only niche manufacturers natively write DNG files, but as an exchange format, it can serve its purpose of transmitting demosaiced, denoised files to other apps.

As far as I am aware, PhotoLab supports the same number of DNG files as it supports RAW files. Which is to say the DNGs it supports are from the same cameras whose RAW files it supports.

Yes, DNG is a supposedly ‘agnostic’ format, but DxO supports devices due to the way its demosaicking works.

I cannot deny the tag “niche” for Pentax, given market share. However, that doesn’t mean it is not a modern brand that produces excellent products that attract industry attention. The Canikon juggernaut is a triumph of marketing.

My mileage varies. All my metadata is written directly into the file so it is simply and completely portable.

Not one corrupted file in over 34,000 so far. (And I know this to be fact because I recently completed an export of every image in my library.)

This is probably the benefit of converting to DNG all non-DNG raw formats. As you write, @zkarj, if DPL supports the gear, all is well. If it doesn’t, we’d need to stick to other developers though.

Good. Same here with Lightroom and sidecar files. I’d not yet trust PhotoLab for such a constellation though.

All of the above being said, converting raw files to DNG seems to be a good way to ensure metadata sanity with PhotoLab, in spite of its missing abilities to keep the assets and the asset catalog (the DB) in sync. Whether said DNGs should be linear or not has not yet been established though. Due to the fact that Adobe writes that its DNG Converter can drop some of the original raw files metadata, it’s a good idea to keep the raw files - unless one is willing to accept such loss. But we’re not talking about the OP’s reduced file size any more :slightly_frowning_face:

The item is not that the DNG converter doesn’t support a specific camera but that it seems that it has to support a camera the same way as any other converter. Both has to support that camera.

If they’ve to support every new camera individual then there’s no new standard, only a new extension.

George

The converter has to be able to understand the layout of the original RAW file or it cannot find the right pieces to write into DNG and it cannot add a descriptor for the image data.

Most software can rely on the DNG to tell it the layout of the data, therefore not needing to understand the original RAW file layout. That makes it a useful new standard. And of course the software which reads DNGs can rely on consistent metadata.

I’m sure PhotoLab could easily be tweaked to open generic DNG files — something I have supported in these forums — however it would not be able to deploy DeepPRIME and lens sharpening for bodies it does not ‘know’. That is the issue with PhotoLab, I believe. Their secret sauce relies on the camera (and lens) being known. If it is known, PhotoLab will happily read original or DNG-converted files.

Put another way, DNG is a means of consistently delivering information about the file contents.

I never looked in the structures of the raw or dng files. But my practical experience, as just aquired, is that both the dng converter as the raw converter needs the development kit for the specific camera. As I see it the dng is just a shell around the raw file.

That’s something I doubt. I downloaded the latest dng converter and converted a Z9 file and a Samsung S5II file. The S5II isn’t recognized by PL6, both the raw and the DNG.

George

Most ← software. That does not include PhotoLab nor PureRAW, hence…

Which one does?

George

Blockquote
I just renamed a NEF to DNG and opened it in PL without a problem.

Blockquote
As I see it the dng is just a shell around the raw file.

The majority of raw file formats are TIFF based the exception being Canon files which are (I believe) QuickTime based. The TIFF based formats contain one or more indexes that point to both the raw data, jpeg previews and meta data. In a true TIFF the metadata allowed is defined in the TIFF file specification as a list of data tags, RAW files use these same tags but the camera manufactures add additional tags as they see fit. These extra tags will describe how the raw data is to be read from the bytes of data as well as other processing instructions designed to be used by those in the know.

The DNG converter takes the raw data and rewrites it in a standard way based on Adobe’s knowledge of the original raw file. Adobe are given this information by the camera manufacturers as they want / need their camera raws to be usable in market leading applications. So the DNG is a transcode of the original raw file; no data is lost it is just rearranged. Every raw file loaded into Adobe LR/PS is processed in the same way before it may be edited.

DxO will do similar otherwise the code base would get horribly complex and difficult to maintain.

Lastly, contrary to what many photographers believe it is quite possible to edit meta data such as keywords in raw files. The principle disadvantage is any changes mean that the whole file will get rewritten to back up media.

2 Likes

One of those little things one forgets but which can have a big impact. Thanks

Both the dng converter as the raw converter needs to support the specific raw file. It doesn’t add anything to my workflow, now and in the future.
When one does convert a dng file from camera x to a rgb-image then the used raw converter must support camera x. As far as I can see.
I’m still curious to see an example of how to use the dng, containing a raw file.
In general I’m wondering what raw data is missing for a raw converter that isn’t visible in exiftools.

George

The choice to use Adobe DNG converter is a personal one. The reason why a photographer may want to convert their files is in order to future proof them. While most camera raw files are Tiff based it does not mean that every value contained within the file is in the public domain as these files are controlled and changed by the owner. Use of these propriety raw formats means that you are trusting the likes of Adobe and DxO to continue supporting the format into the future. It is fine at the moment but in eighty years time who can guess if Adobe will even exist. So there is a risk that images stored in propriety formats may be lost. DNG is different in that the full file specification is in the public domain, this specification allows metadata to be stored within the file and in the case of Adobe products that includes the processing instructions.

In short the DNG file is future proof in ways that propriety formats are not.

DNG files are used in the same way as other raw files except there is no need to store user added metadata such as captions and keywords in sidecar files.

Sorry but I don’t understand your question. No raw data i.e.sensor data is missing but some camera makers tags and values may have been stripped. Sounds bad, but the same values will be ignored by Adobe products.

Exiftools is great but it does not decode every value present in every raw file, it can’t, as Phil Harvey does not have the resources to decode every value of every file type. Having said that I can’t think of any important value that is missing, its a wonderful tool and I say that having written code to extract a few values from raw files, writing data back to raw files as Exiftools can do is far more a complex task than just reading.

You mention a raw converter but there are many available and they may give different results depending on the underlying raw engine being used.

My own use case : I am adding captions keywords and other metadata to my raw Lumix and Olympus images and when complete I convert them to DNG. When I complete an edit that I am pleased with I create either a JPG or Tiff copy which is stored along side the raw/dng raw file. I also shoot with a Ricoh camera which uses DNG as its raw format.

In the final analysis you will probably not see any great advantage when converting images to DNG but your great grand children may.

Simon

Photolab doesn’t “happily read” all dng files from cameras and lenses it knows. I have a Canon EOS R and PL6 does not deploy lens sharpening for DNG files converted from EOS R cr3 raw files, regardless of the lens used. So Photolab does not even fully support dng files produced from a camera it supports the raw files from.

I know that story but I’ve my doubts about it. As long as the used converter still has to support the used camera there is no need to use dng.

I mean the tags. What tag does make a Nikon D700 different from D800 by example.

PL, and I think any converter, needs to support that specific camera. It doesn’t matter if it deals with a native raw file or a dng file. So even over 80 years I need the support of that camera.

George

Yes, any converter would. But once it is in DNG format, camera knowledge is not a requirement for decoding the image. Only for dealing with auto-correction of flaws.

Converted with Adobe DNG Converter? That is PL’s stated requirement (presumably due to variations in DNG generation from other sources).

If yes, and it’s not working, I’d report that as a bug. I can’t find where the requirement is stated (might be in the application help?) but it states that an Adobe-converted or camera-native DNG file for a supported camera/lens combo should always work.