Possible problem in DeepPrime XD support for Pentax K-3iii

Hello,
I own DxO PL5 and have been experimenting with the trial version of the new PL6, and in particular the new DeepPrime XD noise reduction. I think that with one of the cameras I have tested (Pentax K-3 mark 3), it may not behave “as intended”. I will post a few examples in the following. I have taken two similar images at ISO12800 with two different cameras to showcase a general behavior that I am seeing at high (>=1600) ISO with this camera, although it is less noticeable closer to this threshold.

This is shot with a Fuji XF10, 24MP APS-C sensor (non X-trans). Images are named with a suffix contaning the settings, the first number is the master (luminance) slider, if there is a second number I refer to the “noise model” slider that is new in DPL6.

As you can see at the default luminance of 40, DeepPrimeXD produces the same level or slightly less grain in the blurred areas with no artifacts, and preserves even more detail where there is any.
Most importantly, there is a strong dependence of the residual noise on the luminance value, from 0 to 100 (last two pictures) I am clearly exploring the trade-off between detail preservation and removal of grain. The following is a similar shot taken with Pentax K-3iii:

On the contrary with the K-3iii at high iso at L40, DeepPrime XD produces way more grain in the out-of-focus areas, and there is almost no difference in going from 0 to 100 with the luminance slider; this seems rather abnormal to me as it is unlike any other cameras.
Using the new “noise model” slider with negative values I can obtain results similar to DeepPrime (last picture) but more often than not this introduces an unnatural look in the out-of-focus areas (artifacts) and does not significantly improve detail retention with respect to DeepPrime.
Even keeping the “noise model” slider at negative values, there is very little sensitivity to the “luminance” slider for DeepPrimeXD. This is unlike the other cameras that I have tried.

As an additional example, the following is a comparison from a studio test shot available online at iso 25800 with the Pentax KP, that is a “similar” camera in the sense that it features pretty aggressive in-camera raw noise reduction that cannot be disabled, just like the K-3iii. However, DeepPrime XD seems to work very very well, just like with the Fuji; less grain, more detail, and strong sensitivity to the luminance slider, as expected. I would think that similar results could be expected from the K-3iii.
(Unfortunately I was unable to find a RAW studio scene from the same source for better comparison).

As a side note, I have tested the last 6.0.1 update and I saw no changes to this particular issue.
Thanks in advance if you can provide any feedback.

4 Likes

@Simone_S Welcome to these forums, great to see you here!

Although I’m not a K-3 III owner, it remains the most desirable upgrade path on my radar. Playing around with some K-3 III RAWs available on the internet, I have been able to replicate the high-ISO “splotchiness” that Simone found when applying DeepPrime XD, which is absent with regular DeepPrime.

It may well be to do with in-camera noise reduction already baked into the RAW files, which is introduced by Pentax’s Accelerator chip, a feature of their most recent models K-70, KP, K-1 II, and K-3 III. The threshold where that in-camera noise reduction sets in actually seems to vary across models. Independent data published on photonstophotos.net suggests that with K-70, KP, and K-1 II, noise reduction starts at ISO 640, whereas K-3 III applies it from as early as ISO 200:

https://www.photonstophotos.net/Charts/PDR.htm#Pentax%20K-1%20II,Pentax%20K-3%20III,Pentax%20K-70,Pentax%20KP

Perhaps, @Marie or others could pass this issue on to the development/support staff? It would be so great if this could be looked into, and maybe fixed in a future update.

1 Like

Thank you, @Simone_S , for your investigation and for sharing it! There might be a trivial cause, but I hope that the DxO staff will shine some light on this issue.
As a K3 III owner, I will be an interested follower of the thread and wait a bit before upgrading from PL5 to 6 :wink:

3 Likes

I would also like to share that the same issue has been reported before, with the same camera (K-3iii). It is possible there is an issue on how DeepPrime (and XD in particular) supports this specific camera.

DeepPrime XD doing nothing - DxO PhotoLab - DxO Forums

2 Likes

I downloaded the trial version of PL6 and did a short test with a few of my own images.
The first one is a Pentax K5 image, 800ISO. I usually select a low setting for the luminance noise (10 instead of the default 40), because I prefer some LM noise and better detail to the smooth/plastic look of high LM settings.
The detail preservation of DP-XD is better than DP and the noise removal as well, but the differences are small and for these ISO settings I would generally not select DP-XD, because of the processing time.
The second image is a Pentax K3 Mk.III image, ISO 6400 and under exposed by approx. 2 stops. The detail preservation of DP-XD is better and I prefer the noise rendering of DP-XD. It is different from DP, not necessarily better or worse. What I do see as well, is the color cast that is mentioned in other discussions.
If I change the luminosity setting to 50, the effect is as expected: less noise and less detail.
Recapitulation: nothing peculiar, except for the slight color cast.
The uploaded images are from the second file (K3 Mk. III).

2 Likes

Thanks, from the image you have posted it seems to work as intended in terms of the behavior of the luminance slider. I have, however, tested tens and tens of images and always found the (abnormal, in my opinion) behavior described in my original post. That can also be seen in some images found on RAW sample galleries available on some popular websites. There seems to be some degree of dependence on the particular image where in some conditions the algorithm indeed works as expected. But the supposedly abnormal behavior is quite ubiquitous.
Unfortunately, I no longer have access to PL6 as my trial has expired so i’d be unable to do further testing with your image. Maybe I could share one of my test images and you could try. I wonder if there could be an explanation somewhere else…GPU type/drivers (doubt that, but it’s worth investigating), sample variation between cameras (highly doubt that…)…

No problem, you can share some of your images and I will process them. You can share .dop files as well, as the problem can be in the settings. You can make a proposal on how to get the files in my computer :slight_smile:
Sometimes very strange phenomena can occur. I often export as .dng, after corrections in PL5. The resulting linear RAW dng-file is further developed in Picture Window Pro 7, as PWP uses the HSV colour model for RAW conversion, that preserves bright colours better than RGB. In some cases, the result has a very (I mean, VERY) strong magenta cast, as if viewed through a strong magenta filter. It is caused by a combination of specific lenses and the geometrical correction for these lenses. It is only seen in PWP, and disappears if I deselect the geometrical correction. The DxO team tested a few of my files, but couldn’t solve the case. And I can understand that they don’t want to put too much effort in it, since it is a rare phenomenon that occurs in combination with a not very popular RAW editor.

I have uploaded some files via Wetransfer. There’s an ISO1600 picture that is “on the edge” where the problem starts to appear. It is subtle but still noticeable in my opinion. Then you’ll find an iso5000 and an iso12800 image where it is very noticeable, particularly the latter that is not particularly sharp and not particularly well lit. On my system and with the included .dop files, there is almost no difference between Luminance set at 40 and set at 100, and there is significantly more noise and artifacts (with respect to regular DeepPrime) in the out of focus areas, that does not go away by moving the luminance slider. The only way to have a somewhat acceptable compromise is to move the Noise model slider way to the left (but still, there is little sensitivity to the luminance slider. The same effect is noticeable in the ISO5000 image as well.

Thanks for uploading, an interesting problem.
At first sight, I can only reproduce your conclusions. I tried various settings, and found no effects of e.g. switching lens sharpness on/off, lens corrections on/off and other (non-logical) variables. I happen to own the same 55-300mm lens that was used for your sample images and shot a few pictures myself at ISO 12.800, and compared with similar shots with a Sigma 17-70mm. I found no differences, there is no cause in the lens corrections. I think that I didn’t reproduce your problem with the sliders. But I agree, that the NR for this image with Deep Prime, and even Prime, is better than with DP-XD.
It occurs to me, that there is something strange in your 12.800 ISO sample. I think, that the luminance slider has a lot more effect in the vase, than in the background. See crops. Are you sure, that there is not some texture in the background? I set the “bokeh” slider to 0. On the other hand, you have seen it with a lot of different images, and the image of the squirrel shows your problem as well, to a lesser extent. With the Noise Model slider you can suppress noise, but lose detail. I keep it at 0.
I’ll upload a few crops of the images that I shot. You can judge them yourself.

I think, that I don’t see your problem with the luminance slider. I do see, that DP-XD is sometimes inferior to DP.
If there is anything different in our RAW files, than it can be in the camera settings. Your images are exposed correctly, mine are always approx. 1 stop too dark. So I have to correct in PL. The reason is the “protect highlights” setting, if it is on, than the camera effectively under exposes by 1EV. That shouldn’t affect the RAW file, of course, but I am beyond logical reasons. Another setting: I have set high ISO noise reduction to “off”. This will have influence.
Last one: you could update the firmware of your camera, but I don’t expect anything of it.


To be continued, I’m afraid.

2 Likes

Thank you for supporting this “investigation”. From the crops you posted regarding the ISO12800 image, actually I see exactly what I am seeing on my machine. Perhaps I did not explain it clearly but what I mean, and what can be seen in your crops as well, is that from luminance=0 to luminance=100 there is too small a difference, way too small, almost none in the out of focus areas. If you do the same with DP classic, you’d go from very noisy to completely smeared out, with an optimal (subjective) setting somewhere in between. The out of focus areas are not particuarly smooth to begin with in that picture that was shot in relatively bad light, however DP classic manages to handle them fine or at least more gracefully. And the same problems are apparent in the ISO5000 image that is reasonably well-exposed and should have no issues. The ISO1600 sample is quite clean, sharp. Still on the out-of-focus area you can see a glimpse of the “unusual” behavior even if DPXD is usable at that sensitivity (albeit with very little or no gains, with this camera and current implementation/model parameters). Basically, it is not able to obtain the same compromise of DeepPrime where noise is effectively removed from the “bokeh” areas and detail is retained where fine textures and features are present. In the majority of high-iso (>1600) images I am unable to obtain good results unless when aiming for a very “smeared out” look by pushing the noise model slider all the way to the left. In that condition DP and DPXD behave more similarly.

I would exclude that camera settings (I mean, menu settings, naturally exposure, can play a role) are relavant since even the “Noise reduction” setting does not affect raw (I have tried, it does not make a difference). The samples above were shot with firmware 1.5, I have since upgraded to 1.6 but I’m skeptical that it makes a difference…

The only clue that I have is that there is some degree of dependence on the particular subject and the “quality” of the pictures. Images that are rather crisp, relatively well-lit scenes with good micro contrast are handled “decently” but still not better than DeepPrime, and with the same flaws, albeit more subtle. For instance this sample:

Sample1

On the contrary, images that are already soft to begin with are handled very badly, such as this one, which shows almost the same behavior of my (bad) ISO12800 sample even al much lower iso:

Digital Cameras, Pentax K-3 Mark III Digital Camera Test Image (imaging-resource.com)

Well, to be continued…

Hi Simone,

had a look at the referred pics … and yes, they have to be really sharp to profit from DeepPrime XD (like the cats mouth). – Sharp means precise focus and no camera shake. Do not rely on the reciprocal formula (e.g. 200 mm = 1/200 sec). For that focal length and critical sharpness use 1/400 sec minimum, provided your subject doesn’t move too.

While I don’t know your cam, do not overestimate its low light capability. DeepPrime and XD can turn unusable pics (camera noise wise) in usable ones, but they do not wonders. – Now and then I take concert pics with a Nikon D750 in really bad light and have to use high ISO, but so far I’m happy with XD. :slight_smile:

anyway, have fun

I didn’t check the 2 samples yet, that you mention in your last post. The 2nd one has no metadata available, I don’t know at what ISO the image was shot.
But some thoughts: maybe we expect too much. The Noise Model slider has been introduced in PL6, in DP and DP-XD. Suppose, that the setting NM=0 in DP is identical with DP in PL5. If I compare the results of your 12.800 ISO image processed with the settings DeepPrime, lum40 or lum100 (noise model=0) and DP-XD, lum40 or lum 100 (noise model= -100), then I probably prefer the DP-XD result. (Note: I suppose, that the texture that I see in the DP-XD result in the background actually is present and not an artifact of the processing). It seems, that the Extra Detail that is promised with the introduction of DP-XD is strongly dependent on the Noise Model slider, and that with high ISO images you can’t afford a high setting.
I tried once more with an ISO800 image, and I definitely prefer the DP-XD result (settings for DP and DP-XD: lum 10, NM 0). And the result with NM+100 is even better.
Next question: is the difference between DP and DP-XD large enough to pay the upgrade? I doubt it.

At ISO800 and below I find DPXD generally valuable too even on the K-3iii it works well and improves detail while not introducing artifacts and noise in the OOF areas, as promised. The problem is at higher ISO. There should be no discernable “texture” in the OOF areas of the 12800 image, as bad as it is in terms of lighting, it was made on purpose to exacerbate the problem. DP renders it naturally, as it does with the other OOF areas of the two more meaningful and real-life samples. All in all, I am still firmly convinced that the lack of sensitivity to the luminance slider is a clear sign of “suboptimal” or “abnormal” handling of files from the K-3iii. There should be a much more pronounced difference, orders of magnitude more pronounced. I am not seeing this behavior with other cameras (like Fuji XF 10, the old K-3, which I both own, and some generic sample downloaded here and there from other cameras). IMO that’s a smoking gun signaling that “something is wrong”; not necessarily wrong, but at least different from how it is expected to behave on the majority of cameras. If I was a developer working on the algorithm, I would consider the case worthy of further investigation. “Edge cases” are always instructive to understand possible bugs or to put it more mildly “lack of generality” of the implementation. Naturally I understand both the fact that one should not expect miracles, and that it could be practically impossible to have the algorithm perform at 100% of its potential on every camera in existence, unless an unrealistically massive fine-tuning effort is put in place by the DxO team. Unfortunately, the K-3iii seems to be one of the cameras that benefits less (if at all) from DPXD (and for DP as well, but that is more subtle and much harder to point out. I keep it as a strongly personal opinion, whereas the abnormal behavior with DPXD is much easier to spot and to prove).

Also, in general, the “noise model” slider is a useful addition and allows to have better control of the detail retention vs. grain removal compromise. It is useful even for standard DeeoPrime. Is it worth the upgrade? That’s subjective. For the moment I’m holding off. Since DPXD is new and there are other problems being reported with different cameras (different kind of problems) I expect a new and improved version with the next release that maybe will coincidentally fix the issues with the Pentax as well…

1 Like

I completely agree. After half a life of working in R&D as a product developer of printing systems, I know that no matter how much effort you put in testing your product, it is never enough. Some bugs will be found in practical use only.
So I’ll wait and hope that a solution will be found.

1 Like

Forgot to ask: did you submit a ticket with this bug?
If not, I will do so.

Hi, no, I didn’t. Was hoping for some response from DxO on the forums, but maybe opening a ticket is a good idea, if you want ! Thanks.

I sent a message to Svetlana, hope she’ll respond.

In the meantime I have a different problem: I decided to buy PL6, mainly to re-edit some older K5 files. After the latest release (PL6.1) the preview window doesn’t update anymore, e.g. if you change the WB or NR. Making the application more or less useless. I already deleted the 6.0 installer, so I can’t go back to the older version… Fortunately, it has been recognized as a bug already, I hope it will be solved quickly.

1 Like

Thanks for the info…actually I have just opened a ticket today because I noticed that optical corrections for my Pentax cameras stopped working in terms of preview, while working fine at the export stage. Now you made me check also other sliders such as WB and they are not working ! So it’s a “big” bug that will probably be addressed quickly. Do you have any link where they talk about this bug ?
It’s really strange because auto optical corrections still worked with other cameras so I thought it was something specific to some lens correction profiles and more “subtle” to troubleshoot…
I did not noticed that other sliders (which I rarely use) stopped working as well…

This is the link in which I participated: Bug in PL6.1: WB and Color Rendering don't work on exported DNG(NR and Optical corrections only).
But there are more.

1 Like

For NR the big preview only shows HQ, that is, you have to check NR in the small preview !

To get a quick(er) preview update you can try
grafik
( in Windows )

Otherwise for camera related things I can’t help you
– also not talking about the DNG stuff.

1 Like