PL 5.2 Resolve Metadata Conflicts: Sidecar vs Image File

DxO PhotoLab
1

When I tested the new metadata conflict resolution, I noticed the following:

  1. MD in XMP sidecar takes precedence over MD in image files. MD that was changed in the file is simply ignored.
  2. New Conflict: When metadata is changed in the image file (e.g. star ratings written by Canon DPP- and XMP sidecars are absent) the update arrows appear and MD can be read from the image file(s) or “overwritten” by DPL. This leads to a discrepancy between MD contained in the image file and MD contained in the XMP sidecar.

Proposal

  • Conflict resolution should include MD in XMP sidecars plus MD in original files
    (I still prefer DPL to NOT write MD to image files though)
1 Like
2

Hello!

Let me ask @Musashi to have a look at this proposal.

Regards,
Svetlana G.

1 Like
3

Please include me in on this discussion. Apart from for beta testing, I write MD exclusively to RAW (no XMP files) and really need a way of reconciling that.

4

Good morning,
Let me check some things and I’ll get back to you regarding this topic.

Regards

5

Hello everyone

Let me join this discussion as well to clarify part of your statements, just to have a better understanding of your requests

and

Can you be more specific when stating that you’re writing metadata to your RAW image:
1- Do you mean the source RAW file itself?
2- If so, what application do you use for that?
3- Could you show us actual examples?

Thanks in advance for your help :slight_smile:

Best regards,
Captain

6

Output of “exiftool -G -a -u file.ext” for the rest of us to check out:
EOS 5D: 20071103-1571.cr2-MD.txt (36.5 KB)
EOS 5D3: 20140609-8866.cr2-MD.txt (45.8 KB)

Workflow that extracts the tags: List Metadata.workflow.zip (120.4 KB)
Put the (unzipped) workflow into /Users/*****/Library/Services/
A context-click on a folder of images will then write the respective text files

7

Yes

I use the keywording, rating and tagging app that I have been perfecting for the past couple of years. It leverages ExifTool, which I have found to be 100% reliable at preserving the integrity of RAW files. In three years of using it, I have never had a corrupted RAW file.

I’m not sure what you mean by examples. Could you clarify?

My app allows users to interact with metadata in the RAW file, an XMP. sidecar or both.

Capture d’écran 2022-04-13 à 10.28.27

When reading, this preference is checked and the appropriate source is read from. If both options are selected, metadata is read from the XMP sidecar, falling back to the RAW if no sidecar exists.

When writing, if both options are checked, the metadata is written to both, to keep them in sync; otherwise, it is only written to the currently selected option. So it is possible for the two to become out of sync if the user decides to change the reading/writing options after having already keyworded/rated/tagged files.

This discussion now makes it clear that I need to add a reconciliation dialog, which will allow the user to compare the RAW and XMP and select which items need updating in which file.

As a Nikon user, I need to bring up the fact that the D850 (and possibly other models) now provide an option to write the xmp:Rating tag directly from within the camera.

Furthermore, Nikon’s own NX Studio app creates its own NikonApp section in the metadata of the NEF file and stores/maintains all sorts of metadata, including keywords and ratings, there; as well as providing the option to write image editing adjustments directly instead of to a sidecar. So, they obviously have no concerns about editing metadata in a RAW file.

Addenda - I just saw @platypus ExifTool dumps and note with interest that Canon apparently take the same approach to writing metadata and their own editing adjustments directly to the RAW file.

It is becoming more obvious that the idea of a RAW file being inviolable no longer applies and that PL might have to take into account metadata changes and the possibility of reconciling metadata to the RAW file as well as from it.

1 Like
8

If we look at the current state of things, we see that metadata (stars/labels/keywords/…) can (possibly) come up in

  1. application specific sidecars (.dop and others)
  2. .xmp sidecars
  3. the raw file itself, written by the camera or camera specific SW like DPP, NX-2 etc.
9

Oh fun :roll_eyes: :crazy_face:

My personal feeling is that there is now an implicit “obligation” to reconcile “industry standard” metadata from RAW files with XMP sidecars.

What I am not so sure about is any obligation to handle anything from manufacturer provided metadata, whether that be in their own metadata section in a RAW file (NikonApp, CanonVRD, etc), or in proprietary sidecar files/folders.

e.g. from what you have shared, Canon software writes Rating to the xmp:Rating tag but Nikon writes it to NikonApp:Rating, which is not accessible outside of Nikon’s own software.

Addenda

Definitely do not rely on stuff Nikon puts in the NikonApp section in the RAW file, I have just found it duplicates NikonApp:Rating but with two different values.

10

Ok, that clarifies the situation indeed.

One thing that I have to remind here is that DxO, as many other companies in the field of Photo editing application, historically follows the conservative rule of respecting the original RAW files, meaning not modifying it in any ways.
The blatant counter-exemple being the DNG format. :smiley:

There’s a big question here, at the borders of ethic, which we’ll have to consider.
Besides flirting with any possible danger of corrupting an original file, though you did some extensive testing @Joanna, leaving this to individuals isn’t taking the responsability as a company. I’m sure you’ll understand the difference here. :wink:

Anyways, It’s a totally new approach that we’ll have to discuss internally, and see how we can answer to the questions of “how we can handle it?”, “Do we have to handle it?”, balancing the cost to benefit and on top of that, seeing how many people are actually concerned by such features in their day to day workflow.
And this, of course, comes on top of aaaaaaaaaall the features that we already have in our backlog and that may have a greater priority.

But at least now, it’s something we’ll also keep in mind. :slight_smile:

11

I fully understand DxO’s reluctance to write into raw files…but for metadata reconciliation, DPL should read from the raw files as well as from associated sidecars and offer the user a choice.

All we do in life is handle the consequences of choices we took or others made for us - the latter often being more difficult to bear, at least if you’re grown-up :wink:

1 Like
12

Hi @CaptainPO

Like @Joanna and @platypus I have written MD (keywords, description, title, city, country, …) directly in RAWs since years, and all that without any issue with any software, and of course especially DxO Optics Pro and PhotoLab.
I’ve done that with Nikon View NX, NX2, Aperture, Photo Supreme.
For this last one, it allows to choose to put them in separate xmp or include in Raw directly.
This may be an option also for you in Preferences…

13

You use the word “many” but, as is the case with manufacturers own software, this appears to not be a problem.

I have always felt that the inviolability of RAW files was always intended to refer to the image data, which is maintained in a totally separate block from any metadata. What I see is that a RAW file is akin to a Mac app bundle, which looks like a file on the outside but, on the inside, is a collection of folders and files. For example I can quite happily go into an app bundle and change the localisation strings for a part of the UI without touching the compiled code.

Well, yes and no. I have written my app to allow the user the choice but it is up to the user to decide whether they consider writing to RAW files is a risk or not. Simply provide a disclaimer at the point of choice, stating that, from extensive testing, writing to RAW files has been proven to be safe and that camera manufacturers do it all the time.

Absolutely. In which case, if DxO is that reluctant to allow users their right to the free choice of writing to RAW files (note the emotive language :wink:) then they should at least allow for detection of external changes to RAW files as well as XMP files.

That reminds me of the restaurant I went to a couple of months ago where, when I asked for decaf coffee, he said they no longer served it because he had read that it contained more caffeine than regular coffee, which would be bad for me. Since he is not a doctor and I know what regular coffee does to me, I will no longer be going to his restaurant (which is a shame because the food can be good)

My thoughts exactly. Then we can choose whether we want decaf to write to RAW files or not

1 Like
14

As a side note, allow me to describe what my app does to facilitate searching, which seems to me to be the most important reason why we would add metadata.

The only “database” my app contains is a dictionary of keywords and their hierarchical relationships. I don’t maintain an internal record of keywords written to files because macOS does that for me with its own metadata indexing and Spotlight searching mechanism. Thus, I never need to reconcile files with a database or vice versa. Searching in my app is a simple matter of writing a predicate in code and submitting it to the Spotlight mechanism. Selecting a file in the browser automatically (and quickly) reads the metadata directly from the file (RAW or XMP) and, should that change externally, all that is needed is to re-read the file directly.

Writing to RAW files on a Mac has other advantages in that I can use the Spotlight mechanism in Finder to execute the same search as I would from my app, which then gives me a list of all the images, which can be previewed directly without having to find the XMP files that contain the keywords and then have to further search for matching RAW files.

I haven’t implemented it yet but I am looking at being able to load Saved Searches, created by Finder, into my app, as well as being able to write such searches directly from my app. All with no database.

Unfortunately, I gather that Windows was very late coming to searching by metadata party and I can understand why DxO had to use a database in the early days, to provide a common solution.

And I really can’t avoid bringing up the subject of SPOD, which DxO seems to be determined to ignore with an ever increasing number of possible places where metadata can be found…

  1. in a RAW file
  2. in the DOP file
  3. in the database
  4. in an XMP file

Is it any wonder that reconciliation is becoming recon-silly-ation? :crazy_face: :stuck_out_tongue_winking_eye:

15

Hello Pathal,
I am also a PSU user and based on a statement from PSU this only applies to a few “heritage” raw files. For all newer raw files it is xmp-only.

Sigi

16

Single point of definition is a concept rather than a reality, unless the (informed) user makes it one - which will make the user the single point of definition.

As THE single point of definition of the keywords I add to all my photos, I have to

  • decide, which preferred tool I want to write the keywords with
  • find out, how multiple apps interoperate, if I should choose to use more than one tool

Imo, DxO should strive to offer possibilities rather than limitations - without breaking too many dishes on its way to be that solution provider. I’ll try my best to help DxO get there.

17

Totally agreed. As I’ve mentioned before, if there’s anything I can do to help DxO improve my all-time favourite image editing software, they only have to ask.

18

You’re right @Sigi , and for my Sony RX100 III allows only xmp.
Nevertheless I’m not sure that Nikon Z7 is an “heritage” raw file…

19

I found now the file extension where Photo Supreme will store data in the file:
NEF
NRW
TIF (RAW versions some Canon older cams)
ORF
CR2
PEF
Based on a comment from the developer this was/is possible for “historic” reasons. All other raw file formats it is xmp-only

Sigi

20

Hi,

I am surprised that the fact that some (many ?) of us (user of PhotoLab) use Exiftool (very reliable tool used around the world) to write metada directly to raw files was not on your radar until now :thinking:

@Joanna can we say your app is an (advanced) GUI for Exiftool ?
I mean Exiftool is the code dealing with reading / writing files. And thus makes your App pretty safe to use since Exiftool is known to be reliable.
I do not see a problem for DxO to use Exiftool and give some credit and a little bit of chocolate to the author of this code…