Is it finally possible to have the XMP-files as the metadata master?

BHAYT · October 8, 2022, 11:41pm

@Stenis With AS(ON), the original name for the ‘Metadata Synchronization’ and the way that the feature is described with the warning prompt when you select the box, so I use AS(ON) and AS(OFF) to describe the settings, the metadata is read from the image (embedded or sidecar).

and this was originally true for AS(ON) and AS(OFF) in the first discovery case.

But as of PL5.3.0 that was changed for first discovery so that from PL5.3.0 onwards

First Discovery:-

With AS(ON) the metadata is taken from the image when an image is first discovered and entered onto the PLDb.
With AS(OFF) the metadata is taken from the DOP and the existence of a PL5 DOP effectively blocks most if not all metadata (to be confirmed by DxO @sgospodarenko) coming from the image. Blank or non-existent fields in the DOP will effectively result in blank fields in the image, regardless of what is actually held in the image. (Note that this is not or should not be true for earlier DOPs, i.e. PL2, PL3 and PL4 where the DOP would only ever contain ‘Rank’ (‘Rating’ and ‘Rotation’) so for those DOPs PL5 and PL6 should take the remaining data from the image).

Metadata Changes:-

With AS(ON) any changes made to the metadata in another applocation are generally detected by PL5 and PL5 propagates any metadata changes made in PL5 to the image. As you have discovered!

BUT with the AS(ON) feature comes a sting in the tail, PL5 also writes back the metadata to the image but with keywords in its own format and this may well be an unwanted side-effect.

With AS(OFF) PL5 detects changes made externally to the metadata and sets the ‘S’ icon and the user can then ‘Read from image’ to fetch the metadata updates made externally, if desired. Any changes made in PL5 will remain there until a ‘Write to image’ is executed and PL5/6 will not signal the differences with an ‘S’ icon.

If you don’t mind DxPL reformatting your keywords then set AS(ON) and the keywords will be changed in the image to the same as they will be set in any export and you will have all the benefits of the automatic exchange process (and it is good but …).

The very long post that I referenced was an attempt to de-mystify the whole issue of formatting with hierarchical keywords and propose a strategy whereby DxO could emulate the formatting of any other package, both for writing back to the image and for export, but no-one is interested @Musashi.

Stenis · October 9, 2022, 12:02am

@BHAYT

Bryan you are absolutely right and I discovered the problems with structured keywords too in version 5. I have no idea if they have corrected that now but for ny sake it works since I long ago decided to keep it simple and use a flat structure instead. It has worked perfectly fine for me since I started to integrate PM and PL.

… and from what I can see there are no way to import and export vocabularies either like in Photo Mechanic. So from what I can see the only migration way is that Photolab reads the keywords and build it´s own keyword list from what it finds in the images. That works really well with my flat structure but is probably not so straight forward if you have a hierarchy.

BHAYT · October 9, 2022, 12:14am

@Stenis I am sorry but simple keywords don’t help!

This is the image after adding a, m and b to the existing keyword in PM

and this is what happens after PL6 has read and written the metadata back to the image

PL6 (and PL5) write the simple keys to the hierarchical fields as well. Does this really matter …???

danielfrimley · October 9, 2022, 8:21am

May I please ask how you achieve this in PM Plus? Is it a setting somewhere or is it normal behaviour. I’ve been using PL and PM Plus and haven’t found a way where I’m truly confident that I can ensure that PM Plus has total ownership of all metadata changes - I have trouble understanding the wording of the process in both DxO and CameraBits manuals

Joanna · October 9, 2022, 8:35am

The simple answer is, as has been mentioned before, disable any metadata synchronisation in PL, ignore PL’s “out of sync” warnings, and rely solely on PM, or other DAM, to do all your metadata work.

This way, you can even use hierarchical keywords as you wil be both creating and modifying them in the same place. Otherwise, PL will attempt to rewrite your keywords in a different manner, which could then confuse your DAM.

This includes not using PL ratings or colour labels, since even just writing those to your files can cause all metadata to be revised.

danielfrimley · October 9, 2022, 9:32am

Thank you - I know it’s been discussed at length in various places on the forum though I still find it hard to get my head round it.

Let me explain, my settings in PL are as follows:

I import photos from my SD Card and make all my metadata changes in PM Plus to the original OOC Leica DNG (RAW) file. I have no interest in how either package handles changes to TIFF/ JPEG images. No extra XMP files are created, so all metadata must be embedded within the original RAW container.
I edit the image in PL and a DOP file is created which contains those edits and some metadata. I have no interest in what metadata PL stores in either the DOP or the DB as long as PL makes no changes to what I’ve done already in PM, I make no metadata changes in PL and I get PL to display what it can (I’m not precious about how it interprets what it gets) by using “Read from Image”. I do not use “Write to Image”

From my research, I understand this should achieve what I want - metadata created and updated in PM (my single source of truth etc. etc.), embedded in the RAW file and unaltered by PL in any way whatsoever (in theory, if I hash the RAW file after it comes out of PM and once again after it comes out of PL it should be the same value - admittedly I’ve not tested this yet)

What confuses me is this part in the settings:

I have it ticked this way as it seems not to affect the metadata but the wording causes me a problem. Does this only relate to the edits and the reading and writing of DOP files (i.e. separate files) and the DB, or does it also include the embedded RAW data (commonly referred to as XMP sidecars) - if the latter, how does it differ from “sync”?

danielfrimley · October 9, 2022, 9:56am

I’ve answered by own question, apologies I should have done this before I posted but I’ll leave it here in case in helps someone else.

BHAYT · October 9, 2022, 10:04am

@danielfrimley Correct you are preventing DxPL from reading and writing the metadata automatically and controlling the input manually and preventing any write-back automatically and manually!

Because your RAW file is actually a DNG (container) the “rules” for DNG metadata is embedded in the DNG not in a sidecar file.

This is the DOP and if you don’t set both you will not preserve and restore your edits, arguably those edits are what you bought DxPL for in the first place!

But the DOP will also contain the PL5 representation of your metadata as well as your image edits this typically does not matter in normal circumstances!

But please note what I wrote to @Stenis about the PL5.3.0 changes to DOP handling with respect to DOP data, i.e. with AS(OFF), your setting, then any new discovery of an image with a PL5 DOP then the metadata will come from that DOP and if there is any discrepancy between the values in that DOP (which will become the values in the database) and the values in the image metadata YOU WILL NOT, REPEAT WILL NOT, BE TOLD BY DxPL regardless of the fact that there might be PM changes that post date the date of the DOP, the later metadata will not be read and no ‘S’ icon will be created during this “discovery” (import).

So follow the discovery process with a ‘Read from image’ immediately after the discovery to ensure that the metadata is up-to-date with the image metadata

BHAYT · October 9, 2022, 11:27am

Will cause all metadata to be changed and keywords to be reformatted as per the PL5/PL6 “rules” which changed on release PL5.2.0. It must be stated that PL5 using ‘hr’ keywords for simple keywords is not unique to DxPL as the spreadsheet shows

Photo Mechanic is the only one of ‘hr’ aware packages packages I tested, that does not place simple keywords into the ‘hr’ fields but I feel its handling of hierarchical keywords is too limited and shuns any hierarchical keyword elements being placed into the ‘dc’ fields!

This is what PL5 (and I believe what PL6) does to the keywords “lovingly” formatted in a wide array of different interpretations of what hierarchical keywords, in particular, should look like

Joanna · October 9, 2022, 11:27am

But if I may just clarify… The metadata may not be written to an XMP file, but it is certainly written to the DOP sidecar, as well as the PL database, thus ignoring SPOD rules.

But once PL has read from the DNG, it then writes its own interpretation to a DOP sidecar and/or the database and ignores anything that was in the original DNG file, constructing its own version of reality which, from then on, has priority.

Unfortunately, neither PM not PL adhere totally to the common MWG guidelines, so, allowing more than one of them to write metadata is to invite chaos, unless you deliberately and religiously adhere to MWG guidelines the entering metadata.

BHAYT · October 9, 2022, 11:38am

As does every other package as far as I can tell so they are all as “good” or “bad” as each other.

Or persuade one of them to use my keyword formatting strategy, which I presume @joanna that you are building into your own utility as I am building into my Python program!?

None of us are going to abandon our favourite software, particularly given the current price of most of them. I purchased IMatch at Christmas at a bargain price but prefer DxPL for its simplicity, or rather preferred it up to the change in PL5.2.0! But with my own utility I can create whatever format my heart desires and fix whatever “rubbish” DxPL or any other package “spits out”.

Given that all the major packages use an SQLite database, DxPL has gone one step further by preserving its edits in a proprietary file that also preserves the Virtual Copy metadata until PL5.3.0 and now also uses it as a source of data for the [M]aster as well.

I will fix the second spreadsheet later today!

Joanna · October 9, 2022, 11:38am

Using my app, which strictly follows the MWG guidelines, gives…

[XMP]           Subject                         : a, m, b, x, y, z
[XMP]           Hierarchical Subject            : a, a|m, a|m|b

… where only keywords that participate in a hierarchy are mentioned in lr:hierarchicalSubject and all keywords mentioned in lr:hierarchicalSubject are written, flattened, to xmp:subject

Files marked according to this standard can be read from every app I have tried it with.

BHAYT · October 9, 2022, 11:41am

@Joanna Please see my reply just before yours and is your utility freely available on the Mac and I know it isn’t available on Windows and although it may be the most conformant if I don’t like what it does I have no alternative to modify the layout, i.e. you are better than the others by having better compliance and as bad as all of them for not providing any (or many) alternatives!

We need the ability to interwork, not simply replace, which is what my design does!

OXiDant · October 9, 2022, 11:45am

Argh my head hurts.

For most people who arn’t using PM but a cheap one like adobe bridge.
1 i can’t leave bridge yet because there interface is having advantages like templates and file extention filters.
2 it seems that bridge and dxo writing different in a xmp. Which isn’t a big deal IF the READ it the same!
3 But that seems to be not the case.
4 Plv5 did auto update metadata on selecting the folder or by command refresh metadata. So uncheck sync does only prevent writing without your permission in the xmp…
5 plv6 should be working the same right?

The only thing i do inside PL is add somethings in the iptc like gps data from google, edit comments. If i need to change keywords i return to bridge.
So i assume i am safe syncing xmp as long as restrain myself to change keywords inside PL.

Correct?

BHAYT · October 9, 2022, 11:48am

Any metadata change will cause PL5 and PL6 to have a “desire” to write the metadata back but that “desire” is controlled by the option. If it is not selected DxPL won’t write back to the image but will write the updates to the database and the DOP and to any export.

PS:-

If you mix a non-‘hr’ aware program with an ‘hr’ aware program all manner of problems can follow if the ‘hr’ aware program gets to write to the image. Data will be written to the ‘hr’ fields. If you then delete in Bridge the data will be removed from the ‘dc’ fields but left in the ‘hr’ fields and the ‘hr’ aware programs will start re-writing it into the ‘dc’ fields, potentially!!??

OXiDant · October 9, 2022, 11:51am

Yes so i can’t transport this update in a folder. Xmp isn’t update by migration.
I can selectvand manual provoke a writing action to the xmp.
This is what i was doing lately.
Downside is you have a great change to forget this action after editing a folder which doens’t show up in dxopl because it’s running on the database for that.

BHAYT · October 9, 2022, 11:59am

@OXiDant Agreed which is why I have been moaning to DxO to “finish” the ‘Conflict Resolution’ @Musashi it should in my opinion (not humble) be finished as follows

Signal change to image with ‘S’ - completed
Signal change to DxPL with ‘S’ - missing
On new discovery with AS(OFF) when metadata will be taken from DOP signal with ‘S’ - absolutely missing! Use of date timestamp use optional
Use the heads of the ‘S’ to indicate in which direction the mismatch occurs (could be both!) - optional but would round out the product nicely!

Chance of getting this relatively straightforward task …

OXiDant · October 9, 2022, 12:00pm

This exactly is my fear.
Does it write only the changes which is safe or does it behaves as a “Master” and overwrite all old data even when it’s partly the same.
And does it see timestamps of change and act acoordingly?

BHAYT · October 9, 2022, 12:04pm

[quote=“OXiDant, post:29, topic:28657”]
Does it write only the changes which is safe or does it behaves as a “Master” and overwrite all old data even when it’s partly the same.
And does it see timestamps of change and act accordingly?
[/quote

@OXiDant All as far as I can tell! The database in DXPLDb has no knowledge of what cam from the original image nor what has been added in DxPL, a keyword is simple or part of an hierarchy and will be output (back to the image metadata or in an export) as the database and the DxPL rules dictate.

If you can seen the second spreadsheet you will see what DxPL does to Bridge keywords.

DxPL “hangs” on an operating system construct that alerts it when change occur. DxPL then checks the “Date Modified” against some internal “clock” and accepts or ignores the event as it sees fit!

Joanna · October 9, 2022, 12:06pm

It will be available, not freely but for a reasonable price, as soon as I have gotten rid of some bugs (unlike some software writers, I am not going to release buggy software)

As you can tell now, with alternatives comes confusion. The reason why we have the metadata nightmare is simply because everybody seems to want to do their own thing, thus forcing all other software writers to write reams of conditional code “just in case” someone does something stupid.

There are only five characters that have to escaped when writing keywords to XMP file…

original	escaped
"	`"`
’	`'`
<	`<`
>	`>`
&	`&`

Apart from these, all other characters are considered valid as part of a keyword, including spaces and other punctuation.

So, a|m|b is a valid, single, keyword that just happens to contain the ‘|’ character.

It is only when certain software decides it wants to interpret such a keyword as having hierarchical structure that is can be used as such. But not all software does that and so, in order to be compatible with as many apps as possible. I only read and write without “interpretation”.

If I were to type a|m|b into my app’s entry field, it would automatically be written as above and you would only see the three separate letters in the entry field, with the hierarchy being written to the file’s metadata behind the scenes.

Addenda

The <, >, and & characters can be written directly to an image file, as long as they are surrounded by either single or double quotes. Whichever quote character is not used to enclose the others can also be used - single within double or double within single.

This allows for keywords like Black & White or L'art de l'eau