Is it finally possible to have the XMP-files as the metadata master?

[Stenis (Sten-Åke Sändh (Sony, Win 11, PL 6, CO 16, PM Plus 6, XnView))]

So then in the old days hat happened if you turned of the .DOP for the settings and there wasn´t even a database, what happened then if you had a RAW? Where did the editing metadata go then?

If you want to collaborate I guess you can always use a DNG can´t you - if you don´t prefer an old fashion TIFF? .DOP-files seem to be one file to many today. The first RAW-converter I had in 2005 used sidecars (Raw Shooter Premium). They are a risk to get lost in file transactions if you don´t use software like Photo Mechanic which can handle both XMP and DOP at transfers. I guess that´s why DAM-systems in the industry mostly uses JPEG-files for the images since then there is no need at all for sidecars.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Stenis I am sorry but I haven’t a clue what you are talking about or where the stuff about PL6 comes from!?

There are no additional options that I can see and I just ran a test on PL6 that went like this

1. Take a previous test directory that I was going to use where two JPG images had keywords a, b and m set and that had been discovered in PL6 and for which DOPs had been created.
2. Copied the directory contents to a new directory.
3. Changed the keywords to new values in Bridge
4. Discovered the directory in PL6
5. Checked the values in PL6
6. Result = values from DOP (a, m, b) NOT from JPG.
7. “Classic” post PL5.3.0 behaviour.

Keywords in images:-

Keywords from PL6:-

Please note that these are “discoveries” where the DOP is out of sync with the image metadata and PL5 shows nothing, far from taking its lead from the image it is taking its lead from the DOP and ignoring any potential mismatch, that I cannot “forgive”, it is “amateurish” in the extreme @Musashi.

The changes made in PL5.3.0 were ill conceived but easy to implement, essentially a reversion to PL4 handling but with a LOT more metadata. Conflict Resolution is incomplete as I stated earlier and the user is expected to remember what may or may not be the latest metadata, DOP or xmp, xmp or DOP, or a mixture of both or, albeit executing a ‘Read from Image’ after a DOP discovery is a simple “fix”.

The DOP provides a security mechanism for edits and pre PL5 some metadata found its way into the DOP. Pre PL5.3.0 the metadata for the [M]aster was written to the DOP but never used, only the metadata from the Virtual Copies was used.

Some users complained about the change, totally inadequate documentation caused users to rely on the DOP and not capture sidecars etc. and DxO did a U turn, with no warning, with no option and with minimal actual complaints from the customers as far as I can tell.

That mechanism still remains, as far as I can tell. The system is perfectly adequate, the PL5.2.0 change made the hierarchical keyword handling less satisfactory than it had been but other than that I do not understand your desire/concern.

I simply do not understand what this concern about “master metadata source” is about or even what it really means but as far as I can tell (and I might be wrong) nothing has changed with PL6 except some IPTC fields and Exif fields about which I know nothing and that is as far as it seems to go.

I have switched to PL6 but all the research I did on PL5 seems to be holding good, so far.

One thing that did happen was that I decided to copy metadata from a JPG to a RAW and when I went to the paste the menu offered was as follows @sgospodarenko

There are missing options - this was copying from the last JPG to the first RAW

But when copying from the last but one JPG to the first RAW the options are all present!?

@Stenis

I tried to update it and had RAW selected. When I updated it the sync system in PL immediately updated both the RAW and the JPEG. When I deleted a text i the Header element/field the data was deleted also in both files in Photolab. A “bravo” for that because I have seen other DAM-systems having problems with that. I start to be pretty confident that the sync in PL 6 will really work for me.

This sounds more like PM doing this than PL doing it! As far as I know PL does not understand any “link” between images in the same way as some other software does. I will test this tomorrow, sorry later today!!

[OXiDant (Peter)]

@BHAYT

@OXiDant I am sorry I lied, not deliberately!

No problem.
This matter and behaviour between keyword,iptc managers is that complex that it’s not posible to be sure how it works before testing and even then your never sure your not missing something.
My present workflow is.

• Ingest by sdcard and panasonic’s software. Close that application.(i don’t use it for other things)
• Win explorer, rename folder and take out oocjpegs who are twins of the rawfile (some are made by panoramamodes or something else so those are unique. Separate the video files.
• open FastRawViewer and start culling, rating and now labeling a color. (it creates a xmpfile)
• open bridge and use my template to fill iptc, and select keywords.
• then dxopl open for editing and adding iptc data along the way.
  Gps, image comment as in what where how and reminders as select for B&W and such.
• (edit) throwaway the oocjpgs because i don’t need those other then fast viewing purposes.

Adding a forgotten keyword means reopen bridge and add it in there.

As i am reading your comment correct your saying following the workflow above means it’s safe to activate autosyncronise xmp file.
The misbehaving of dxo database is mostly in the way it writes keywords in the xmp-file. Right?
In iptc fields it’s maybe guessing if it’s the same field as in bridge named, because i use the dutch version inthere and dxo is englisch, but that’s easy to check by keeping both open to see what’s changed.

Edit,
I haven’d looked closely to the new user interface of keyword exif and iptc manager of dxoplv6 yet.
But i missed template’s, the presets so to speak.
And the keyword selector screen was a bit ockwards in the earlier versions so quick multi image keywording wassn’t that easy as in Bridge.
Then again searching images in dxopl is much better then the folder restricted search in Bridge.

So if dxo gets it’s interface commands up to date and create a xmp iptc managing window which holds presets, templates and even better if they can create a history line of 5 changes per metadata i don’t need Bridge anymore.
Resume:
1 possible lock for changes outside dxo on the xmp.
Simple question xmp is changed outside dxopl update yes/no?
(very important in case of not sure what’s happening is correct to prevent damaging of your xmpdata.)
2 template, preset manager. Partial pasting ofcoarse.
Option a overwrite everything with this data.
Option b only write in empty fields
Option c manual selecting which fields to overwrite.
3 create a 3 tab windowsystem.
Library
Meta data manager
Editor

Keeps the overview much simpler if library is just a search and view function with some commands to change iptc data on 1 file per action.
Now Bridge is my third tab window “Meta data manager”

[Joanna (Joanna Carter)]

@Stenis

So then in the old days hat happened if you turned of the .DOP for the settings and there wasn´t even a database

That has never been the case. There has always been a database.

@Stenis

what happened then if you had a RAW? Where did the editing metadata go then?

You are mixing metadata with image editing data.

Any changes made to an image has always been written to the database and, additionally/optionally, to a DOP sidecar file.

Metadata was only introduced with PL5 and was written to the database and, optionally, DOP files, if they were already being used for image editing data. It was at this point that PL started to use XMP sidecar files for metadata, but also wrote them to the database as well as DOP files.

@Stenis

If you want to collaborate I guess you can always use a DNG can´t you - if you don´t prefer an old fashion TIFF?

How, when image editing data isn’t part of the XMP/EXIF data standards and there is no way to store image edits?

@Stenis

The first RAW-converter I had in 2005 used sidecars (Raw Shooter Premium). They are a risk to get lost in file transactions if you don´t use software like Photo Mechanic which can handle both XMP and DOP at transfers

PL users have been using DOP sidecars for years and, if you use PL to move files, the image, the DOP and the XMP files get moved at the same time. Using any other software like Finder obviously doesn’t do so and, when moving files in such, you are aware of that.

@Stenis

I guess that´s why DAM-systems in the industry mostly uses JPEG-files for the images since then there is no need at all for sidecars.

Where did you get this idea from? Firstly metadata can be written directly to RAW files and, secondly, most pro photographers don’t use JPG files except for transmitting exported copies of edited images.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@OXiDant I think I was doing alright until part of the “Edit” so lets see if I can fathom it out while I type this!?

My own “workflow” is as follows:-

1. Ingest/import/read from SD card using ACDSee as I have done since way, way back, to a directory created by ACDSee which has the form of

(a) ACDSee provides a Metadata template which I use for the Author and Copyright only but as you can (hopefully) see I could add a lot more data at that time. I also currently split the JPGs and RAWs at this time and back up to the Q drive which is an NVME drive to provide a speedier editing “platform”.
(b) Unfortunately ACDSee will only apply the metadata template to the JPGs
(c) I delete automatically at this time which is arguably too soon!
(d) So JPGs are no longer “pristine” but RAWs are, and both have been deleted from the SDCard but could easily be recovered if required!? RAWs are in a subfolder and both are backed up (temporarily).

2. Use FastStone Image Editor to open the directory and rename, typically add location, who (grandchildren), camera, lens, and .F if any family photos, i.e. “2022-10-02 - Visit with **** & **** & ****, St. Leonard’s & Hastings Museum - G9(200).F”
3. I have a copy of FastRawViewer and will sometimes use that to weed out really bad images, principally with the “sharpness” screen, “launching” from FastRawViewer.
4. But typically I pick a photo (generally a JPG) and launch DxPL and apply one or other of my own presets to see how good, bad or indifferent the image looks, refine the applied adjustments and copy before pulling all the images in and selecting the “worst”, generally with lighting issues, and apply the adjustment I have just “made” and when satisfied, navigate to the RAW directory and start for “real”!

So with AS(OFF) the above and your workflow will have no DOPs to start with so the metadata will be taken from the xmp (embedded or sidecar), i.e.

DxPL first encounter

With AS(OFF):-

• NO DOP then uses Image metadata
• With DOP then uses the DOP metadata on releases after PL5.3.0, including PL6
  (- On releases prior to PL5.3.0 uses the image metadata)
• With DOP prior to PL5 uses ‘Rank’(‘Rating’) and ‘Rotation’ from DOP, the rest from the image

With AS(ON):-

• DOP or no DOP use the image metadata ONLY

ANY metadata changes in DxPL

e.g. ‘Rating’, ‘Rotation’ (now ‘Orientation’), IPTC fields, GPS etc.

With AS(ON):-

• immediate write back to image with keywords in the post PL5.2.0 format (+ the changes made that triggered the write back). In truth the Read from and Writes to are for ALL the metadata!!

With AS(OFF):-

• No write back of any metadata until ‘Files’/‘Metadata’/‘Write to Image’, when and if the user decides to transfer the data back to the image metadata

Any external metadata changes

With AS(OFF):-

• DxPL detects that a change has occurred and signals with the ‘S’ icon
• No automatic reading of the updated metadata until a ‘Read image data’ is made by the user which will potentially overwrite any “pending” updates in DxPL

With AS(ON):-

• DxPL will execute the ‘Read image data’ on detecting a change and overwrite the data in DxPL

In all cases it is an all or nothing operation, there is no selective metadata ADD operation (something I asked for way back in 2021 towards the end of PL5 Beta testing) given that DxPL possesses powerful internal capabilities like this

Why can these not be used for the transfer of metadata from Image to DxPL and from DxPL to image @Musashi. However, @OXiDant, although it might be a little tricky for some users to use such facilities I would like the full set of options available when Copying and pasting between images inside DxPL to be available to the operations from image to DxPL and vice versa, i.e. the screen above and

@OXiDant

And the keyword selector screen was a bit ockwards in the earlier versions so quick multi image keywording wassn’t that easy as in Bridge.

Agreed, I have managed to move one keyword to within another, classic old age issue of lazy mousing or an old mouse where the microswitches have become sticky or both! I asked for an option to disable drag and drop some time ago I believe @Musashi.

Templates for metadata would be good, currently you could emulate them by creating your own

1. Create a directory of suitable images
2. Assign metadata to those images and write back to the image, adjusting the name of the image to represent the “Template”
3. When you want to use a template, navigate to the appropriate “Template” image, copy the template metadata and past either all or selected fields

@OXiDant

1 possible lock for changes outside dxo on the xmp.
Simple question xmp is changed outside dxopl update yes/no?
(very important in case of not sure what’s happening is correct to prevent damaging of your xmpdata.)

Not entirely sure what you mean. There is no way that DxPL could lock the metadata externally. In theory it could lock the image record itself (but would also need to lock the xmp sidecar as well). Typically the programs all hold the locks on images for the minimum time possible to reduce potential clashes!

With AS(OFF) external updates are not going to cause an update within DxPL nor any update from DxPL “interfering” with the image but you now have a dilemma which set of data is “correct” and which is “wrong” and no simple way of telling! You need a “Compare” facility to show both sets of metadata side by side.

@OXiDant

2 template, preset manager. Partial pasting ofcoarse.
Option a overwrite everything with this data.
Option b only write in empty fields
Option c manual selecting which fields to overwrite.

My option above would allow you

to create your own and have instant access to all the capabilities for partial updating tomorrow at the “expense” of managing the entries yourself!?

@OXiDant

3 create a 3 tab windowsystem.
Library
Meta data manager
Editor

We could formulate a request to DxO but …

Hope the above is not too rambling.

Regards

Bryan

[OXiDant (Peter)]

Hy Bryan @BHAYT ,
Great and clear summary in your first section.
AS, automatic synchronisation, let me feel still a bit icky, not comfortable and your excelent summary let me decide to keep it off. To much danger of unwanted changes in my xmp files. So a read only is fine by me for now. (change a iptc field inside dxopl means or it lives only in dxo database and is not effecting my archive or i have to provoke a write action on the xmp file.)

@BHAYT

Templates

Yes there are workarounds but i can’t grasp why it’s not a function inside dxopl.
All the underlaying programming is already there. This is one of the reasons i stil need Bridge.

@BHAYT

Not entirely sure what you mean

Automated reading all metadata and a Asking of it may write back inside the xmp. (dopfile is self made so no risk in that when it auto updates.)
Then the AS [on] isn’t a danger to your precious xmp files anymore because you can decide if it must write(update) or not.

@BHAYT

Hope the above is not too rambling

No not at all.

If suggested this third tab long ago in order to give the user interface some extra space for the metadata toolset.

When my internet is back on my desktop i can install v6 and actual test some things.
Friday my internet was totally gone. Someone did something wrong in a utility cabinet somewhere in the neigbourhood and cut me off from internet.
Repair should be done between 8 and 12 this morning but no show yet.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@OXiDant

1. I created a “document” in Word
2. Took a snapshot big enough that PL6 would actually let me edit it!
3. Stored it in ___Basic Metadata Template & opened in PL6
4. Cleared all image adjustments
5. Set basic IPTC fields, mostly with a bit of nonsense
6. Exported the DOP (it would have been written anyway)
7. Added to a project
8. Opened an existing directory, selected an image and now we have a problem because the users needs to “navigate” between the metadata images and the real images. We need metadata presets but ones that can have the standard metadata “paste” facilities used when they are applied @Musashi.

9. If the same metadata template is useful for an entire directory of image then to the template project and copy all metadata. Then navigate to image directory and paste, All or some

Job done

@OXiDant

@BHAYT

Templates

Yes there are workarounds but i can’t grasp why it’s not a function inside dxopl.
All the underlaying programming is already there. This is one of the reasons i stil need Bridge.

Understood and agreed. Just not a head turner when making a new release!

@OXiDant

@BHAYT

Not entirely sure what you mean

Automated reading all metadata and a Asking of it may write back inside the xmp. (dopfile is self made so no risk in that when it auto updates.)
Then the AS [on] isn’t a danger to your precious xmp files anymore because you can decide if it must write(update) or not.

@OXiDant I and others asked for this a long time ago and I have been asking for a “merge” function since early in PL5 Beta testing, before the rather excellent selective pasting function had been invented by DxO.

Effectively it is the re-writing of keyword formats @Musashi that is preventing the much freer use of metadata interchange between DxPL and other products, i.e. via the image metadata. My proposal for allowing keyword format templates to emulate the format of other packages is, I believe, very powerful but currently this one issue is stopping people using other features of metadata, e.g. ‘Rank’, ‘Rotation’, IPTC adjustments etc. being made in DxPL and transferred to the image for “fear” of damaging the keyword formats, which are going to be “damaged” on export anyway.

Is not the simplest compromise to provide a keyword (b)lock so that even with AS(ON) no keyword data leaves DxPL but all the other metadata can be freely exchanged and (optionally) updated keyword data can be input from the image !

When it comes to export, the metadata would be taken from the image in its current (at that time) format and added to the metadata from DxPL as I have discussed and you have acknowledged before @Musashi ?

So added to the ‘Metadata Synchronisation’ there needs to be an option for keywords synchronization or not and/or with the ‘Read from’ and ‘Write to’ add the equivalent to the ‘Paste metadata options’, either the current two levels of paste options or just the ‘paste selected’ screen, i.e.
‘Read from image’, ‘Read selected from Image’, ‘Write to Image’, ‘Write Selected to Image’

While having the full gamut of the paste options available to a Read/Write operation would potentially be useful currently it is the keywords (keyword formatting) that are causing the problem. I am unclear what procedures (code/code libraries) DxO are using in DxPL for handling image metadata and the “granularity” available.

Effectively DxPL would need to read all the metadata from the image (some of which might have changed since it was last read by DxPL - less likely if using AS(ON) but then there needs to be a global block on keyword traffic to the image) as if it was a new image, then paste the selected fields and then write the data back to the image in as many operations as it currently uses. This might be overkill or not?

@OXiDant Sorry to hear about you internet given that mine is currently achieving all of 2.5Mbits download and 0.5Mbits upload at least it is mostly working but with occassional breaks for a minute or so!?

[OXiDant (Peter)]

@BHAYT

@OXiDant Sorry to hear about you internet given that mine is currently achieving all of 2.5Mbits download and 0.5Mbits upload at least it is mostly working but with occassional breaks for a minute or so!?

Owh i have activated guestwifi by the neighbours so i have a little more then your shocking 2.5Mbits…(that’s nearly slower then the old modem internet of the old day’s!) i have now around 30Mbits down and 26Mbit up. Normally wired around 88Mbit and 50Mbit on vsdl new type of adsl, copperwired telephone connection.
Problem is my desktop is wired only.
This is a tablet with popupkeyboard so i see only my typingtext.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@OXiDant our connection is VDSL (fibre to cabinet) but the copper from the house to the cabinet is very long and we have a damaged line somewhere along its length. The speed should be closer to 14Mbits/sec and 1.5Mbits/sec and no Virgin (cable) available because we live on a “private” (un-adopted) road where we are responsible for the verges, and to the centre of the road.

When Nynex (as was) were laying fibre they were making a complete hash of the pavements and roads so the residents said NO!! So no cable.

But it hasn’t stopped me writing some of the longest posts on this forum and I must complain to the provider this week!!

Take Care

[Joanna (Joanna Carter)]

We live in rural Brittany and use a 4G+ modem connected to the Free.fr 5G network for only €15 per month, with 210GB of data. I’ve just done a Speedtest and get 88.7Mbits down and 7.6 Mbits up.

In theory, in a couple of weeks, we will be getting FTTH fibre on a special offer of only €15 per month for a year but, unless they change the normal price from €35 per month after the year, we shall be changing back to the 4G+ modem

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Joanna we live next to the point where the A24 and A27 merge in the lee of the South Downs and in a not spot for mobile phones and in a bungalow. So bad mobile phone reception, near non-existent 4G and no real height to talk about. We would need to give Elon Musk some of our pension!!

[John7 (John Barrett)]

Carrier pigeon

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@John7 We have plenty of wood pigeons that feed regularly from the bird table but they don’t look particularly useful, except for lunch but all feathers and bone and my bird food.

Sorry not really practical, I might get them to peck away at the keyboard but I have then got to clear up all the …

[John7 (John Barrett)]

Oh well we got so desperate with BT and appalling line condition they pigeons looked appealing at one time when we not only had no internet and then no phone (mobile not spot again, interesting to see the BT engineers when they found they had no reception, (one poor sod sent out from London for experience) close to tears after being told where in the village to go to stand a chance to getting reception to run their tests!). Our problem was a corroded main phone cabling through the valley into Wales

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@John7 We had the same problem back at the start of the pandemic in 2020 when the engineer came to investigate a problem and was stunned that he had no EE signal! We believe our problem may well be a collapsed cable duct on the main roundabout stretching the cables, i.e. we have a high resistance problem.

@Stenis The following was "buried in a response to @OXiDant but I believe that it needs to be seen without any of the rest of that response.

DxPL first encounter:-

With AS(OFF) [or MS(OFF) = Not Selected on the ‘Edit’/‘Preferences’ screen]:-
i.e.

• NO DOP then DxPL uses the Image metadata
• With a DOP then DxPL uses the DOP metadata on all releases after and including PL5.3.0, including PL6. No attempt is made to check for any potential mismatch between the metadata from the DOP and that in the image so no potential mismatch is signalled,
  (- On releases prior to PL5.3.0 DxPL used the image metadata on first discovery regardless of the setting of AS(ON) or AS(OFF))
• With a DOP prior to PL5, i.e. PL4, PL3 etc. DOPs, DxPL uses ‘Rank’(‘Rating’) and ‘Rotation’ from the DOP (which can be 0), and the rest of the metadata comes from the image (including the keywords because prior to PL5 keywords were only stored in the DxPLdb and not in the DOP)

With AS(ON):-

• DOP or no DOP DxPL uses the image metadata ONLY

ANY metadata changes in DxPL

e.g. ‘Rating’, ‘Rotation’ (now ‘Orientation’), IPTC fields, GPS etc.

With AS(ON):-

• DxPL immediately writes the metadata back to the image with the keywords in the post PL5.2.0 format, i.e. ALL the metadata is written back to the image, including whatever triggered the change, some of which may have come from the image originally, some of which may have been added/changed in DxPL!!

With AS(OFF):-

• No write back of any metadata until ‘Files’/‘Metadata’/‘Write to Image’ is executed, if and when the user decides to transfer the metadata back to the image metadata and this will be all the metadata as in the AS(ON) case, only the trigger process is changed (from automatically to manually invoked)! The fact that there is a potential mismatch is not flagged in any way, at any stage.

Any external metadata changes

With AS(OFF):-

• DxPL detects that a change has occurred and signals with the ‘S’ icon
• No automatic reading of the updated metadata until a ‘Read image data’ is made by the user which will potentially overwrite any “pending” updates in DxPL. Clicking on the ‘S’ icon will cause a prompt to come up enabling the user to read the new metadata.

With AS(ON):-

• DxPL will execute the ‘Read image data’ automatically on detecting a change and overwrite the data in DxPL, potentially overwriting any pending DxPL metadata updates, hence the following warning when setting AS(ON)

Please note:-

The so called synchronisation is not a merging of any data it is a replacement of one set of data by another. Effectively, with AS(ON) PL5 will update the image metadata as soon as any changes are made (data out) and read the metadata in whenever an external change is detected (data in), hence, at the end of each transfer the two sets of data are “in sync”.

If all goes well the process will flip-flop from one to another (DxPL to image, image to DxPL, DxPL to…) and it will look as if the data is actually merged and nothing will be lost but the synchronicity of the data is achieved by one set of metadata overwriting the other!

Effectively, DxPL is executing ‘Read from image’ and ‘Write to Image’ operations alternately as changes are made via DxPL and made externally to the image by software of some description (armed with its own rules) and detected by DxPL.

On Windows, DxPL requests an interrupt from the operating system whenever the current directory is changed and then checks the ‘Date modified’ timestamp(s) to locate any change(s), or so I believe.

The exact process is “locked” in the code and DxO have never chosen to share the exact workings @Musashi. With PL6 I am hoping that we will see a greater openness from DxO personnel @sgospodarenko, rather than having to construct convoluted tests, make giant leaps (small bounds/tiny steps) in our understanding of the processes so that I am actually able to write the above with any degree of certainty (the standard disclaimer still applies!)

If there is any missed step in the process then there will be the possibility of data flowing the wrong way! In most of my tests, including PM scripts that changed the ‘Rating’ on 877 images one by one, PL5 kept up and never missed detecting an update. On a more local test, PL5 missed an update and an ‘F5’ did not recover the situation because it was simply not trapped by DxPL or was trapped but the timestamp test “failed” and the update was not in the database. How this can happen I don’t know and it is not easy to reproduce!