Win 10 PL5.1.4 "Corrupted" DOP plays havoc with PL5.3.1.4762 keywords (Probably finger trouble)

BHAYT · July 11, 2022, 2:50pm

Please NOTE:- AS(OFF) in all tests on both machine (Main and Test), therefore

PL5.1.4 behaviour when discovering new image - take metadata from image (including sidecar)
PL5.3.1 behaviour when discovering new image - take metadata from DOP

Summary:-

In preparation for further testing, a group of already used test directories were moved (secured) to a subdirectory. Another subdirectory was then created and PL5 directories were copied to those subdirectories to be available as input to a number of different packages. One of those directories copied over contained DOPs with unexpected keyword entries!? The values are now in the original test directory and 12 new test directories!

The tests that resulted in the creation/updating of xmp sidecar files and DOPs were conducted as follows:-

Add the keywords to PL5 either by and/or from the keyword list
Write the metadata back to the image (sidecar)
Check with PIE and “fix” any unexpected errors (typically typos) and re-export
The xmp sidecar files contain the correct data
Two DOPs in the PL5.1.4 directory have unexpected entries @sgospodarenko

Looking at the date/timestamps of the DOP and the xmp sidecar files shows that the DOPs were created just after the xmp sidecar files.

At a guess it looks like the “a|m|b” hierarchy was moved into the “as|ms|b|bb” hierarchy! I have had problems in the past with picking up keywords and moving them accidentally! In this particular case had I been running with AS(ON) the metadata would also have been “destroyed” at the same time!

It is far too easy to move the keywords in the keyword table by accident (and very hard to resolve the problem, if you even realise that it has happened!). This is less of a wonderfully flexible feature and more of a “trap”!

@Musashi the following features are more of a liability to the user than an asset

Please provide a “button” to enable/disable keyword drag and drop @Musashi!
Please disable the DOP read feature being automatically enabled when AS(OFF) is selected and return the product to the original (pre PL5.3.0) state for AS(OFF), i.e. initial discovery uses image metadata not DOP metadata.
Please provide an option to enable DOP metadata reading e.g.
1- As a ‘Preferences’ option that will then change the character of the discovery between Pre PL5.3.0 and Post PL5.3.0
2- Add a metadata ‘Read from DOP’ command
3- Both

Currently with Post PL5.3.0 the user needs to do the following to restore the old behaviour

Allow PL5.3.0 to discover the image when the metadata will be taken from the DOPs
Mark all images and use ‘File’/‘Metadata’/‘Read from Image’ to restore the metadata from the image, i.e. the product has been changed to “favour” those users who want to only preserve the DOP, or because it could be implemented “cheaply” by the method chosen!

The suggested change would require the following to be executed

Do nothing but use the restored Pre PL5.3.0 behaviour to automatically take metadata from the new image when discovered

OR

Allow PL5 to take the metadata from the image by default and then (optionally) replace that with the DOP metadata using the ‘Metadata’/’‘Read from DOP’

Those that always want to work from the DOP or always want to work from the image metadata would favour a ‘Preferences’ option which allows the desired behaviour to be established and no further intervention, by the user, is needed!

But the ability to recover metadata from the DOP is a potentially useful option for all users!

Problem Investigation:-

Because PL5.3.1 uses the DOP when discovering a new image, that was the first place to look and the DOP looks like this

But the xmp sidecar, written using a ‘Write to Image’ after all changes were completed shows the following, so why the difference (perhaps because the hierarchy was corrupted after the metadata write)?

BHAYT · July 11, 2022, 2:53pm

For stats monitoring

platypus · July 11, 2022, 3:35pm

can’t see the forest for the trees, but I’ll try.

add a>b>c to IMAGE with DPL 5.1.4 and make it write the .dop sidecar.
open IMAGE with DPL 5.3.1, make it read the .dop sidecar (written by 5.1.4)
check keywords of IMAGE in DPL 5.3.1 and find something that is not a>b>c

Is this what you say?
Did you delete the database between 1. and 2.?

BHAYT · July 11, 2022, 4:01pm

@platypus It is but the problem is that the keyword tree became corrupted (somehow, or it looks that way), potentially by a mouse movement over the keyword hierarchy and I wound up with “a>m>b” “mashed” with the tree for “as>ms>b>bb” (animals>mammals>bear>black bear) somehow (or so it appears)!?

Then a few minutes after I had done the proper test and ‘Write to image’ the DOP was written and the keywords had been changed to the “nonsense” that I then encountered when the images, now in a new directory, were discovered by PL5.3.1!

I am presuming that it was “finger” (mouse) trouble and given I did two groups of tests with PL5.1.4 on Test but using directories on Main and two groups of tests with PL5.3.1 on Main and “only” two of the 20 images were “damaged”, I don’t think that this is reproducible but thanks anyway.

I encountered this … by accident as any other user might. The keyword tree is way too sensitive and I have damaged it before, mix that with the PL5.3.0 change where metadata is taken from the DOP and PL5 is “good” at keeping the DOP up-to-date and we have a “perfect storm” (in a teacup!).

No databases were thrown away but I did move the original test images to a new directory for safe keeping but positively the wrong thing to do if you need to check the database that knew where they were (and are not any more!)

PS:- This is less a bug and more a set of unfortunate coincidences, which I “like” way less than a nice, reproducible bug!

platypus · July 11, 2022, 7:56pm

…this means that you accidentally changed the structure?

BHAYT · July 11, 2022, 8:07pm

@platypus In truth I am not sure what happened! But the most likely sequence was

Write metadata as last part of test
Somehow changed hierarchy just after that
PL5 then wrote the DOP with the changes!
Following day (today) prepared for new tests etc.
Discovered two images had “weird” metadata (taken from DOP because of PL5.3.0 changes, i.e. discovery with AS(OFF) means read metadata from DOP!!
“Moan” to DxO about how easy it is to corrupt the keywords with “lazy” mouse action and the way that AS(OFF) has been changed!
But still puzzled!?
So yes I believe I accidentally changed the structure!

platypus · July 11, 2022, 8:40pm

Okay, then its not PhotoLab, mixing up keyword structures, which would have been really disturbing.

Reading hierarchical keywords from files can wreak havoc. I’ve seen it, testing C1, Lr and PL, but the issue I found was when I read Lr keywords with C1 (version 12) and DPL didn’t cause issues during these tests.

BHAYT · July 11, 2022, 11:17pm

@platypus I think it is more likely that I caused the issue. Both my machines have a separate keyboard and mouse so it can get a bit crowded on my desk or I did it deliberately to test something but I hate dragging keywords so that is unlikely, sadly I cannot remember exactly what I was doing!

Looking at the PL5.1.4 database we have

The order of data entry would be “a”, “m”, “b”, rows 24, 25 and 26, followed by “a|m|b”, rows 28, 27 and 24, going in reverse i.e. “b” to “m” to “a”, followed by “as|ms|b|bb” for Tests 1, 2 and 3 respectively.

For this to work properly the entry at 24 should be NULL but it actually points to entry 29 (“as”, part of “as|ms|b|bb”), i.e. the hierarchies intersect, which they shouldn’t and I believe that they didn’t 3 minutes before the DOP write when the xmp sidecar was written/updated!?

When I started the topic I believed it was a bug, then decided it was “damage” started by me causing the hierarchies to intersect (somehow) but it is a good object lesson in the keyword handling issues and the repercussions of the DOP processing change!

The “fix” is to execute a ‘File’/‘Metadata’/‘Read from image’.