PL5 Tag Field not read from .dop file

@Sandersoni not least because I appear to have lost the tags in the Synchronisation test I undertook where I reported another fault (a “possible” variant on my PL4 complaint about Ghosting or a completely different problem) in the post above this one.

Hi Keith,
thank you very much for your reply! I did not expect that the database access is that easy and not secured or encrypted. I simply used HeidiSQL which I normally use to connect to a MariaDB Server. I had never worked with SQLite before but I really like the concept.

What is really interesting: I expected that the Item properties like rating are not processed because I changed the drive where my photos are stored. But the target folder entry is correct and all Rating entries for my PL 5.x edited Photos have a ‘NULL’ entry. So it was definitly a bug which obviously overwrote all rating entries in the database (they are still available in the PL 5.x dop files but not read)

What I could do is to write a programm which goes through all entries of the Item table and checks if a dop file exists for them, parses the dop file for the rating and then update the referring column in the Item table. But actually this should be done by PL since I payed for the Software :wink:

1 Like

Hi Bryan,
no need to apologise at all, thanks for your replies!
I use Windows 10 and I also experienced that PL4 dop file ratings are available in PL5.

@Johnny In my tests I think that I discovered that PL5 takes the the PL4 DOP ‘Rating’ in preference to the photo (embedded or sidecar) but the photo ‘Rating’ in preference to the PL5 DOP. However, while that is acceptable in my test because I am hacking a DOP which should be identical to the database (the Uuid test will be passed) in the case of you presenting effectively new files to PL5 then the DOP should be taken into account in some way, even if that is “gated” by a date/timestamp check etc.

This is actually a bug @sgospodarenko and I will try a test to verify this situation - most of the data is already present on my system and just needs moving to a new folder and “ingested” by PL5!

Would I? :stuck_out_tongue_winking_eye:

If you followed my procedure, to the letter, doing all the intermediate tests, and not getting the same results, this indicates a difference between Mac and Win and, instead of being a purely UI issue, this is a matter of data integrity.

Just to make sure, I reran my test

Preparation

  1. Delete databases for both PL4 and PL5
  2. Delete DOP and XMP files for test image
  3. Check Rating was not set in RAW file by camera

Set Rank in PL4

  1. Open image in PL4
  2. Set ‘“Rank” to 3
  3. Close PL4

DOP file after setting in PL4

	Software = "DxO PhotoLab 4.3.6 72",
        …
			Rank = 3,

Read Rating in PL5

  1. Open PL5
  2. Notice no stars for Rating
  3. Close PL5

Unless I make a change to the image adjustments, PL5 doesn’t update the DOP file!!!

Write Rating in PL5

  1. Reopen PL5
  2. Set Rating to 2
  3. Close PL5

DOP file after setting in PL5

	Software = "DxO PhotoLab 5.1.1 52",
        …
			Rating = 0,

Rank of 3 has been deleted and Rating of 0 has been added!!!

Reopen PL5

DOP file now contains…

	Software = "DxO PhotoLab 5.1.1 52",
        …
			Rating = 2,

XMP file has been created…

         <xmp:Rating>2</xmp:Rating>

So, it looks like neither the DOP file is updated, nor the XMP file is created until PL5 is reopened for a second time after the Rating was set.

However, if after setting the Rating in PL5, instead of closing PL5, I switch to Finder, to look at the folder that contains the file, Whilst I am switching, the DOP file gets updated and the XMP file gets created and I see it appear in Finder.

Crucial difference

  • closing PL5 without changing focus to another app does not write either the DOP file or the XMP file
  • changing focus to another app does write both the DOP file and the XMP file

Curiouser and curiouser, as Alice would have said

I set DPL to NOT automatically import/export/sync sidecar files.

When I …

  • change a rating in DPL4 to ***
  • manually export the settings
  • manually import .dop sidecars in DPL5

…the changed rating comes along correctly.

When I then …

  • read .xmp sidecars (no rating present), rating is set to 0
  • read .dop sidecars from DPL4, rating is set to *** again

Summary:

  • DPL5 does what I expect with .dop sidecars from DPL4
  • DPL5 does NOT the same as above with DPL5 .dop sidecars

Conclusion

  • DxO needs to fix this bug
1 Like

@Johnny @Joanna @sgospodarenko Sorry I have been “sleeping” on the job and not testing as I should. Now to make amends.

Took one of my previous group of test photos and copied to a new directory. These are a mix of JPGs, RAW (RW2 and ORF) and the photos contained the same ‘Ratings’ as the DOPs as a result of the tests that I had run.

  1. Used FRV to clear all ‘Ratings’ so only the DOPs carried the previous ‘Ratings’.

  2. Opened PL5 and navigated to the new directory containing photos with no embedded or sidecar ‘Ratings’ and DOPs with various ‘Ratings’ set.

  3. Result - NOTHING not a ‘Rating’ in sight, nowhere (except in the DOP). Sorry PL5 has just “trashed” all the DOPs as well, Rating=0 in all cases.

I consider this to be a bug although I have to ask why your ‘Ratings’ were only in the DOPs but if you were running with the ‘Preferences’ ‘Sync’ option off and never used the ‘File’/‘Metadata’/‘Write to Image’ then the DOP is the only place they would reside!!!

I repeated the test by copying the original DOPs back into the directory and got the “looked” for Virtual Copies (VCs) which show what the result should have been if PL5 had done it correctly in the first place.

[M] represents what PL5 did and [1] represents what is should have done!

I am afraid that I was testing some of this following a list of scenarios when I encountered the PL4 Ghosting issue and wound up down the keyword “rabbit-hole” plus others. I must stop trying to “fix” a problem but actually get to the actual symptoms of a problem and then re-create it. However, many of the symptoms reported in the posts must “chime” with the developers and it would be good if that info is fed back to the posts so we don’t go chasing “ghosts” (“pun” intended).

Please get the restricted forum entry problem fixed @sgospodarenko it is “cramping” my input :upside_down_face:

The problem is that not everybody goes to that extent and, for those of us who expects it to “just work”, it is causing problems.

That procedure shouldn’t be necessary or required

1 Like

A good work around @platypus but it should not be necessary. @Joanna it almost looks as if they have implemented the PL5 code for PL4 DOPs (on the Mac) and as my post above suggests this is also a bug, so you may have two bugs for the price of one or just two bugs if your PL5 import does the same as I am suggesting in the above test @sgospodarenko.

Fully agree. But my settings would have saved my bacon…
Automatic things are nice if they work. In this field, they don’t.

Conclusion

  • DxO needs to fix this bug (ceterum censeo…)

@joanna Absolutely!!

Truly bizarre!

In my tests PL5 has not updated the PL4 DOPS after using them to set the database, but I haven’t checked how many restarts of PL5 it takes before PL5 starts using its own DOPs! In my test above it is quick to replace the 5.1.2 DOP with its own 5.1.2 DOP where all Rating=0, thereby destroying user work!!

I am using PhotoLibrary throughout and setting no keywords during these tests, Ratings only. From other tests ‘Tags’ are a whole new “can of worms”!

I promise I will complete the spreadsheets update for keywords by the end of the weekend, i.e. ACDSee keywords imported and updated (synced to the original photo) by the package and ACDSee imported and a keyword added by each package, Bridge appears to be the biggest “failure” with the last one but I will explain that in the other (also inappropriate, but at least “my” topic) rather than here.

@Musashi, @sgospodarenko Thank you for your post and for providing a sneak preview of elements of the FAQ but I am afraid I strongly disagree with your opening statement which according to my tests is positively the wrong thing to do!!

Namely,

“Since version 5, PhotoLab now reads rating from .xmp files and no more from .dop files”

  1. This statement is factually incomplete, PL5 reads from .xmp sidecar files and from the embedded ‘xmp’ data contained within the photo (RGB or RAW format) and no more from .dop sidecar files.

  2. This statement casts many users coming from using PL4 adrift by actually changing the way that ‘Ratings’ data is handled and may require users to change their work flow. This change should have been highlighted at the time of the release not months later.

  3. Arguably worse than any of that, having collected the ‘Ratings’ data from a new ‘xmp’ source PL5 then “destroys” the data in the DOP, potentially eradicating that data forever. Unfortunately users have been used to DxPL never destroying their data from release to release until along comes PL5 and many may not have backups (because there has never been any need for them, until now)!

  4. The PL4 DOPs get “special” treatment, at least on Win 10, according to my testing but Joanna’s testing on the Mac shows that the Mac release is apparently ignoring the PL4 DOPs i.e. apparently adopting the new, PL5 strategy! The handling of PL4 .dop sidecars is not shown in the table.

I ran a test on PL4.3.6 which turned out to be something of a “mixed bag”! Why did I run such a test on a previous release and the answer is because I thought it would be quick and give some insight into the upgrade process and the thinking behind the PL5 strategy. Simple it was not!!

  1. PL4 can read the ‘Rating’ (Rank) from either the .dop or the .xmp sidecar file (if one exists) with RAW photos, I didn’t test it with embedded ‘xmp’ metadata (laziness).

  2. I started with all 4 RAW files containing no ‘Ratings’ (Rank), actually PL4 refers to ‘Rating’ in the product but stores as ‘Rank’ in the .dop sidecar.

  3. I added ‘Ratings’ in PL4 and .dop sidecar with the correct ‘Rank’ files were created with the correct ‘Rank’but no .xmp sidecar files were added. With this test the DOPs were created when I closed PL4.

  4. I deleted the images in PL4 and added them back and assigned ‘Rating’ values in FRV which only creates .xmp sidecar files for RAW images.

  5. PL4 took the ‘Rating’ from the .xmp sidecar and showed them in yellow, no DOPs were created at this point (no other edits had been made)

  6. Changing the .xmp ‘Ratings’ externally was immediately picked up by PL4 and shown correctly.

  7. Setting the ‘Tag’ in PL5 resulted in the creation of the .dop sidecar (DOP) to hold the new flag but the DOPs had Rank= 0 at this point! Is there a flag in the DOP to tell PL4 that the ‘Rating’(Rank) if being externally set? If the ‘xmp sidecar is lost at this point the DOP will be of no use for recovery.

  8. I changed the ratings of two of the photos externally and the DOP of those two photos changed, the ‘xmp sidecars remained unchanged (PL4 immutability at work perhaps) and the “star” colour changed from yellow to white.

  9. I changed the values of the ‘Ratings’ in all photos externally as might be guessed those with PL4 assigned values showed no change but the other two were changed in line with the external product used

  10. Mixing and matching PL4 ‘Rating’ assignment with external assignment means that ‘Ratings’(Rank) is spread between sidecars types, i.e. .dop sidecars and .xmp sidecars

So at the end of the test PL4 might believe one thing and the external program another if the two methods of assigning ‘Ratings’ are used!

More importantly PL5 needs to be able to handle the situation of this potential mixture of sidecar files in the migration from PL4 to PL5. Arguably PL5s initial source of all such data is the PL4 database but as some of us found that upgrade didn’t go as well as it might have and the database was “lost” to PL5 during the upgrade . PL4 keywords were only held in the database (as far as my tests show).

I managed to recover my database but the loss of the database came as a real shock to longtime DxPL users.

Effectively prior to PL5 the 'Ratings’ (Rank) field was not held external to the PL5 system because of the “immutability” mandate of pre-PL5 releases with respect to JPGs etc so the .dop sidecar was treated as the primary (or only) source. PL4 did not write the ‘Rating’ (Rank) to the .xmp sidecar file as well as the .dop sidecar file as I thought earlier in my testing so there is now a potential conflict between the differing data that can be held.

Hence, many users coming from that background were not really treating ‘Rating’ (Rank) as anything other than DxPL metadata held in the DxPL database and, for portability, in the .dop sidecar.

I am afraid that I would suggest that such an expectation would be carried forward by those users into PL5.

Suddenly changing the rules because the ‘xmp’ data is now available to PL5 for update and that is, arguably, “where that data belongs” is unacceptable without any suitable warning. Forcing a change in work flow is arguably unacceptable (Skylum tend to do it often), changing it without adequate warning is …!!

Hence, taking a null or 0 value from the ‘xmp’ embedded’ or sidecar data is rather short sighted because of the heritage of PL4 which many users are coming from. Users may not want to use PL5 ‘xmp’ metadata and actually considered ‘Rating’ (Rank) as DxPL metadata.

The decision has effectively cut DOP ‘Ratings’ “loose” and made it a WOM (Write Only Memory) field like the keywords which are also stored in the DOP.

However, the important thing is that the old PL4 processing flow involved both the .dop and the .xmp sidecar files and and the new PL5 processing flow completely ignores the .dop with respect to ‘Ratings’.

Given that PL5 cannot distinguish between a Null and a 0 ‘Rating’ entry, plus the fact that NEF and my Panasonic camera set the value to 0 in the original photos the adopted strategy is doomed to corrupt the data of unsuspecting users, which is exactly what has happened.

The current PL5 algorithm is wrong, in my opinion, it needs to be more sophisticated in handling the data potentially stored in .dop sidecars, .xmp sidecars (for RAWs) and embedded in the photo for (RGB and potentially RAWs as well).

In addition it is possible that a different set of rules should be in force for the first introduction of a directory to PL5 (i.e. the database) versus subsequent navigation to the directory. However this still poses the problem if the .xmp data (sidecar and/or embedded) contains any ‘Rating’ information! Effectively the metadata dates should be taken into account and the .dop sidecar should also be considered as a candidate for the data, certainly on the first discovery of the directory and, potentially every time!

Therefore discover the potential Rating data available and rank (not Rank) that data based upon the date/timestamp from latest to first, i.e. determine the latest data available and that should include keywords, GPS data etc… (I haven’t tested them in this context at all yet!!)

While one alternative solution might be to force a write of the .xmp sidecar data even when the ‘Preferences ‘Sync’ option is off and the customer has not (and possibly never will) use the ‘Metadata’ ‘Read from disk’ and ‘Write to disk’ commands PL5 could force a write to secure the ‘Ratings’ in the .xmp sidecar but users needs to be aware that these need to be secured in any backups and that they have value and that doesn’t help any users currently “trapped” in the situation of PL5 DOPs with ‘Ratings’ in them!

The current algorithm is way too simplistic (I feel like saying naive but that would be rude), it does not take into account the potential loss of the database (in the PL4 to PL5 upgrade or later because the database becomes damaged/lost etc.). It fails to cope with the possibility that users may not embrace the new features of PL5 and start using PL5’s new found “powers” of being able to store metadata and continue to believe in the power of the DOP as a backup for the database!

Particularly if there was no timely warning about the potential consequences of the PL5 changes.

Please change the strategy and/or provide a ‘Preference’ to allow PL5 to use the PL4 strategy and please discuss this with the users, you created the product and it is excellent but we use it, many on a daily basis, in a wide variety of ways and we might just have something constructive and useful to say!

2 Likes

Wow! The large edit window is back in style! :laughing:

I’m off out for a walk and shall digest it more fully when I return. From a quick glance, it doesn’t look good from a potential data loss point of view.

@joanna Have a nice walk or hope you have had a nice walk.

That one was done on a word processing package (before the fix) so that I could keep notes as I ran tests on PL4 (trying to understand the legacy of the past and its potential influence on PL5, if any, and the “baggage” that might accrue from the past in terms of data and the “habits” and “expectations” of the users).

There is a little (!?) “second-guessing” but that is inevitable given the nature of testing software in a partial vacuum.

@musashi and @sgospodarenko

I feel that I need to add a comment to my excessivley large post above.

Basically I consider downgrading the ‘Rating’ in the DOP (.dop sidecar) is actually the correct thing to do but then I would say that because I have never used ‘Rating’(Rank) in PL4 and may or may not use it in PL5.

The tests that I ran were to try acquaint myself with the working of the previous version and help me to understand the legacy of PL4 ‘Ratings’(Rank) and what the implications for users coming from that background might be, plus what might the implications of that change be in less favourable circumstances e.g. (I should have used i.e. but e.g. covers the case that I might miss something!)

  1. The transition from PL4 to PL5 does not go well (database loss etc.)
  2. The PL5 database is subsequently lost accidentally or deliberately (many, many times in my testing)

and what might happen to different groups of users with the new PL5 strategy.

The Win 10 PL5 “special status” for PL4 DOPs helps migration but tests on the Mac appear to show that feature is missing on that release on that platform!?

Hence the (very) long post with lots of words and few pictures (they would have made the post even longer)! @sgospodarenko is there a prize for the longest post, other than getting banned from the forum!!

EDIT 01:-

I decided to check the last directory used in the PL4 test by opening in PL5. The following is a snapshot of the comparison of two snapshots taken of PL4 and PL5, both run one after the other and not simultaneously. PL5 was not via any database transfer it was simply by discovering the Photos, PL4 DOPs and .xmp sidecar files in the directory after basic navigation.

Ran another test in order to see how DPL copes witch changes and upgrades from DPL4 to DPL5.

  • added one star to test images with Canon DPP4 and saved the files (DPP writes into the .cr2 files)
  • opened test images in a cleaned DPL4 → all images presented one yellow star
  • opened test images in LrC and upgraded all images to two stars, saved .xmp sidecars
  • opened test images in a cleaned DPL4 → all images presented two yellow stars
  • then, I upgraded all images to three stars and saved .dop sidecars

This means that the images now have * in the file, ** in the .xmp sidecar and *** in the .dop sidecar

  • opened test images in a cleaned DPL5 → all images presented two stars (.dop files were ignored)
  • told DPL5 to import .dop files → all images presented three stars (.dop files were read indeed)
  • told DPL5 to read .xmp files → all images presented two stars (.xmp files were read indeed)

Again, DPL did the expected, except that DPL5 did not update the thumbnails, they kept circling arrows, unless I switched to another folder and back.

  • told DPL5 to export .dop sidecars, from there on, DPL5 presented two stars, no matter what I did
  • rewriting .dop sidecars in DPL4 brought back the three stars in DPL5 - after manual .dop import.

Summary:

  • The upgrade and change works as long as you don’t let DPL5 write .dop sidecars.
  • Destroyed ratings can be revived by exporting .dop sidecars in DPL4 (which ignores DPL5’s .dops)

Best practice

  • Do not let DPL write sidecars automatically

Well, this is maybe okay on your (and my) little island. Adopting a wider view of things, I strongly feel that DxO must (and will) fix the mess and present the user with a selection of options in cases where ratings (or other metadata) differ between different metadata holders (original files and sidecars).

@platypus

You must realise from my previous long post that the work that went into that post was all about the wider user experience and not my “own little island” at all!

I was simply stating that repatriating ‘Ratings’ to where they belong is arguably the right strategy but not by leaving users “high and dry” by deprecating the DOP features with no adequate safeguards and a safe route towards the “new tomorrow” except that PL5 is already here and has been for the last two months!

I am only too well ware of the issues after doing the work that went into the “long post” which was undertaken as a result of trying to understand all the ramifications of what was done in PL4 and what is now being done in PL5 and how those changes might affect users.

Thank you for re-iterating the conclusions I made in my “long post” it is always good to have another voice to add weight to the argument. An argument or rather discussion that I feel should be made with DxO and not with me!

There is an expression used at times which is “don’t shoot the messenger” a certain amount of which goes on in this forum, I didn’t write the code, nor engineer the faults I simply reported the faults in what I admit was a very, very long and wordy post and came to and wrote the same conclusions you have also come too.

My additional post was simply to state that I understand the strategy and my previous post stated “shame about the implementation”.

I was about to add the following to my post just above to show that in my case the two versions had arrived at the same point but were not inline with the other metadata packages I have been running except by the time I got to those packages PL5 had “re-aligned” the .xmp sidecar data with the .dop data (remember I was coming from 4.3.3 DOPs) and “destroyed” the external .xmp data!!

Had the same problem during my test but have not bothered to report so good to have it out in the open.

But most users actually use DxPL for photo editing if you do not write the DOPs then the “only” place for the preservation of the edits is the database itself.

The problem manifests itself when PL5 DOPs are discovered by PL5 and there is no corresponding entry in the database (otherwise there will be VCs if there is a mismatch) when hopefully it will import the editing and Tag data but now ignore the ‘Rating’ flag (it must be provided in an accompanying .xmp file(or embedded) which PL5 now “expects” to be present).

If there is a database entry already in PL5 for the data then that entry will either match the DOP in which case they should be identical copies of all the data or there will be a mismatch and here come the VCs but they will contain all the data in the DOP including ‘Rating’!

The real issue comes if all the metadata synchronisation is turned off (and not manually written) and the data is brought from one system to another when any rating data will be lost (it will still be in the original system PL5 database and using the metadata options will provide the necessary DOP and .xmp data required to successfully move the data.)

Do not turn off .dop sidecar creation but turn on metadata sync (with any attendant risk) or flush them to .xmp sidecar files so that PL5 has all the data it needs to hand!

My concern was that none of this was made clear at the time and has consumed large quantities of time trying to get to the bottom of the problem.

Sorry about the long post when I could have summarised it like I have attempted to here.

And get PL5 to add ‘File’/‘Sidecar’/‘Import - with Rating’

Aaaarrggghhh!!! What happened to SPOD?

ì will repeat my cry - “why not write to original RAW files?” After all, PL5 happily writes to to non-RAW files, thus making this whole mess unnecessary for them but, for some obscure fear of the bogey-man, refuses to write to the metadata part of the RAW file.

All sorts of other software does allow this and we don’t hear howls of despair as RAW files get trashed.

However, we seem to be hearing howls of despair from folks who have just lost all their hard work rating files!

But for the average user, this is not where they want to be going. All they want is to be able to rate images by clicking on a star and expect whatever version of PL, plus any other software, to be able to read that rating.

Which is why, for years now, I use my own software to write directly to both RAW and non-RAW files. Thus ensuring that whatever software I use, I always get the same answer.

Sure, prefer XMP files over direct RAW but, for pity’s sake, at least, synchronise the two.

Maybe this wasn’t a problem but, with the advent of cameras that can rate images and camera manufacturers that provide in-RAW-image editing, this has now become much more of an issue and data integrity in PL is being compromised because of an irrational fear.


BTW, macOS Finder writes Finder tags to the metadata of any file, RAW or not, in such a way that that file can be transferred to any other MAC, iPad or iPhone whilst retaining those tags.