PL5 Tag Field not read from .dop file

@Johnny @Joanna @sgospodarenko Sorry I have been “sleeping” on the job and not testing as I should. Now to make amends.

Took one of my previous group of test photos and copied to a new directory. These are a mix of JPGs, RAW (RW2 and ORF) and the photos contained the same ‘Ratings’ as the DOPs as a result of the tests that I had run.

  1. Used FRV to clear all ‘Ratings’ so only the DOPs carried the previous ‘Ratings’.

  2. Opened PL5 and navigated to the new directory containing photos with no embedded or sidecar ‘Ratings’ and DOPs with various ‘Ratings’ set.

  3. Result - NOTHING not a ‘Rating’ in sight, nowhere (except in the DOP). Sorry PL5 has just “trashed” all the DOPs as well, Rating=0 in all cases.

I consider this to be a bug although I have to ask why your ‘Ratings’ were only in the DOPs but if you were running with the ‘Preferences’ ‘Sync’ option off and never used the ‘File’/‘Metadata’/‘Write to Image’ then the DOP is the only place they would reside!!!

I repeated the test by copying the original DOPs back into the directory and got the “looked” for Virtual Copies (VCs) which show what the result should have been if PL5 had done it correctly in the first place.

[M] represents what PL5 did and [1] represents what is should have done!

I am afraid that I was testing some of this following a list of scenarios when I encountered the PL4 Ghosting issue and wound up down the keyword “rabbit-hole” plus others. I must stop trying to “fix” a problem but actually get to the actual symptoms of a problem and then re-create it. However, many of the symptoms reported in the posts must “chime” with the developers and it would be good if that info is fed back to the posts so we don’t go chasing “ghosts” (“pun” intended).

Please get the restricted forum entry problem fixed @sgospodarenko it is “cramping” my input :upside_down_face:

The problem is that not everybody goes to that extent and, for those of us who expects it to “just work”, it is causing problems.

That procedure shouldn’t be necessary or required

1 Like

A good work around @platypus but it should not be necessary. @Joanna it almost looks as if they have implemented the PL5 code for PL4 DOPs (on the Mac) and as my post above suggests this is also a bug, so you may have two bugs for the price of one or just two bugs if your PL5 import does the same as I am suggesting in the above test @sgospodarenko.

Fully agree. But my settings would have saved my bacon…
Automatic things are nice if they work. In this field, they don’t.

Conclusion

  • DxO needs to fix this bug (ceterum censeo…)

@joanna Absolutely!!

Truly bizarre!

In my tests PL5 has not updated the PL4 DOPS after using them to set the database, but I haven’t checked how many restarts of PL5 it takes before PL5 starts using its own DOPs! In my test above it is quick to replace the 5.1.2 DOP with its own 5.1.2 DOP where all Rating=0, thereby destroying user work!!

I am using PhotoLibrary throughout and setting no keywords during these tests, Ratings only. From other tests ‘Tags’ are a whole new “can of worms”!

I promise I will complete the spreadsheets update for keywords by the end of the weekend, i.e. ACDSee keywords imported and updated (synced to the original photo) by the package and ACDSee imported and a keyword added by each package, Bridge appears to be the biggest “failure” with the last one but I will explain that in the other (also inappropriate, but at least “my” topic) rather than here.

@Musashi, @sgospodarenko Thank you for your post and for providing a sneak preview of elements of the FAQ but I am afraid I strongly disagree with your opening statement which according to my tests is positively the wrong thing to do!!

Namely,

“Since version 5, PhotoLab now reads rating from .xmp files and no more from .dop files”

  1. This statement is factually incomplete, PL5 reads from .xmp sidecar files and from the embedded ‘xmp’ data contained within the photo (RGB or RAW format) and no more from .dop sidecar files.

  2. This statement casts many users coming from using PL4 adrift by actually changing the way that ‘Ratings’ data is handled and may require users to change their work flow. This change should have been highlighted at the time of the release not months later.

  3. Arguably worse than any of that, having collected the ‘Ratings’ data from a new ‘xmp’ source PL5 then “destroys” the data in the DOP, potentially eradicating that data forever. Unfortunately users have been used to DxPL never destroying their data from release to release until along comes PL5 and many may not have backups (because there has never been any need for them, until now)!

  4. The PL4 DOPs get “special” treatment, at least on Win 10, according to my testing but Joanna’s testing on the Mac shows that the Mac release is apparently ignoring the PL4 DOPs i.e. apparently adopting the new, PL5 strategy! The handling of PL4 .dop sidecars is not shown in the table.

I ran a test on PL4.3.6 which turned out to be something of a “mixed bag”! Why did I run such a test on a previous release and the answer is because I thought it would be quick and give some insight into the upgrade process and the thinking behind the PL5 strategy. Simple it was not!!

  1. PL4 can read the ‘Rating’ (Rank) from either the .dop or the .xmp sidecar file (if one exists) with RAW photos, I didn’t test it with embedded ‘xmp’ metadata (laziness).

  2. I started with all 4 RAW files containing no ‘Ratings’ (Rank), actually PL4 refers to ‘Rating’ in the product but stores as ‘Rank’ in the .dop sidecar.

  3. I added ‘Ratings’ in PL4 and .dop sidecar with the correct ‘Rank’ files were created with the correct ‘Rank’but no .xmp sidecar files were added. With this test the DOPs were created when I closed PL4.

  4. I deleted the images in PL4 and added them back and assigned ‘Rating’ values in FRV which only creates .xmp sidecar files for RAW images.

  5. PL4 took the ‘Rating’ from the .xmp sidecar and showed them in yellow, no DOPs were created at this point (no other edits had been made)

  6. Changing the .xmp ‘Ratings’ externally was immediately picked up by PL4 and shown correctly.

  7. Setting the ‘Tag’ in PL5 resulted in the creation of the .dop sidecar (DOP) to hold the new flag but the DOPs had Rank= 0 at this point! Is there a flag in the DOP to tell PL4 that the ‘Rating’(Rank) if being externally set? If the ‘xmp sidecar is lost at this point the DOP will be of no use for recovery.

  8. I changed the ratings of two of the photos externally and the DOP of those two photos changed, the ‘xmp sidecars remained unchanged (PL4 immutability at work perhaps) and the “star” colour changed from yellow to white.

  9. I changed the values of the ‘Ratings’ in all photos externally as might be guessed those with PL4 assigned values showed no change but the other two were changed in line with the external product used

  10. Mixing and matching PL4 ‘Rating’ assignment with external assignment means that ‘Ratings’(Rank) is spread between sidecars types, i.e. .dop sidecars and .xmp sidecars

So at the end of the test PL4 might believe one thing and the external program another if the two methods of assigning ‘Ratings’ are used!

More importantly PL5 needs to be able to handle the situation of this potential mixture of sidecar files in the migration from PL4 to PL5. Arguably PL5s initial source of all such data is the PL4 database but as some of us found that upgrade didn’t go as well as it might have and the database was “lost” to PL5 during the upgrade . PL4 keywords were only held in the database (as far as my tests show).

I managed to recover my database but the loss of the database came as a real shock to longtime DxPL users.

Effectively prior to PL5 the 'Ratings’ (Rank) field was not held external to the PL5 system because of the “immutability” mandate of pre-PL5 releases with respect to JPGs etc so the .dop sidecar was treated as the primary (or only) source. PL4 did not write the ‘Rating’ (Rank) to the .xmp sidecar file as well as the .dop sidecar file as I thought earlier in my testing so there is now a potential conflict between the differing data that can be held.

Hence, many users coming from that background were not really treating ‘Rating’ (Rank) as anything other than DxPL metadata held in the DxPL database and, for portability, in the .dop sidecar.

I am afraid that I would suggest that such an expectation would be carried forward by those users into PL5.

Suddenly changing the rules because the ‘xmp’ data is now available to PL5 for update and that is, arguably, “where that data belongs” is unacceptable without any suitable warning. Forcing a change in work flow is arguably unacceptable (Skylum tend to do it often), changing it without adequate warning is …!!

Hence, taking a null or 0 value from the ‘xmp’ embedded’ or sidecar data is rather short sighted because of the heritage of PL4 which many users are coming from. Users may not want to use PL5 ‘xmp’ metadata and actually considered ‘Rating’ (Rank) as DxPL metadata.

The decision has effectively cut DOP ‘Ratings’ “loose” and made it a WOM (Write Only Memory) field like the keywords which are also stored in the DOP.

However, the important thing is that the old PL4 processing flow involved both the .dop and the .xmp sidecar files and and the new PL5 processing flow completely ignores the .dop with respect to ‘Ratings’.

Given that PL5 cannot distinguish between a Null and a 0 ‘Rating’ entry, plus the fact that NEF and my Panasonic camera set the value to 0 in the original photos the adopted strategy is doomed to corrupt the data of unsuspecting users, which is exactly what has happened.

The current PL5 algorithm is wrong, in my opinion, it needs to be more sophisticated in handling the data potentially stored in .dop sidecars, .xmp sidecars (for RAWs) and embedded in the photo for (RGB and potentially RAWs as well).

In addition it is possible that a different set of rules should be in force for the first introduction of a directory to PL5 (i.e. the database) versus subsequent navigation to the directory. However this still poses the problem if the .xmp data (sidecar and/or embedded) contains any ‘Rating’ information! Effectively the metadata dates should be taken into account and the .dop sidecar should also be considered as a candidate for the data, certainly on the first discovery of the directory and, potentially every time!

Therefore discover the potential Rating data available and rank (not Rank) that data based upon the date/timestamp from latest to first, i.e. determine the latest data available and that should include keywords, GPS data etc… (I haven’t tested them in this context at all yet!!)

While one alternative solution might be to force a write of the .xmp sidecar data even when the ‘Preferences ‘Sync’ option is off and the customer has not (and possibly never will) use the ‘Metadata’ ‘Read from disk’ and ‘Write to disk’ commands PL5 could force a write to secure the ‘Ratings’ in the .xmp sidecar but users needs to be aware that these need to be secured in any backups and that they have value and that doesn’t help any users currently “trapped” in the situation of PL5 DOPs with ‘Ratings’ in them!

The current algorithm is way too simplistic (I feel like saying naive but that would be rude), it does not take into account the potential loss of the database (in the PL4 to PL5 upgrade or later because the database becomes damaged/lost etc.). It fails to cope with the possibility that users may not embrace the new features of PL5 and start using PL5’s new found “powers” of being able to store metadata and continue to believe in the power of the DOP as a backup for the database!

Particularly if there was no timely warning about the potential consequences of the PL5 changes.

Please change the strategy and/or provide a ‘Preference’ to allow PL5 to use the PL4 strategy and please discuss this with the users, you created the product and it is excellent but we use it, many on a daily basis, in a wide variety of ways and we might just have something constructive and useful to say!

2 Likes

Wow! The large edit window is back in style! :laughing:

I’m off out for a walk and shall digest it more fully when I return. From a quick glance, it doesn’t look good from a potential data loss point of view.

@joanna Have a nice walk or hope you have had a nice walk.

That one was done on a word processing package (before the fix) so that I could keep notes as I ran tests on PL4 (trying to understand the legacy of the past and its potential influence on PL5, if any, and the “baggage” that might accrue from the past in terms of data and the “habits” and “expectations” of the users).

There is a little (!?) “second-guessing” but that is inevitable given the nature of testing software in a partial vacuum.

@musashi and @sgospodarenko

I feel that I need to add a comment to my excessivley large post above.

Basically I consider downgrading the ‘Rating’ in the DOP (.dop sidecar) is actually the correct thing to do but then I would say that because I have never used ‘Rating’(Rank) in PL4 and may or may not use it in PL5.

The tests that I ran were to try acquaint myself with the working of the previous version and help me to understand the legacy of PL4 ‘Ratings’(Rank) and what the implications for users coming from that background might be, plus what might the implications of that change be in less favourable circumstances e.g. (I should have used i.e. but e.g. covers the case that I might miss something!)

  1. The transition from PL4 to PL5 does not go well (database loss etc.)
  2. The PL5 database is subsequently lost accidentally or deliberately (many, many times in my testing)

and what might happen to different groups of users with the new PL5 strategy.

The Win 10 PL5 “special status” for PL4 DOPs helps migration but tests on the Mac appear to show that feature is missing on that release on that platform!?

Hence the (very) long post with lots of words and few pictures (they would have made the post even longer)! @sgospodarenko is there a prize for the longest post, other than getting banned from the forum!!

EDIT 01:-

I decided to check the last directory used in the PL4 test by opening in PL5. The following is a snapshot of the comparison of two snapshots taken of PL4 and PL5, both run one after the other and not simultaneously. PL5 was not via any database transfer it was simply by discovering the Photos, PL4 DOPs and .xmp sidecar files in the directory after basic navigation.

Ran another test in order to see how DPL copes witch changes and upgrades from DPL4 to DPL5.

  • added one star to test images with Canon DPP4 and saved the files (DPP writes into the .cr2 files)
  • opened test images in a cleaned DPL4 → all images presented one yellow star
  • opened test images in LrC and upgraded all images to two stars, saved .xmp sidecars
  • opened test images in a cleaned DPL4 → all images presented two yellow stars
  • then, I upgraded all images to three stars and saved .dop sidecars

This means that the images now have * in the file, ** in the .xmp sidecar and *** in the .dop sidecar

  • opened test images in a cleaned DPL5 → all images presented two stars (.dop files were ignored)
  • told DPL5 to import .dop files → all images presented three stars (.dop files were read indeed)
  • told DPL5 to read .xmp files → all images presented two stars (.xmp files were read indeed)

Again, DPL did the expected, except that DPL5 did not update the thumbnails, they kept circling arrows, unless I switched to another folder and back.

  • told DPL5 to export .dop sidecars, from there on, DPL5 presented two stars, no matter what I did
  • rewriting .dop sidecars in DPL4 brought back the three stars in DPL5 - after manual .dop import.

Summary:

  • The upgrade and change works as long as you don’t let DPL5 write .dop sidecars.
  • Destroyed ratings can be revived by exporting .dop sidecars in DPL4 (which ignores DPL5’s .dops)

Best practice

  • Do not let DPL write sidecars automatically

Well, this is maybe okay on your (and my) little island. Adopting a wider view of things, I strongly feel that DxO must (and will) fix the mess and present the user with a selection of options in cases where ratings (or other metadata) differ between different metadata holders (original files and sidecars).

@platypus

You must realise from my previous long post that the work that went into that post was all about the wider user experience and not my “own little island” at all!

I was simply stating that repatriating ‘Ratings’ to where they belong is arguably the right strategy but not by leaving users “high and dry” by deprecating the DOP features with no adequate safeguards and a safe route towards the “new tomorrow” except that PL5 is already here and has been for the last two months!

I am only too well ware of the issues after doing the work that went into the “long post” which was undertaken as a result of trying to understand all the ramifications of what was done in PL4 and what is now being done in PL5 and how those changes might affect users.

Thank you for re-iterating the conclusions I made in my “long post” it is always good to have another voice to add weight to the argument. An argument or rather discussion that I feel should be made with DxO and not with me!

There is an expression used at times which is “don’t shoot the messenger” a certain amount of which goes on in this forum, I didn’t write the code, nor engineer the faults I simply reported the faults in what I admit was a very, very long and wordy post and came to and wrote the same conclusions you have also come too.

My additional post was simply to state that I understand the strategy and my previous post stated “shame about the implementation”.

I was about to add the following to my post just above to show that in my case the two versions had arrived at the same point but were not inline with the other metadata packages I have been running except by the time I got to those packages PL5 had “re-aligned” the .xmp sidecar data with the .dop data (remember I was coming from 4.3.3 DOPs) and “destroyed” the external .xmp data!!

Had the same problem during my test but have not bothered to report so good to have it out in the open.

But most users actually use DxPL for photo editing if you do not write the DOPs then the “only” place for the preservation of the edits is the database itself.

The problem manifests itself when PL5 DOPs are discovered by PL5 and there is no corresponding entry in the database (otherwise there will be VCs if there is a mismatch) when hopefully it will import the editing and Tag data but now ignore the ‘Rating’ flag (it must be provided in an accompanying .xmp file(or embedded) which PL5 now “expects” to be present).

If there is a database entry already in PL5 for the data then that entry will either match the DOP in which case they should be identical copies of all the data or there will be a mismatch and here come the VCs but they will contain all the data in the DOP including ‘Rating’!

The real issue comes if all the metadata synchronisation is turned off (and not manually written) and the data is brought from one system to another when any rating data will be lost (it will still be in the original system PL5 database and using the metadata options will provide the necessary DOP and .xmp data required to successfully move the data.)

Do not turn off .dop sidecar creation but turn on metadata sync (with any attendant risk) or flush them to .xmp sidecar files so that PL5 has all the data it needs to hand!

My concern was that none of this was made clear at the time and has consumed large quantities of time trying to get to the bottom of the problem.

Sorry about the long post when I could have summarised it like I have attempted to here.

And get PL5 to add ‘File’/‘Sidecar’/‘Import - with Rating’

Aaaarrggghhh!!! What happened to SPOD?

ì will repeat my cry - “why not write to original RAW files?” After all, PL5 happily writes to to non-RAW files, thus making this whole mess unnecessary for them but, for some obscure fear of the bogey-man, refuses to write to the metadata part of the RAW file.

All sorts of other software does allow this and we don’t hear howls of despair as RAW files get trashed.

However, we seem to be hearing howls of despair from folks who have just lost all their hard work rating files!

But for the average user, this is not where they want to be going. All they want is to be able to rate images by clicking on a star and expect whatever version of PL, plus any other software, to be able to read that rating.

Which is why, for years now, I use my own software to write directly to both RAW and non-RAW files. Thus ensuring that whatever software I use, I always get the same answer.

Sure, prefer XMP files over direct RAW but, for pity’s sake, at least, synchronise the two.

Maybe this wasn’t a problem but, with the advent of cameras that can rate images and camera manufacturers that provide in-RAW-image editing, this has now become much more of an issue and data integrity in PL is being compromised because of an irrational fear.


BTW, macOS Finder writes Finder tags to the metadata of any file, RAW or not, in such a way that that file can be transferred to any other MAC, iPad or iPhone whilst retaining those tags.

@joanna Unfortunately I never felt the urge to start writing code for the PC, I have two Raspberry Pi’s that I persuaded one of my sons to buy me for Birthday presents on two occasions plus a couple of Arduino’s and they are still in their boxes so I must “put up” with other peoples coding and “like it or lump it” or complain (as politely as I can muster).

Once again I don’t think that the proliferation of data is as much to blame as the original immutability decision of DxO which sent the product down the wrong path and elements of the transition from that state to a more “normal” state are what is causing the problems now.

Sorry about the long post that came because I needed to navigate my way through what I thought I was seeing and left all the “workings out” in the post. Lets see what I would like to see happen (you may object to some or all but …)

  1. Add a ‘File’/‘Sidecar’/‘Import - with Rating’
  2. Add a ‘File’/‘Sidecar’/‘Import - Keywords’ or ‘Import - Keywords & Rating’ or provide a checklist of what is on offer to import etc.

The main thing is to provide “experienced” (not a good description but ) users with options while not overly complicating the product for those that want a good basic RAW editor and are not interested in all the “bells and whistles”.

In the meantime users need to

  1. Be careful about presenting photos with PL4 DOPs to PL5 because they may cause external metadata (certainly Ratings) to be overwritten in .xmp sidecar files (if they exist).

  2. Be careful about presenting photos with PL5 DOPs to PL5 as new imports if the user has been setting ‘Ratings’ and the DOP is not accompanied by an associated .xmp sidecar file or as embedded xmp in the photo.

Currently presenting such files in 2 above will result in the data in the DOP being ignored and data from the .xmp sidecar (if any) being used or data from the Photo being used .

Since the latter (data from the photo) is typically undefined or 0 that will result in a ‘Rating’ of 0 which will be written to the database and back to the DOP and the original ‘Rating’ will be lost (except, hopefully back in the database on the system that created the DOP in the first place, unless that DOP came from the only system which lost its database and that is why the data is being presented ).

It is the last (Recovery) scenario which is the most worrying, when a PL5 DOP represents the only source of the ‘Rating’ data then the only resort is a return to the PL4 DOP if it exists and any new work undertaken on PL5 since that time risks being lost. The decision then is to either use the DOPs and (re)gain the editing data but lose the ratings or don’t use the DOPs and lose both.

Before using any DOPs in this situation

  1. Dump the database

  2. Make a copy of the DOPs because they can be used to recover the data, if only manually.

  3. If you are good at programming/scripting you need to take the ‘Ratings’ data out of the DOP and change one line in the ‘xmp’ sidecar that I included in a previous post

Or we need to persuade some kind person at DxO to create such a program @sgospodarenko until changes are made to the product to tackle this problem properly!

PS:

Joanna. @platypus is a Mac user (?) but the description of moving data from PL4 to PL5 seems to suggest that the PL4 DOP can be used but requires an explicit sidecar ‘import’ on PL5 rather than the automatic discovery I get on the Win 10 product does that fit with your testing if it does then the “only” issue of the Mac is a lack of automatic updating (I think or have I missed something? can’t test don’t have a Mac).

Two things spring to mind with this thread, the lack of any DXO input (though hopefully its being watched, but…) and DXO have never been very good at backward compatibility. Now this has moved over into the DAM side and its clearly creating a lot of problems. But this I regret merely fits in with doing this on the RAW process side. This lack of warning for those effected compared to Photo Supreme where an update that is going to change the data base has warnings before you run it is stark.

@John7 I sort of agree I think that DxO had not subjected the re-design to all the possible scenarios, i.e. they had missed at least one and sadly that exposed what then appears to be an “obvious” flaw but it took some time to work out why that “flaw” might exist given that others had experienced the flaw directly and I and others had detected the problem by testing.

I was particularly interested in how users could “fall” into the “trap” not just in the nature of the “trap” itself. Once found it is “obvious” but that is the nature of so many “bugs”, when found you “kick” yourself for not realising earlier and if you are the coder you hope that the fault is detected before the software is released.

With respect to a DxO response if you look at the amount of words written in these topics is that really so surprising, albeit a bit disappointing sometimes!

I would like to see none of the above. There should be absolutely no need to “import” metadata from sidecars.

According to the rest of the world, metadata can be held in either the image file (even if it is RAW) or an XMP sidecar file.

Unfortunately, PL4 used DOP files for some metadata and the database for other. And this is where it gets decidedly nasty.

I have never tried it but I would hope that installing PL5 for the first time would have imported the PL4 database and updated DOP files at one at the same time. The problem being that it would cause immediate loss of backwards compatibility to PL4.


Do you know what? I’m am getting tired of arguing this through, since nobody apart from us enthusiasts are doing any of the talking and nothing is getting said from DxO. Maybe that will change when the beta starts.

In the meanwhile, the “sneak preview” of the FAQ basically says, “We’ve changed things and you are going to lose your ratings unless you happen to discover this FAQ before you installed PL5” Something which is unlikely to happen because folks rarely look in the FAQ before they find a problem.

At the moment, the truth is that, unless you are totally tech-savvy and aware of the problems, which shouldn’t happen, in simply closing a previous version and opening a new version of PL, you are never going to make it unscathed.

Mind you, judging by the reactions to PL5’s metadata and keywording functionality in these fora, not many are making much use of it, with the vast majority using external DAMs, rather then risk their considerable investment in time and energy, setting up their assets so far.

I started writing my “mini-DAM” on the 21st July 2019, before DxO even mentioned the idea.

It all started as a simple project to provide an image browser whose folder hierarchies were flattened, in order to avoid having to keep on choosing sub-folders in date nested structures.

I mentioned it to DxO - no interest.

Then a couple of my beta testers suggested adding keywording and searching, which I have done, very successfully. A user doesn’t have to index their folders before searching because indexing is always going on on the background, using Apple’s Spotlight mechanism.

I don’t have a database - I use Spotlight in coordination with writing metadata to either original files and/or XMP files. And if the user chooses to write to both RAW and XMP, they are kept in sync with each other.

I have never had a data loss or image file corruption in all that time and, unless DxO can match that, I will be sticking to my app for everything not directly image related.

I am sure that DxO is reading this longish thread with interest and will try to come up with a solution real soon, my only fear is that DxO will simply fix one automatic issue with another one…

Single Point Of Definition? Good concept…but as change happens, the point will come when we need or want to move and then, we need control - or a spotless automaton.

Throwing cabbage will help DxO to cope with cabbage…