PL4.3.6.32 Ghosting Ratings for Same image in different directories (Win10)

BHAYT · January 5, 2022, 9:32am

@sgospodarenko In a post I accused PL4.3.6.32 of “ghosting”, i.e. finding data from other images in the database with identical names. This is going to happen a lot when testing with respect to repeating tests with or without variations.

I believed after I withdrew my “allegations” that I had seen it again so I set up a test with 4 identical directories each with 4 photos. The first had ratings assigned of 4, 3, 2, 1, the second all 0s, the third 1, 2, 3, 4 and the fourth all 0s. I opened the first in PL4 and it showed them correctly but for the rest they all showed the same Ratings of 4, 3, 2, 1!

PL is not supposed to be “duplicate” image copy sensitive like some other software and certainly not in such a way that it “assigns” values from one image to the others (at least for display purposes).

Whether this happens with PL5 I will check but I need to clean up the directories first because the erroneous values are in the DOPs, i.e. PL5 would inherit the false values and then corrupt the photos depending on the setting for the ‘Sync’ option in the ‘Preferences’!

In other posts you will probably have seen calls for the ability to remove images from the database without deleting from disk. This is an important feature particularly if PL starts either corrupting the database or corrupting the view it presents (or both + DOPs and possibly embedded and sidecar ‘xmp’, the possibly statement I have not verified and it might only be a (concern) and no more).

Update 1:

I removed the PL4.3.6.32 DOPs from the 01 directories but forgot about the DOPs that had been assigned to the base directory and opened PL5 on that base directory and it hung! Terminated PL5 in Task Manager, removed DOPs in base directory and restarted PL5 and no hangs and all correct with respect to the Rating values in the various directories!

Joanna · January 5, 2022, 9:57am

Bryan, you might like to re-check this. My findings are that PL might write Rating to DOP files but it doesn’t read Rating from DOP files. It reads it first from the image if it contains a Rating, from an XMP file if that exists, or lastly from the database. At least, that’s what happens on Mac.

BHAYT · January 5, 2022, 10:16am

@joanna My tests earlier indicate the same (I forget when I wrote this post) but I will re-instate the DOPs that I removed before testing PL5 in the same way that I tested PL4, i.e. never seen directories and test PL5.

However, in my update I had a problem with PL5 which hung on the outer directory where I had copies of the photos and had missed deleting the PL4 DOPs so there may be another problem there.

I will try later because it is very cold here without the central heating, the new header tank is nearly in place but there are a few hours of plumbing to re-route the vent pipe and the overflow pipe to try to reduce the problem I have put up with since 2007 when trying to navigate the loft.

Short of moving the entire header tank platform the overflow pipes from the main tank and the header tank will continue to require either limbo or low jump tactics and I just want to remove the leaning that also went with the limbo (ducking) for the header tank overflow!

Joanna · January 5, 2022, 10:25am

We had a much simpler solution when we lived in the UK - a combi boiler for both instant hot water and central heating with no header tank - simples

BHAYT · January 5, 2022, 10:53am

@joanna The answer to that is yes and no. The boiler and the system is very old (bad with respect to boiler efficiency) and the boiler failed with a thermocouple fault and I don’t touch gas so had to wait for the heating engineer. But because we are using stored hot water we have been able to manage with convector heaters and the immersion heater until after Christmas when the boiler was repaired.

Unfortunately two old zone valves had started to leak so I fixed those but managed to block the system with the dirt from the old steel header tank. Hence the partial rebuild but I still need to get a good flow of clean water to flush out the …

While it is possible to install a combi-boiler or a sealed system boiler there are potential issues that the system is so full of dirt that no plumber would guarantee the modern (even more temperamental) brand of boilers without replacing the whole system. i.e. 13 radiators most serviced from above (concrete floors). So £2,000 possibly more for having the old boiler and flue removed and a new one installed and £5,000 or more for all the radiators and new copper + £?,??? for the re-decoration of every room and we are supposed to be moving closer to our younger son and family etc etc.

So back to my “wonderful” collection of spanners …

Take care.

BHAYT · January 6, 2022, 12:54am

Because PL5 treats PL4 DOPs as a special migration situation, “Ghosting” does cause PL5 to corrupt Photo Ratings:-

Joanna (@Joanna) unfortunately in my post Lost all ratings in PhotoLab Elite 5.1.0 ??? - #18 by BHAYT I indicated that whenever PL5 navigates to a new directory with PL4 DOPs present, the PL4 DOPs take precedence. In this case the “corruption” resulting from the “Ghosting” problem causes PL5 to overwrite the original ‘Ratings’ thereby completely corrupting that element of the meta data.

Sync was on in the Preferences deliberately.

@sgospodarenko the promised FAQ is overdue and issues like the one above make it even more important to see it and have a discussion about the “rules”. The fact that as a result of the strategy corruption occurred does not necessarily indicate a failure in the strategy but rather a failure in the code but when put together the consequences for the user could be damaging.

We (users) are already recommending more frequent backups of the PL database, the above issue would suggest that care needs to be taken with respect to backup of the images themselves. I am being over dramatic is this particular case because it is being caused by one issue in PL4 manifesting itself in PL5 in a damaging way. But once again it could be laid at the doorstep of PL4 to PL5 migration.

Currently the recommendations to anyone implementing any new release of PL must be at least to:-

Backup the database before installing any new release.
Backup the user.config file.
Turn off the ‘Sync’ option in Preferences until some testing has been undertaken (if the user uses that feature at all).
Others would recommend turning off the automatic import of DOPs etc. but that is a sure fired way of creating Virtual Copies as and when it is turned (back) on.

In the past users have never really considered any of these issues and the migrations have passed without (obvious) incident but the PL4 to PL5 and even PL5.0.1 to PL5.1 upgrades have caused incidents to at least some users.

PL5 Hangs repeatedly:-

@sgospodarenko While working my way through the test directories that were the output of the PL4 “Ghosting” tests PL5 hung on each and every directory in turn and had to be terminated! After restart the directory that had “caused” the hang could be reviewed without problem, but PL5 then hung on the next directory, all the way from directory 01, 01_2 to 01_5 i.e. 5 hangs in a row!

In other similar situations I have been sure that one directory always caused a hang only to discover that the fault appeared to have been cleared but by continually navigating the directories a hang would eventually occur.

In this particular case every directory contained PL4 DOPs.

BHAYT · January 6, 2022, 12:15pm

@joanna & @jch2103 & any other metadata Guru I have a question.

FastRawViewer looks at the ‘xmp’ sidecar first and does not seem to look at the RAW for ‘Ratings’ in this case. However, if the ‘xmp’ sidecar is missing then it takes the ‘Rating’ from the photo. Photo Express Viewer ignores the ‘xmp’ sidecar but will take ratings from the photo.

I believe that PL5 is using a hybrid strategy though how it works out the priorities of the ‘xmp’ data if in both the sidecar and the embedded I am not sure (yet more testing or @sgospodarenko someone from DxO could actually take the opportunity to clarify this issue).

Are there any standards that seek to clarify what (metadata) should be taken from where and when or is it a free for all (I have reported the PEV sidecar issue to SILKYPIX developers)?

BHAYT · January 7, 2022, 10:49am

@Joanna, @jch2103 & @sgospodarenko I did the test to see if PL5 would happily switch between embedded and sidecar ‘xmp’ ‘Ratings’ for a RAW photo happily. Something “light” to do before dismantling the pipework to the boiler chasing this blockage (it has only been 3 years)!!

The software being used was FastRawViewer (FRV) that shows both but only sets sidecar data, Photo Mechanics (PM) that has been set to change the embedded in the RAW, Fast Express Viewer (FEV) which prefers the embedded, will set the sidecar and gets hopelessly “lost” and ExifPro (EP) simply being used as a monitor program, plus Xplorer2 to show sidecar contents and file directories, all courteousy of Sticky Previews

PL5 is the only program that updates the displayed data without always needing a ‘refresh’; PM requires the thumbnail to be clicked, FRV requires a reload of the photo, EP requires an ‘F5’ and FEV ???

Steps tested:-

FEV exported a setting of 3 to the sidecar (could have used FRV, and PM but only after changing current setting, and EP but not used at this stage), PL5 picked up new sidecar immediately, others after refresh.
Deleted sidecar (changed name), PL5 required ‘F5’ to pick up the change and revert to embedded ‘Rating’ others always after refresh, FEV lost!
Change value to 3 in FRV which creates new sidecar, PL5 picks up automatically other after refresh etc. (PEV not at all started using EP after this test).
Change embedded value to 5 in PM (sidecar left unchanged at 3), PL5 auto detects others after refresh (except FEV).
Change sidecar value in FRV to 1, PL5 autodetects others after refreshes (except FEV)
Change embedded value to 5 in PM (sidecar left unchanged), PL5 autodetects others after refresh etc. etc.
Changed value to 0 in PL5, this appears to have removed the rating value in the sidecar.
Removed sidecar, PL5 shows 5 after ‘F5’, FEV restarted, FRV and PM refreshed but EP appears to be stuck and restart does not change the result?

Please note that during some of the tests it appears that PM auto-updated and it is possible that there is a parameter to change the timing of such updates?

PL5 seemed to track both PM and FRV and mostly automatically (Sync on) without using refresh and FEV and EP need work!! I have included the final snapshot in the post and attached the rest in a pdf just in case anyone is interested!?

PL5 embedded versus RAW tests 2022-01-07_01.pdf (2.9 MB)

Joanna · January 7, 2022, 12:05pm

What I would like to know is why any metadata is being written to DOP files, when the correct place would be either in the image files themselves or in XMP sidecars?

Then, this morning, I suddenly saw that DOPs is the reverse of SPOD!

As I have mentioned elsewhere, on many occasions, SPOD (single point of definition) means that any data is only ever recorded in one place, thus avoiding any synchronisation issues.

But, for some reason best known to DxO, Rating (previously known as Rank), which had nothing to do with image editing, got stuffed into DOP files, along with that image editing data.

I just ran a quick test of editing only the rating on a JPEG image with PL4… the rating was stored in Rank in the DOP.

Closing PL4 and opening the same image in PL5 doesn’t show any rating. This confirms my earlier finding that PL5 does not read a rating from the DOP file…

… or, at least, I thought it did, until I tried manually importing the sidecar from the File menu. At which point, the Rating appeared?!

So, even though my preferences are set to automatically both import and export sidecars, which works for the edits made to the actual image, in order just to see the Rating, I had to manually import the sidecar. Not exactly expected or intuitive.

Next, I added two non-hierarchical keywords to the JPEG file…

This added both keywords to the xmp:subject tag and to the xmp:hierarchicalSubject tag of the JPEG file, even though the keywords were non-hierarchical. This is totally wrong and violates the integrity of the metadata that other software may try to read.

I had sort of accepted that DxO is entitled to store keywords how they want in their database or DOP files, but writing them incorrectly into an original file is way out of order.

It also set both the xmp:rating and the exif:rating tags of the JPEG file. ** This is wrong because, if no rating tag has yet been set, it should only set the xmp:rating tag and not the legacy exif:rating tag.

Having messed up the original JPEG file metadata, it then went on to replace the Rank in the DOP file with an equivalent Rating and wrote the keywords to it in their own peculiar format…

			Keywords = {
				{
					"Couple",
				},
				{
					"Bridal",
				},
			},

So, we now have, not SPOD, but MPOD for metadata in a JPEG image.

Incorrect rating and keyword metadata in the original JPEG file
incomprehensible keyword metadata in the DOP file
yet another copy of rating and keyword metadata in the database

If I delete both the database and the DOP file, PL5 happily reads the metadata from the original JPEG file. So, my question is, why does DxO insist on writing metadata to the DOP file?

To finally prove that PL5 doesn’t read metadata from the DOP file, I closed PL5 and removed all keywords, hierarchicalKeywords and rating tags from the JPEG file, but left them in the DOP file.

I then opened the file in PL5 and found that none of the rating, keywords or hierarchicalSubject values were read in from the sidecar, even if I force read the sidecar from the File menu.

Conclusion

Bryan, it’s not surprising that you are getting incorrect ratings when PL5 is capable of making such a mess of a simple JPEG file.

platypus · January 7, 2022, 4:00pm

Single Point Of Definition makes sense imo, even if DxO writes metadata to both .xmp and .dop sidecars.

.dop sidecars serve PhotoLab and PhotoLab alone, but DPL should also make sure that information overlapping in .dop and .xmp sidecars can be handled correctly, be it automatically or offering the user respective options.

With MPOD operations, we can get different info in a) the database, b) the .dop sidecar and c) in the .xmp sidecar. In such a case, DPL should

present a popup that lists differing items, possibly with timestamps
let the user select, which item should be copied over to the other two locations
offer a way to automatically prioritise info by source, e.g. xmp > dop > database

Note that such an option will raise hell if you move or change items outside of DPL, hell getting hotter in proportion to the number of changes done outside of DPL…

Using .dop sidecars as a backup for the database is fine (I’m repeating myself here), but a few functions are missing before we get there (am I repeating myself here?)

Joanna · January 7, 2022, 4:11pm

And it is this that is really annoying me, because, at the moment, PL might write metadata to DOP files but, unlike editing data, it never reads it back from the DOP file, preferring either the original file or the XMP.

I will repeat… DOP files should be for editing data alone; metadata should only ever be written to XMP files. What is the point if it never gets read back?

According to a popular saying, “Hell hath no fury like a woman scorned”. Should we now change that to “Hell hath no fury like an XMP file changed outside of PL”?

platypus · January 7, 2022, 9:45pm

…and that’s is why we’re not there yet. I hope that DxO is going to make it all fit, before coming up with another paid update…

John7 · January 7, 2022, 9:47pm

That’s what a paid update is for!

platypus · January 7, 2022, 10:08pm

Imagine: Release a new version with 5 half-baked features. You can then fix one of them in the next paid update, which brings in another half-baked feature… That kind of policy is even worse than a subscription!

jch2103 · January 7, 2022, 10:11pm

I certainly hope someone at DxO is paying attention to the various discussions on the board about serious metadata problems currently being produced by PL5. I haven’t seen much, if any, feedback so far.

John7 · January 8, 2022, 5:11am

There has been non at all, they have been on holiday but on this and a number of other problems there has been nothing. The current version is looking like the old CorelDRAW days poor new version spend a year fixing some of the bugs and bring out a new paid for version with many more fixed ones but with even more more bugs for the next year. Many of the current problems are years old and never been fixed. I have had in V5 one bug fixed, another said to be due before March with a third left hanging that’s years old.I think the problems were made worse by the DAM and to me it look like the same problem that lead to the DXOone mess of not taking any notice of anything said about what they are doing or going.

BHAYT · January 8, 2022, 8:59am

@Joanna, @platypus, @John7, @jch2103 Just a quick update to apologise for not joining in this conversation yesterday. The central heating is back together, new header, re-arranged overflow (still need to duck), new vent (now no need to shimmy round it) but no heat flow i.e. the boiler and pump get the hot water only so far!

Talk with the plumber Thursday night about ideas to troubleshoot so removed pump and choked with baked on crud, new pump should arrive today! However, cheap spare pump didn’t achieve much so removed return pipe at the back of the boiler and removed skin on right arm about 2.5 x 1 inch in old Imperial in the process, ouch!! No baked on crud on pipe so back to the “drawing board”.

This has somewhat pre-occupied my mind and time and I am now feeling very tired but will try some tests along the lines of Joanna’s tests in a few minutes.

With respect to recent DxO reaction to the forum posts it does seem a little quiet but hopefully that will change soon and I don’t regret buying the upgrade, I simply want the product to work the way I feel that it can (and should). However, this release has been a shock to many users who have never really considered the upgrade process to carry any risks!

I may be partly to blame for the keywords being put in the DOP because I did suggest they might be useful as a source of data for recovery in one of my posts during PL5 beta testing, but I doubt it!

In truth the risks with replicake data is just that, plus when you have replicate versus replicake which one takes priority over the other. However, personally I would always prefer having too much data versus too little, providing I then have control over what is used i.e. one versus the other or the more complex additive situation, i.e. “this” from here plus “that” from there etc…

This is not a particularly complex situation just that it seems that someone(s) got it wrong.

On a high note the tests I was writing up at the same time that Joanna made her post showed DxO able to follow the rules perfectly (I think) with respect to ‘Ratings’ held in either the ‘xmp’ sidecar or the ‘xmp’ data embedded in the RAW with only the the deletion of the sidecar file requiring an ‘F5’ refresh (do you remember the ‘F5’ @jch2103?).

There remains a test to substitute an old sidecar where FRV, PM and DxO are displaying the embedded ‘Rating’ to see how they all react my hope is that it will be ignored but if the rule is always sidecar over embedded then …

BHAYT · January 8, 2022, 10:03am

@Joanna, @platypus, @John7, @jch2103 Oh dear I have “dissention in the ranks” (of the programs)!!

I changed the name of an “old” sidecar (back to .xmp) after checking that all the programs were inline and using the embedded ‘Ratings’.

PL5 and FRV used the sidecar ‘Rating’ value but PM stuck with the embedded value, so I started up Imatch and this also chose the sidecar value. please note that Imatch was started after the changes were made and not before!! Running 5 applications across three screens is a nightmare!!

Changed embedded Rating = 1 in PM and all tracked the change ignoring the ‘Rating’ = 3 in the sidecar file.
Changed ‘Rating’ = 2’ in FRV and all tracked the change, i.e. all using sidecar at this point!
Changed ‘Rating’ = 0 in PM and all tracked the change
Saved the ‘xmp’ sidecar file with name change and noted that Rating=0?
Checked values of rating in other programs still 0
Renamed old sidecar and refreshed all and only FRV shows Rating = 3!

Yet more testing required with respect to changes while running versus detection at startup (the latter is a testing nightmare because the Sticky Previews are use to capture the different programs all need to be set up again after every restart)!

Joanna · January 8, 2022, 10:35am

Ah! So you’re the culprit

Well, however it came about, it might have been useful if somebody at DxO hadn’t created the present WOM (write only memory) model, where stuff gets written but never read.

And I still don’t get why metadata needs writing to a DOP file at all. Surely, once it’s written to either the original file or an XMP sidecar, what are the chances of data loss that any other DAM wouldn’t be susceptible to? And if DxO want to write it to the database, for indexing purposes, then, at least, the data will only be in two places instead of three and if the database corrupts, it can always be rebuilt from original files and/or XMP sidecars.

I think someone got into the “make a change, write it to the DOP” mentality without thinking about separation of concerns.

Don’t shout too soon

I see your tests and I’ll raise you some of mine

Seriously, I’m just in the middle of documenting how a JPEG file gets treated and, in the process, I’ll try and put together a testing strategy, hopefully covering all possible possibilities, both for JPEG and RAW/XMP scenarios, if that will help?

Now to read your latest post.

BHAYT · January 8, 2022, 10:55am

My posts are so long I am not sure anyone reads them anyway!!

In truth there is a problem even with ‘xmp’ sidecar versus embedded ‘xmp’ let alone DOP sidecar versus embedded PL5db! The main problem is the proud insistence of DxO with respect to immutability pre PL5.

So we have the need to store keywords, Ratings (Rank) and the Tag somewhere in PL4. The keywords went into the database along with the Exif data and the ‘Rank’ and Tag went into the database and the DOP. Hence the photo carried the Exif etc. and the DOP carried the editing, Rating (Rank) and Tag and the keywords went down with the database!!

It is actually logical that you extend that philosophy to the keywords so they don’t sink with the ship (database) except that with the change in immutability the keywords already have a “universally” accepted sidecar in the shape of the ‘xmp’ sidecar for RAW images and the image file itself for RGB images so the DOP becomes redundant for that role!

Basically putting the keywords in the DOP was at least a release too late and the Ratings could have been removed from the DOP in PL5 because they are either in the image file or the ‘xmp’ sidecar but what would have happened to backwards compatibility!? In truth the final PL4 release should have included as much backward compatibility with PL5 DOPs as practical (in which case all the elements should still have been in the DOP etc. etc.).

As far as I can tell the Tag has no ‘xmp’ counterpart and PL5 does not use the “pretty” colour scheme at all!!

With PL4 the final test shows Rating=3 taken from the sidecar (the PL5 DOP shows Rating=0) and all the ratings are in a “pretty shade of yellow”!

Further Update:-

I changed Rating=5 in PM and PL4 immediately detected and showed the change!

On restart PL5 found Rating=5.
Close PL5 set Rating=2 in FRV (i.e. sidecar Rating=2).
On restart PL5 found Rating=2 but without the pretty yellow!