Lost all ratings in PhotoLab Elite 5.1.0?

Hi Joanna,

Projects: sorry, mix-up — I’m simply importing my photos into a folder hierarchy and then process them with PL — so far, I have not using projects. I have placed a set of pics with dop files in my dropbox, at Dropbox - DxO - Simplify your life — the files are organized as follows:

  • 2018/PL 2.2.2 — 3 photos with original PL 2.2.2 dop sidecars
  • 2018/post PL 5.1.0 — same 3 photos and dop sidecars, but after opening the folder with PL 5.1.0
  • 2021/PL 5.0.2 — 3 photos with original PL 5.0.2 dop sidecars
  • 2021/post PL 5.1.0 — same 3 photos and dop sidecars, but after opening the folder with PL 5.1.0

I hope this helps?

OK. I’ll take a look

1 Like

sorry, my bad: in my first attempt, all files in “2021” had the rating already reset to 0 (took them from a “bad” source directory). I have updated the files in “2021/PL 5.0.2” a few moments ago (16:49).

OK Rolf.

I have run a couple of tests and found that PL5 is definitely screwing up any ratings previously held in the DOP file under the “Rating” tag.

The tag remains intact until you change anything in the image, when it gets overwritten to whatever Rating value happens to be in either the NEF file or an accompanying XMP file.

It would seem that PL5 takes absolutely no notice of anything previously written to the DOP file unless it is also written to either the NEF file or an XMP file. Therefore, because your NEF file contains a default value of 0, as soon as you make an edit in PL5, the DOP file gets written and the 0 Rating value is copied from the NEF file.

This is definitely a bug, or at least unforeseen behaviour. Whatever you do, don’t risk any more data loss until you have a full backup of all valid files and this is fixed.

@sgospodarenko this seems a fairly urgent problem. Who are we best talking to about getting this fixed PDQ?

1 Like

Thanks a lot, Joanna! I’m looking forward to getting this fixed in the hopefully not too distant future! Best wishes & merry Xmas, -Rolf

@RKyburz and @Joanna I made comments in a similar but not identical Topic PL5 Tag Field not read from .dop file - #23 by Crispy and @sgospodarenko made a comment about a forthcoming FAQ to better understand the processes

and

  • Well the algorithm of the prioritization of the files containing this data has been changed with the introduction on xmp support (it also depends if it’s a master image or VC). Everything will be explained in details in FAQ very soon.

Regards,
Svetlana G.
[/quote]

I had originally intended to stop testing for a bit and concentrate on other things but decided to have one last try at this (more fool me!!). I cannot yet repeat @Joanna’s findings but believe I have seen something similar (I have done so many tests I believe I have seen a photo of a UFO somewhere amongst the tests!!??) .

I have run some tests which are a little worrying (to be posted) but decided to use some test photos in a new directory structure and use PL4 to set up certain scenarios and then import that database into PL5 and continue testing (all tests up till now have been made using PL4 and then accessing the DOPs and photos using by PL5; they have not included an imported/upgraded database).

However @sgospodarenko this testing “fell at the first hurdle” when I opened the photos in PL4.3.6.32 and as the snapshot shows the photos have Ratings (Ranking) externally assigned. I have seen this before during testing but this is an entirely new directory and FRV, PM, ExifPro and Exif Pilot all show no ‘Rating’ present in the photos!?

While it is possible that I am looking at the wrong directory in PL4 could it also be that the photo entries have “bled” from elsewhere in the PL4 database (the photos have been used many, many … times?)

From here perhaps

I will abandon testing with this set of photos and pick some that have never been used in any tests before and continue trying to test what I set out to test.

UPDATE 1:-

@sgospodarenko I apologise for suggesting a database “bleed through” I re-initialized the PL4 database and the issue still persists. I also enlisted Imatch and it also shows no ‘Ratings’ set; so where is PL4 getting the data from!? Comparing the different photos (Beyond Compare) the only field that I could see that might relate to the problem was ‘Rating Percentage’ which showed 1, 25 and 50 (for photos that have a Rank of 1, 2 & 3) while the actual ‘Rating’ in all cases showed as 0!!

Opening the directory in PL5 shows no ‘Ratings’ at all (in the absence of any PL4 DOPs), which is in-line with the other programs. My tests have indicated that PL5 will import the DOP values for ‘Rating’ (‘Rank’) in preference to those for ‘Rating’ in any photo, if the DOP exists. This is inline with the fact that JPGs are considered “immutable” in PL4 and so the photo will not be changed, any Tag, Rank and keyword changes will only be stored in PL4db and the DOP.

Hence for a photo with a PL4 DOP the (PL4) DOP appears to be taken as the only source of ‘Rating’ data whereas I would contend it should be taken as the “prime” source and any “competing” photo data must be analysed rather than just dismissed.

By analysed I mean that the algorithm needs to be “clever” enough to determine if there have been changes made by other software e.g. LR, PM Imatch etc. that post dates the PL4 database (hence my planned tests) or post dates the DOP which I have already tested and it seems to be that the DOP will always be preferred BUT I have not experimented with date/timestamps except that all my Photo changes should (but may not) supersede the DOP data!!

The issue of “immutability” was always treated by DxO as almost something “sacred” but it means that interworking with other software was a potential nightmare and further complicates the transition from PL4 to PL5!

UPDATE 2:-

Getting PL5 to ‘Metadata’/‘Write to Image’ made the ‘Rating percentage’ vanish and PL4 picked up the Photos without any ‘Rank’. However, because I had not made any changes to the metadata in PL5 the ‘sync’ option did not make any “routine” writes to the photo and the manual write was required.

Just a reminder to everyone: Nikon (and perhaps others) writes a default value of ‘0’ into the Ratings tag in NEF files. (One can change this in the camera to a preferred Rating, although I don’t know how many people actually to this.) Any program reading and writing metadata needs to take this default value into account. Otherwise, ratings added in other software (including the software doing the reading!) can be overwritten by the default ‘0’ value.

It appears that PL5 is causing this exact problem. As I and others (e.g., @joanna) have said before, metadata is a mess. DxO needs to proceed very carefully to avoid creating major problems for users.

@jch2103, @Joanna, @RKyburz, @sgospodarenko

@jch2103 you and others have been concerned about the standards adopted for handling metadata in nPL5. While I am more concerned about how it seems to go about certain tasks.

The FAQ to be published should give us a clue to the way that the software engineers think but in the meantime I have been making a few tests.

So the scenarios that need to be handled are (I believe) ;

Scenario 1 - Database Only
Scenario 2 - Database & DOP
Scenario 3 - Database & DOP & Photo
Scenario 4 - Database & Photo
Scenario 5 - DOP only
Scenario 6 - DOP & Photo
Scenario 7 - Photo Only
Scenario 8 - Empty DOP & Photo

In all of the above the photo must be present but those identified as including “Photo” mean Photos with e.g. ‘Rating’ set in the photo metadata.

In only one of the above scenarios were the values in the Photo actually used, namely scenario 7 (this was written before I added Scenario 8). Unfortunately this assumes that PL4 was working in a “vacuum” or recognised any and all changes to the Photo as they occurred and noted them in the database and DOP.

Firstly I consider this model too simplistic because there is a risk that data in the Photos will be “lost” unless a forced metadata read is made (not a bad idea) but worse it seems to run contrary to the complaints that have been made about loss of ‘tags’ and ‘Rank’ (‘Ratings’) in the move from PL4 to PL5!!

According to my tests the only situation where data would be “lost” is if PL5 thought that a PL4 photo was actually to be imported, since this is devoid of any of the metadata that we are looking for, i.e. the photo is devoid of any data because of PL4 immutability so there is nothing to import into PL5.

Given that in all the scenarios I reviewed only one took anything from the photo and in all the others PL4 data was always there I fail to see what else could be going on (though that could just be me being short sighted)!!

That still doesn’t explain why it works for some users and not others. I used none of the options(Tags etc) when using PL? for my own work, so when my database went missing during the upgrade I didn’t really care but managed to salvage it anyway!

The rules do change once PL5 is working with its “own” database records and DOPs, the Photo becomes the major source of the metadata (though that depends on the Sync option and the manual reads and writes). This data will be missing if the photo is a PL4 photo and that might point to how PL5 starts “losing” data. What triggers a transition from the data being treated as old PL4 to new PL5 I am not sure, in the DOP it is clearly identifiable but in the database? My tests also indicated that the presence of the PL4 DOP is sufficient to cause the data to be taken from the DOP rather than the photo.

I haven’t tested a mix of the scenarios in one directory/sub-directory.

@sgospodarenko Unfortunately I decided to copy the tests that start with a PL4 database so that I can repeat those pertinent with a PL5 database that has already “seen” the PL4 data. Unfortunately, I accidentally killed a copy and was left with a right mess of a pair of directories. I put the data back together (or so I thought) and hung PL5 right from the start. Eventually I restarted the machine and restored the directories but now PL5 is hanging while attempting to read the directories and display the photos.

More work is needed to get things back the way that they were and avoid the following (see snapshots), a job for another day!!



Can you clarify which data you were testing? Rating? And/or other tags? And comparing input and output files for content?

Thanks.

@jch2103 Sorry during the tests I have been looking at ‘Ratings’ (PL4 'Rank) I should have made that clear. Now I have got to clean up the mess (side-line the directories and recreate from scratch) so that I can do the tests again and verify what I thought I saw was correct.

And herein lies the problem.

One of the basic principles of good reliable software design is SPOD (single point of definition)

Something that should be becoming more and more important as DxO try to move into the metadata market.

I have written elsewhere about the mess that keywords have become in PL5 but, here, I want to focus on the “simple” subject of the Rating tag.

Not considering a DAM, there are two possible places that the Rating tag can be stored…

  • in the image file
  • in an XMP file

Now, there is a common phobia about storing metadata in RAW files and yet this is something that some software does without problem.

This then means that we are forced to use XMP sidecar files for RAW images, which then means that, for PL5, we can now have the rating recorded in…

  • the RAW file by the camera
  • the XMP file by some other DAM
  • the DOP file by PL5
  • the PL5 database

Extensive testing, however, proves that the rating might get written to the DOP file but it never gets read from the DOP file and from a quick search of the database, even thought the relevant column is still marked “ZRANK” it doesn’t contain a value that equates to that written to the DOP file.

Certainly, for users who have used previous versions of PL to rate images, there is a need to read the DOP file and it will get updated to a new value written in PL5, but from there on, it is redundant data, unless the image gets exported. But, even then, if an image file contains a rating written by the camera, that rating will get both exported and written to a newly created DOP file.

The major inconsistency, caused by an unwillingness to write to a RAW file, is that the RAW file’s rating never gets changed and, should it get separated from its XMP and DOP sidecars, its rating will revert to that written by the camera.

What is more, as Rolf has found to his cost, if it is dealing with a Nikon NEF previously rated in PL5.0, PL5.1 will write the default rating of 0, found in the NEF file, to the DOP file when it applies any edits, regardless of whether there was any other rating already set.


PhotoLab needs to take account of and modify ratings set by cameras in RAW files. Non-RAW files are updated as their rating changes and thus integrity is maintained and, from a metadata point of view, files can be moved without their sidecars; the same should also apply to RAW files.

In fact, the writing of metadata to DOP files is totally superfluous to requirement. If metadata is written to any type of image file, RAW or not, all manner of software can search within those files. If people are nervous about writing metadata directly to RAW files, then that metadata gets written to XMP sidecar files.

It is apparent that PL5 uses its database for searching, so writing metadata there is valid but, as far as I can see, DOP files serve no purpose at all when it comes to metadata and would respectfully suggest that no metadata gets written to them as it only adds another potential point of definition that can cause confusion.


In summary, I would propose that DOP files are kept solely for image editing data. All metadata should only be stored either in the image file itself or in an XMP sidecar (with the exception of an indexed copy of the metadata stored in the database for searching purposes).

Special consideration also has to be made for the case where metadata, like the Rating tag, gets written to a RAW file by the camera.

Not my call. I’d like sidecars to be “backups” of the database (excluding history) so that the DB can be trashed if needed, which can be necessary - until PhotoLab gets proper database maintenance.

Note: This also means that DxO will have to sort out priorities concerning metadata import and export between sidecars (xmp and dop) and the database :innocent:

1 Like

For image editing data, I am totally agreed. But, for metadata it is totally redundant when it should already be stored in either the image file or an XMP sidecar.

Agreed…as soon as PhotoLab can handle this correctly, which is not the case now.

2 Likes

@Joanna I did not expect quite such a reaction to my “innocent” post. I was running the test to learn something about the PL4 to PL5 transition and whether Rolf (@RKyburz) might successfully recover his work by backing up his PL5 database and importing his PL4 database with all his work into the latest PL5.

Because of the issue of NEF files, thank you @jch2103 for reminding me again - yes I did not fully appreciate the danger which is the same as the one I wrote about if PL5 loses its way!

@sgospodarenko (RSVP) does PL5 actually read the ‘Rating’ data from the NEF RAW or does it expect that data only from an ‘xmp’ sidecar file only, given that it will take GPS data from RAWs. If PL5 does not look for data in the RAW then this is a “non issue”, hence my tests to see if I could get PL5 “confused”.

However, the problem for this topic appears to be something I cannot test i.e. the jump from PL5.0.1 to a later version i.e. 5.1.2.4700 . Nevertheless, it may still be possible for Rolf to recover the situation if he imports the last PL4 database into PL5.1.2.4700, having removed (or “hidden” by moving or name changing) all PL5.? DOPs .

I agree with what you wrote and well understand SPOD having developed database systems that also interacted with conventional files etc. and sometimes older databases etc.etc… I also believe that it is possible to survive with what is in place providing there are no coding slip ups. In truth some software packages I have tested did offer to put keywords back into my Olympus ORF RAW files but I am not sure about my Lumix RW2 RAW files?

I believe that the main issue is that DxO continue to monitor the discussions, bug reports, and heated discussions that take place in the Forum but rarely if ever reach out for an honest, round-table discussion with the users.

I wrote a post way back during PL5 beta testing all but pleading for a more pro-active approach to development/beta testing, in particular engaging with the various DAM users to ensure PL5 would interact correctly with their software. Had they bothered to change their ways and consult more with their informed users PL5 metadata handling would have been all it should have been.

But…

PS @RKyburz I forgot to mention that I think that it might be advisable to turn the ‘Sync’ option off in the ‘Preferences’ but I would welcome a more round-table discussion on how Rolf could proceed. I had decided to stop testing for a bit but the bug to investigate this issue on a very grey day “possessed” me and now look where I am!!

1 Like

Sound advice to turn xmp auto-sync off. I also turn off auto import/export of settings files…

Thanks to all for working on this topic — I hope that this will find a proper and comprehensive (& comprehendable!) solution soon! Thanks in particular for the advice to turn off the Sync option in preferences. I did this now, and I also turned off auto import / export of settings. Fact is that since I made these findings I have not dared to use PL on pre-PL 5.1 data, as I did not want to increase the mess. My prime goal now is to recover the old ratings. This may be non-trivial and laborious, as I have completely reorganized my photo archive—inadvertently right after installing PL 5.1, not realizing that with every peek into my photo folders I was adding to the corruption. Still, I have a plan for this, and I hope this is a workable solution. Preferably, I’d like to tackle this when handling of rating information is fixed in PL. Needless to say that I had no clue how complex (& messy) the handling of rating information (photo vs. XMP vs. dop) is. My approach to PL may have been far too naïve & simplistic: I started off as user of Apple’s Aperture, which originally suited my needs very well, until it all fell apart, first performance-wise, then as software altogether…

@platypus “Oh ye of little faith” there is little as “joyous” as seeing the star (‘Rating’) icons on the PL5 thumbnails changing immediately when you make changes from PM or FRV as if by magic!!

PM requires that you touch (click on) the thumbnail to force a refresh and FRV requires a trip to the menus etc. (except on the version I am testing which is still not as magical as PL5 but better than it was!).

When I said that DxO missed an opportunity by not consulting with the user base I meant that it was a real wasted opportunity, because I like the features available in PL5 and the way that I can interact with them or (optionally) not!

While it’s always a pleasure to explore something new, I’d still recommend to NOT use the new kitchen knife during a heart transplant…

I’ve switched the auto-sync features off (years ago) in all applications that interact in a field that I consider important, metadata in this case. An update killing its users work is a different story though, and I really hope (and expect) DxO to act on this issue. I suppose that we’ll read something about it next year, which is only a few days away…

As for reconstruction, I’d go this way. It’s a sketch only, so anyone is well advised to make a backup and think before acting.

  • Install a previous version of PhotoLab, preferably a version 4 and make sure that it points at an empty folder
  • Switch autosync off (…here it comes again), then quit PL4
  • Find folders with sidecars from PL5 (use the Finder and search for content as proposed way up)
  • Start PL4 and let it rewrite sidecars of the compromised folders, then quit PL4

…something like that…

If the number of files is not too large, you could edit the sidecar files and duplicate the Rating line and rename Rating to Rank (or the other way 'round), then at least there is a chance that PL5 might read the rating/rank correctly.

Remember what I stated early on: “The problem is not with the Rank conversion, though. I have dop files created under PL 5.0.1 with intact “Rating” entries, and I have literally watched these being reset to Rating = 0 upon opening with PL 5.1.0 (47) — hence, the problem is specific to PL 5.1.x, not 5.x.”