Lost all ratings in PhotoLab Elite 5.1.0?

Just a reminder to everyone: Nikon (and perhaps others) writes a default value of ‘0’ into the Ratings tag in NEF files. (One can change this in the camera to a preferred Rating, although I don’t know how many people actually to this.) Any program reading and writing metadata needs to take this default value into account. Otherwise, ratings added in other software (including the software doing the reading!) can be overwritten by the default ‘0’ value.

It appears that PL5 is causing this exact problem. As I and others (e.g., @joanna) have said before, metadata is a mess. DxO needs to proceed very carefully to avoid creating major problems for users.

@jch2103, @Joanna, @RKyburz, @sgospodarenko

@jch2103 you and others have been concerned about the standards adopted for handling metadata in nPL5. While I am more concerned about how it seems to go about certain tasks.

The FAQ to be published should give us a clue to the way that the software engineers think but in the meantime I have been making a few tests.

So the scenarios that need to be handled are (I believe) ;

Scenario 1 - Database Only
Scenario 2 - Database & DOP
Scenario 3 - Database & DOP & Photo
Scenario 4 - Database & Photo
Scenario 5 - DOP only
Scenario 6 - DOP & Photo
Scenario 7 - Photo Only
Scenario 8 - Empty DOP & Photo

In all of the above the photo must be present but those identified as including “Photo” mean Photos with e.g. ‘Rating’ set in the photo metadata.

In only one of the above scenarios were the values in the Photo actually used, namely scenario 7 (this was written before I added Scenario 8). Unfortunately this assumes that PL4 was working in a “vacuum” or recognised any and all changes to the Photo as they occurred and noted them in the database and DOP.

Firstly I consider this model too simplistic because there is a risk that data in the Photos will be “lost” unless a forced metadata read is made (not a bad idea) but worse it seems to run contrary to the complaints that have been made about loss of ‘tags’ and ‘Rank’ (‘Ratings’) in the move from PL4 to PL5!!

According to my tests the only situation where data would be “lost” is if PL5 thought that a PL4 photo was actually to be imported, since this is devoid of any of the metadata that we are looking for, i.e. the photo is devoid of any data because of PL4 immutability so there is nothing to import into PL5.

Given that in all the scenarios I reviewed only one took anything from the photo and in all the others PL4 data was always there I fail to see what else could be going on (though that could just be me being short sighted)!!

That still doesn’t explain why it works for some users and not others. I used none of the options(Tags etc) when using PL? for my own work, so when my database went missing during the upgrade I didn’t really care but managed to salvage it anyway!

The rules do change once PL5 is working with its “own” database records and DOPs, the Photo becomes the major source of the metadata (though that depends on the Sync option and the manual reads and writes). This data will be missing if the photo is a PL4 photo and that might point to how PL5 starts “losing” data. What triggers a transition from the data being treated as old PL4 to new PL5 I am not sure, in the DOP it is clearly identifiable but in the database? My tests also indicated that the presence of the PL4 DOP is sufficient to cause the data to be taken from the DOP rather than the photo.

I haven’t tested a mix of the scenarios in one directory/sub-directory.

@sgospodarenko Unfortunately I decided to copy the tests that start with a PL4 database so that I can repeat those pertinent with a PL5 database that has already “seen” the PL4 data. Unfortunately, I accidentally killed a copy and was left with a right mess of a pair of directories. I put the data back together (or so I thought) and hung PL5 right from the start. Eventually I restarted the machine and restored the directories but now PL5 is hanging while attempting to read the directories and display the photos.

More work is needed to get things back the way that they were and avoid the following (see snapshots), a job for another day!!



Can you clarify which data you were testing? Rating? And/or other tags? And comparing input and output files for content?

Thanks.

@jch2103 Sorry during the tests I have been looking at ‘Ratings’ (PL4 'Rank) I should have made that clear. Now I have got to clean up the mess (side-line the directories and recreate from scratch) so that I can do the tests again and verify what I thought I saw was correct.

And herein lies the problem.

One of the basic principles of good reliable software design is SPOD (single point of definition)

Something that should be becoming more and more important as DxO try to move into the metadata market.

I have written elsewhere about the mess that keywords have become in PL5 but, here, I want to focus on the “simple” subject of the Rating tag.

Not considering a DAM, there are two possible places that the Rating tag can be stored…

  • in the image file
  • in an XMP file

Now, there is a common phobia about storing metadata in RAW files and yet this is something that some software does without problem.

This then means that we are forced to use XMP sidecar files for RAW images, which then means that, for PL5, we can now have the rating recorded in…

  • the RAW file by the camera
  • the XMP file by some other DAM
  • the DOP file by PL5
  • the PL5 database

Extensive testing, however, proves that the rating might get written to the DOP file but it never gets read from the DOP file and from a quick search of the database, even thought the relevant column is still marked “ZRANK” it doesn’t contain a value that equates to that written to the DOP file.

Certainly, for users who have used previous versions of PL to rate images, there is a need to read the DOP file and it will get updated to a new value written in PL5, but from there on, it is redundant data, unless the image gets exported. But, even then, if an image file contains a rating written by the camera, that rating will get both exported and written to a newly created DOP file.

The major inconsistency, caused by an unwillingness to write to a RAW file, is that the RAW file’s rating never gets changed and, should it get separated from its XMP and DOP sidecars, its rating will revert to that written by the camera.

What is more, as Rolf has found to his cost, if it is dealing with a Nikon NEF previously rated in PL5.0, PL5.1 will write the default rating of 0, found in the NEF file, to the DOP file when it applies any edits, regardless of whether there was any other rating already set.


PhotoLab needs to take account of and modify ratings set by cameras in RAW files. Non-RAW files are updated as their rating changes and thus integrity is maintained and, from a metadata point of view, files can be moved without their sidecars; the same should also apply to RAW files.

In fact, the writing of metadata to DOP files is totally superfluous to requirement. If metadata is written to any type of image file, RAW or not, all manner of software can search within those files. If people are nervous about writing metadata directly to RAW files, then that metadata gets written to XMP sidecar files.

It is apparent that PL5 uses its database for searching, so writing metadata there is valid but, as far as I can see, DOP files serve no purpose at all when it comes to metadata and would respectfully suggest that no metadata gets written to them as it only adds another potential point of definition that can cause confusion.


In summary, I would propose that DOP files are kept solely for image editing data. All metadata should only be stored either in the image file itself or in an XMP sidecar (with the exception of an indexed copy of the metadata stored in the database for searching purposes).

Special consideration also has to be made for the case where metadata, like the Rating tag, gets written to a RAW file by the camera.

Not my call. I’d like sidecars to be “backups” of the database (excluding history) so that the DB can be trashed if needed, which can be necessary - until PhotoLab gets proper database maintenance.

Note: This also means that DxO will have to sort out priorities concerning metadata import and export between sidecars (xmp and dop) and the database :innocent:

1 Like

For image editing data, I am totally agreed. But, for metadata it is totally redundant when it should already be stored in either the image file or an XMP sidecar.

Agreed…as soon as PhotoLab can handle this correctly, which is not the case now.

2 Likes

@Joanna I did not expect quite such a reaction to my “innocent” post. I was running the test to learn something about the PL4 to PL5 transition and whether Rolf (@RKyburz) might successfully recover his work by backing up his PL5 database and importing his PL4 database with all his work into the latest PL5.

Because of the issue of NEF files, thank you @jch2103 for reminding me again - yes I did not fully appreciate the danger which is the same as the one I wrote about if PL5 loses its way!

@sgospodarenko (RSVP) does PL5 actually read the ‘Rating’ data from the NEF RAW or does it expect that data only from an ‘xmp’ sidecar file only, given that it will take GPS data from RAWs. If PL5 does not look for data in the RAW then this is a “non issue”, hence my tests to see if I could get PL5 “confused”.

However, the problem for this topic appears to be something I cannot test i.e. the jump from PL5.0.1 to a later version i.e. 5.1.2.4700 . Nevertheless, it may still be possible for Rolf to recover the situation if he imports the last PL4 database into PL5.1.2.4700, having removed (or “hidden” by moving or name changing) all PL5.? DOPs .

I agree with what you wrote and well understand SPOD having developed database systems that also interacted with conventional files etc. and sometimes older databases etc.etc… I also believe that it is possible to survive with what is in place providing there are no coding slip ups. In truth some software packages I have tested did offer to put keywords back into my Olympus ORF RAW files but I am not sure about my Lumix RW2 RAW files?

I believe that the main issue is that DxO continue to monitor the discussions, bug reports, and heated discussions that take place in the Forum but rarely if ever reach out for an honest, round-table discussion with the users.

I wrote a post way back during PL5 beta testing all but pleading for a more pro-active approach to development/beta testing, in particular engaging with the various DAM users to ensure PL5 would interact correctly with their software. Had they bothered to change their ways and consult more with their informed users PL5 metadata handling would have been all it should have been.

But…

PS @RKyburz I forgot to mention that I think that it might be advisable to turn the ‘Sync’ option off in the ‘Preferences’ but I would welcome a more round-table discussion on how Rolf could proceed. I had decided to stop testing for a bit but the bug to investigate this issue on a very grey day “possessed” me and now look where I am!!

1 Like

Sound advice to turn xmp auto-sync off. I also turn off auto import/export of settings files…

Thanks to all for working on this topic — I hope that this will find a proper and comprehensive (& comprehendable!) solution soon! Thanks in particular for the advice to turn off the Sync option in preferences. I did this now, and I also turned off auto import / export of settings. Fact is that since I made these findings I have not dared to use PL on pre-PL 5.1 data, as I did not want to increase the mess. My prime goal now is to recover the old ratings. This may be non-trivial and laborious, as I have completely reorganized my photo archive—inadvertently right after installing PL 5.1, not realizing that with every peek into my photo folders I was adding to the corruption. Still, I have a plan for this, and I hope this is a workable solution. Preferably, I’d like to tackle this when handling of rating information is fixed in PL. Needless to say that I had no clue how complex (& messy) the handling of rating information (photo vs. XMP vs. dop) is. My approach to PL may have been far too naïve & simplistic: I started off as user of Apple’s Aperture, which originally suited my needs very well, until it all fell apart, first performance-wise, then as software altogether…

@platypus “Oh ye of little faith” there is little as “joyous” as seeing the star (‘Rating’) icons on the PL5 thumbnails changing immediately when you make changes from PM or FRV as if by magic!!

PM requires that you touch (click on) the thumbnail to force a refresh and FRV requires a trip to the menus etc. (except on the version I am testing which is still not as magical as PL5 but better than it was!).

When I said that DxO missed an opportunity by not consulting with the user base I meant that it was a real wasted opportunity, because I like the features available in PL5 and the way that I can interact with them or (optionally) not!

While it’s always a pleasure to explore something new, I’d still recommend to NOT use the new kitchen knife during a heart transplant…

I’ve switched the auto-sync features off (years ago) in all applications that interact in a field that I consider important, metadata in this case. An update killing its users work is a different story though, and I really hope (and expect) DxO to act on this issue. I suppose that we’ll read something about it next year, which is only a few days away…

As for reconstruction, I’d go this way. It’s a sketch only, so anyone is well advised to make a backup and think before acting.

  • Install a previous version of PhotoLab, preferably a version 4 and make sure that it points at an empty folder
  • Switch autosync off (…here it comes again), then quit PL4
  • Find folders with sidecars from PL5 (use the Finder and search for content as proposed way up)
  • Start PL4 and let it rewrite sidecars of the compromised folders, then quit PL4

…something like that…

If the number of files is not too large, you could edit the sidecar files and duplicate the Rating line and rename Rating to Rank (or the other way 'round), then at least there is a chance that PL5 might read the rating/rank correctly.

Remember what I stated early on: “The problem is not with the Rank conversion, though. I have dop files created under PL 5.0.1 with intact “Rating” entries, and I have literally watched these being reset to Rating = 0 upon opening with PL 5.1.0 (47) — hence, the problem is specific to PL 5.1.x, not 5.x.”

I can answer that. If there is no DOP or XMP file, the rating is initially read from the NEF file. As soon as any changes at all is made to the image, a DOP file is generated and that rating is copied to the DOP file.

Changing the rating in PL5 alters the rating in DOP file and generates an XMP file with the same new rating. The rating in the NEF file remains at its original value.

If I now remove the DOP file, PL5 reads the altered rating from the XMP.

If I restore the DOP file, with the altered rating, and delete the XMP file, PL5 then reverts to reading the original rating from the NEF file, completely ignoring the altered one in the DOP file.


ExifTool can both read and write Lumix RW2 files ExifTool by Phil Harvey

My personal opinion on trying to recover from Rolf’s situation may be to write everything important directly to the NEF files using something like ExifTool. That might take time but it will ensure that PL can never lose it again since it won’t write to the NEF files.

Understood. Therefore, you’d want to restore your files while eliminating the step that hurt your files.I tested my concept (sketch) with PL4 and PL5 (both current versions) and got no loss or ratings.

Yes, exactly. Best to keep DOP files as only repositories of ‘developing’ information. Adobe uses XMP sidecar files to store ‘developing’ information as well as metadata, but at least that’s mostly consistent with available standards. I understand that has the side effect of slowing down Adobe’s metadata management (because the XMP files can get quite large), but at least it doesn’t break anything.

I should note that IMatch is programmed to ignore NEF ‘0’ ratings. Extra work for the developer, but it avoids the kinds of complications that DxO is encountering. Perhaps related, IMatch relies on ExifTool in its metadata management.

I noticed after a long sesion keyword and iptc editing in Adobes bridge when i closed the app it got show a message of reducing cachesize before closing. Probably losing all data which not synced with xmp’s
I think the dxo data base should never be a single info holder and always be on sync with the external metadata. Dop files and xmp files. So you never get impressions of more data then there is out side the db. The Data Base should be used for lookup table functionality. Speedingup metadata visualisation by changing folders and images.

@Joanna the reason for my question was that there is a lot of fuss being made about NEF RAW images from Nikon cameras and I wondered why they are considered special in some way!? The process you describe I well understand but thank you for documenting it and yes the ‘Rating’ from the camera is potentially overwriting the valid data.

I checked my own two RAWS, Lumix (Panasonic) RW2 from my G9 and Olympus ORF from my EM1 MkII, with PIE (Picture Information Extractor) which showed a Rating of 0 explicitly defined for the RW2 in the embedded ‘xmp’ and nothing in the ORF!

Surely the important fact is not that there is value of 0 set for the ‘Rating’ in a NEF (or a RW2 or a …) but why is PL5 even looking at the photo in the cases we are investigating when we believe that it should have the data it needs much closer to hand which it is apparently ignoring and/or trashing.

The fact that we have all the “extraneous” data between PL5 and the photo, i.e. the database, the DOP, the ‘xmp’ sidecar and then the photo itself and it still manages to get it wrong either because certain of the elements are missing from the “puzzle” or PL5 believes that they are or manages to destroy that data on the way to updating the database etc.!

In addition Joanna you have stated before that PL5 uses or must use ExifTool. I realise that you know a lot about ExifTool because your product uses that program to handle metadata on its behalf.

Given that I am on Win 10 and you are on a Mac I have a real problem with this assertion because throughout Beta Testing ExifTool would not work on my test system for every program that wanted to use it! I later discovered that it works perfectly on the system from which I cloned the Beta machine but somewhere in completing the software build something went very wrong.

I removed every copy of ExifTool and used a program to install it but nothing worked. Process Lasso which I use to “look after” the machines reports ‘Instances exceeded’ whenever a program attempts to execute ExifTool! If it is somehow embedded in PL5 such that it runs but as another component then that might avoid the issue with ExifTool instances alternatively PL is not using ExifTool (on Win10 at least) but might use a library like Exiv2?

Incidentally if anyone has a fix for my ExifTool problem (other than a complete rebuild of the machine, checking that ExifTool is still working every step of the way) I would be grateful.

Rolf (@RKyburz) I understand the issue is “mid-way” in the chain of PL5 releases so my tests were to use PL4 as the source data and go straight to PL5.1.2. However your re-organization will complicate going back to PL4DB and then using that as input to PL5.1.2… Wherever the new re-organized filing system deviates from the old structure PL5 will simply abandon the orphaned database entries and absorb the new structures as and when (but only as and when you navigate to them).

The same would apply if you attempted to bring up PL4 on the new structures plus whenever a PL5.? DOP was encountered in a directory that had remained intact it typically would not match the Uuid of the database and Virtual copies will be the “reward”.

@jch2103 here we are looking at ‘Ratings’ specifically but as I stated above this “unique” element of NEF, which turns out not to be so unique really is not the problem! The problem is that PL5.1 “lost its way” and it could happen with any metadata embedded in the photo if/when PL5 turns its attention to the photo and decides that is the only place that it “has” to go.

That is valid the first time that PL5 encounters a photo, if the ‘xmp’ sidecar was abandoned in favour of always updating the RAW and this was consistent across all DAMs, all photo managers and all photo editors and accepted by all photographers that their “precious” RAWs were not sacrosanct then we could dispense with this additional element and there would be a reason for PL5 re-reading the photo for metadata much more often (potentially increasing the risk) but not currently.

The arguments being made are that all these “bits” make it more complicated, I would contend that they are not the root cause and they don’t really add much complexity, the root cause is coding errors and inadequate testing of the product before it is released. There may be countless reasons for needing to rush out releases but we are witnessing the consequences!!

As a parting anecdote, I used to work in Bristol every week for 2 years leaving home Sunday night or Monday morning and returning Friday evening, normally a little late to miss the traffic. One Friday I needed to apply a “fix” to the database so I wrote a program that day, tested it in the testbed and 'phoned the operators @ 19:00 and they trusted me enough to give me the go ahead so I set the program running against a live system while I started packing up to leave.

At 19:30 I checked on the programs progress (it was displaying statistics regularly) and it was doing well so I estimated the completion time and then checked a sample from the database and … I was certainly changing the database but not the way I was supposed to!!!

Program stopped, profuse apologies to the operators, I then I had to work out how to back out the errors I had created (a database roll back on a running system was a no-no) and then fix the problem (we had configured a name field a particular way and wanted to change it but early customers had been given yet another configuration and I tested on the testbed which was only configured for the previous scheme, not including the original scheme hence the tests “faked good”)! By 21:00 the program was running again and cleaning up the damage and fixing both configurations correctly and I left and got home at 24:00 somewhat chastened by the experience.

We can all make mistakes, but we need to code carefully, test thoroughly and then test again and then … hope. Oh and make sure we take all scenarios into account!

Some issues seem to exist with Nikon files treated by Nikon software. Check out the following and see if it applies…
https://exiftool.org/fix_corrupted_nef.html