I’m devastated. I have tens of thousands of pictures with which I have spent literally many months of my life carefully screening, ranking, processing, selecting, etc. — and now I find that all rankings appear to be reset to 0, which renders my huge photo archive essentially unusable. If I look into the *.dop before opening the directory in Photolab, the Ratings are still there, but when I display the directory / project in Photolab, the original ratings quickly flash up, but then are instantly reset to 0 — and indeed, the *.dop files now have “Rating = 0,”. This is a disaster — HELP!!!
1 Like
I know it shouldn’t have happened but I have to ask, do you have a backup?
How did you add the ratings?
If from within PL, which version?
Are we talking about RAW files and, if so, were the ratings stored in XMP sidecars?
I have to look this up, but I may not have a complete backup of my photo archive — at least not one which is easily accessible. To make matters worse, I have very recently reorganized my photo archive, and in that process, I have peeked into most projects in order to see what they contain — and in that process, I may have erased most of my ratings — inadvertently, of course.
This is all (or most) with raw files (*.NEF), and I have added the ratings in PL, initially with earlier versions, such as PL 4, now with PL 5.x. Adding the ratings is part of my screening and evaluating projects with 500 - 1000 shots each, right in PL, either in the image browser, by clicking on stars, or with the appropriate keys via keyboard. I may be slightly lucky with more recent ones, as there, I have added “picked” and “rejected” tags, so there, I don’t quite need to restart from scratch. I have never worked with XMP files, never even considered in-camera rating…
I am guessing but this may have come about because PL4 used “Rank” in DOP files but PL5 now uses “Rating”, even though this doesn’t appear to ever get read from the DOP file but from the database.
Which do you see in the DOP files?
There is also a vague chance that, if you have a backup of the database from before this started to happen, especially the PL4 one, you may be able to re-import the PL4 database, but all this is guesswork on my part.
Roughly how many images have you added since migrating to PL5?
Are you on Mac or Windows?
I do have older projects processed with Photolab 2.2.2 and up, and these do still contain the original Rank values. The problem is not with the Rank conversion, though. I have dop files created under PL 5.0.1 with intact “Rating” entries, and I have literally watched these being reset to Rating = 0 upon opening with PL 5.1.0 (47) — hence, the problem is specific to PL 5.1.x, not 5.x.
I haven’t assessed the numbers of affected pics yet (looking into this today, may take a while). Of course, once opened with PL 5.1.0, the dop file no longer contains the original software version, but now “DxO PhotoLab 5.1.0 47”. This is on a Mac (macOS Monterey version 12.1).
If you need to find which DOP files contain either Rank or Rating, try a Finder search like this…
And you could put “Rating=0” to find only those that have been “corrupted”
Thanks for the hints, Joanna! I have actually opened a terminal window and started using egrep / grep (old Linux habits from my previous life!), but will complement this with the Finder search, in order to assess the damage… It’s a little more complicated, as “Rating = 0” may also be the result of my rating, i.e., be the legitimate value. Moreover, I don’t want to use PL to inspect existing projects, as this will inevitably just increase the corruption…
Best wishes, -Rolf
Hang on a minute! You don’t by any chance have Time Machine running on your Mac do you? On mine, it goes back to October, which is before PL5 was released.
Any chance?
Hi Joanna, that thought is obvious — but with my recent reorg of all my photo projects, I may lose that reorg when using the time machine — I’ll have to see what I can do there… Recovery is one thing, but my prime concern & worry now is that I want to avoid further corruption when working with older projects in my archive. I currently assume that I can still work with new/upcoming projects without PL 5.1.0 erasing its own, newly added rating values? In other words, I hope/assume that the problem I’m facing is only with opening existing, pre-5.1.0 project folders?
My suggestion would be to copy an “at risk” folder to somewhere else, then choose one of the copied files and “Open with…” PL5 and see what happens. That’s what I do during beta tests of PL.
Do you fancy either posting here or sending me via PM an older file and its DOP for me to test?
Just noticed, you mention Projects. Could this be only those files that are referred to in Projects?
Hi Joanna,
Projects: sorry, mix-up — I’m simply importing my photos into a folder hierarchy and then process them with PL — so far, I have not using projects. I have placed a set of pics with dop files in my dropbox, at Dropbox - DxO - Simplify your life — the files are organized as follows:
- 2018/PL 2.2.2 — 3 photos with original PL 2.2.2 dop sidecars
- 2018/post PL 5.1.0 — same 3 photos and dop sidecars, but after opening the folder with PL 5.1.0
- 2021/PL 5.0.2 — 3 photos with original PL 5.0.2 dop sidecars
- 2021/post PL 5.1.0 — same 3 photos and dop sidecars, but after opening the folder with PL 5.1.0
I hope this helps?
OK. I’ll take a look
1 Like
sorry, my bad: in my first attempt, all files in “2021” had the rating already reset to 0 (took them from a “bad” source directory). I have updated the files in “2021/PL 5.0.2” a few moments ago (16:49).
OK Rolf.
I have run a couple of tests and found that PL5 is definitely screwing up any ratings previously held in the DOP file under the “Rating” tag.
The tag remains intact until you change anything in the image, when it gets overwritten to whatever Rating value happens to be in either the NEF file or an accompanying XMP file.
It would seem that PL5 takes absolutely no notice of anything previously written to the DOP file unless it is also written to either the NEF file or an XMP file. Therefore, because your NEF file contains a default value of 0, as soon as you make an edit in PL5, the DOP file gets written and the 0 Rating value is copied from the NEF file.
This is definitely a bug, or at least unforeseen behaviour. Whatever you do, don’t risk any more data loss until you have a full backup of all valid files and this is fixed.
@sgospodarenko this seems a fairly urgent problem. Who are we best talking to about getting this fixed PDQ?
1 Like
Thanks a lot, Joanna! I’m looking forward to getting this fixed in the hopefully not too distant future! Best wishes & merry Xmas, -Rolf
@RKyburz and @Joanna I made comments in a similar but not identical Topic PL5 Tag Field not read from .dop file - #23 by Crispy and @sgospodarenko made a comment about a forthcoming FAQ to better understand the processes
and
- Well the algorithm of the prioritization of the files containing this data has been changed with the introduction on xmp support (it also depends if it’s a master image or VC). Everything will be explained in details in FAQ very soon.
Regards,
Svetlana G.
[/quote]
I had originally intended to stop testing for a bit and concentrate on other things but decided to have one last try at this (more fool me!!). I cannot yet repeat @Joanna’s findings but believe I have seen something similar (I have done so many tests I believe I have seen a photo of a UFO somewhere amongst the tests!!??) .
I have run some tests which are a little worrying (to be posted) but decided to use some test photos in a new directory structure and use PL4 to set up certain scenarios and then import that database into PL5 and continue testing (all tests up till now have been made using PL4 and then accessing the DOPs and photos using by PL5; they have not included an imported/upgraded database).
However @sgospodarenko this testing “fell at the first hurdle” when I opened the photos in PL4.3.6.32 and as the snapshot shows the photos have Ratings (Ranking) externally assigned. I have seen this before during testing but this is an entirely new directory and FRV, PM, ExifPro and Exif Pilot all show no ‘Rating’ present in the photos!?
While it is possible that I am looking at the wrong directory in PL4 could it also be that the photo entries have “bled” from elsewhere in the PL4 database (the photos have been used many, many … times?)
From here perhaps
I will abandon testing with this set of photos and pick some that have never been used in any tests before and continue trying to test what I set out to test.
UPDATE 1:-
@sgospodarenko I apologise for suggesting a database “bleed through” I re-initialized the PL4 database and the issue still persists. I also enlisted Imatch and it also shows no ‘Ratings’ set; so where is PL4 getting the data from!? Comparing the different photos (Beyond Compare) the only field that I could see that might relate to the problem was ‘Rating Percentage’ which showed 1, 25 and 50 (for photos that have a Rank of 1, 2 & 3) while the actual ‘Rating’ in all cases showed as 0!!
Opening the directory in PL5 shows no ‘Ratings’ at all (in the absence of any PL4 DOPs), which is in-line with the other programs. My tests have indicated that PL5 will import the DOP values for ‘Rating’ (‘Rank’) in preference to those for ‘Rating’ in any photo, if the DOP exists. This is inline with the fact that JPGs are considered “immutable” in PL4 and so the photo will not be changed, any Tag, Rank and keyword changes will only be stored in PL4db and the DOP.
Hence for a photo with a PL4 DOP the (PL4) DOP appears to be taken as the only source of ‘Rating’ data whereas I would contend it should be taken as the “prime” source and any “competing” photo data must be analysed rather than just dismissed.
By analysed I mean that the algorithm needs to be “clever” enough to determine if there have been changes made by other software e.g. LR, PM Imatch etc. that post dates the PL4 database (hence my planned tests) or post dates the DOP which I have already tested and it seems to be that the DOP will always be preferred BUT I have not experimented with date/timestamps except that all my Photo changes should (but may not) supersede the DOP data!!
The issue of “immutability” was always treated by DxO as almost something “sacred” but it means that interworking with other software was a potential nightmare and further complicates the transition from PL4 to PL5!
UPDATE 2:-
Getting PL5 to ‘Metadata’/‘Write to Image’ made the ‘Rating percentage’ vanish and PL4 picked up the Photos without any ‘Rank’. However, because I had not made any changes to the metadata in PL5 the ‘sync’ option did not make any “routine” writes to the photo and the manual write was required.
Just a reminder to everyone: Nikon (and perhaps others) writes a default value of ‘0’ into the Ratings tag in NEF files. (One can change this in the camera to a preferred Rating, although I don’t know how many people actually to this.) Any program reading and writing metadata needs to take this default value into account. Otherwise, ratings added in other software (including the software doing the reading!) can be overwritten by the default ‘0’ value.
It appears that PL5 is causing this exact problem. As I and others (e.g., @joanna) have said before, metadata is a mess. DxO needs to proceed very carefully to avoid creating major problems for users.
@jch2103, @Joanna, @RKyburz, @sgospodarenko
@jch2103 you and others have been concerned about the standards adopted for handling metadata in nPL5. While I am more concerned about how it seems to go about certain tasks.
The FAQ to be published should give us a clue to the way that the software engineers think but in the meantime I have been making a few tests.
So the scenarios that need to be handled are (I believe) ;
Scenario 1 - Database Only
Scenario 2 - Database & DOP
Scenario 3 - Database & DOP & Photo
Scenario 4 - Database & Photo
Scenario 5 - DOP only
Scenario 6 - DOP & Photo
Scenario 7 - Photo Only
Scenario 8 - Empty DOP & Photo
In all of the above the photo must be present but those identified as including “Photo” mean Photos with e.g. ‘Rating’ set in the photo metadata.
In only one of the above scenarios were the values in the Photo actually used, namely scenario 7 (this was written before I added Scenario 8). Unfortunately this assumes that PL4 was working in a “vacuum” or recognised any and all changes to the Photo as they occurred and noted them in the database and DOP.
Firstly I consider this model too simplistic because there is a risk that data in the Photos will be “lost” unless a forced metadata read is made (not a bad idea) but worse it seems to run contrary to the complaints that have been made about loss of ‘tags’ and ‘Rank’ (‘Ratings’) in the move from PL4 to PL5!!
According to my tests the only situation where data would be “lost” is if PL5 thought that a PL4 photo was actually to be imported, since this is devoid of any of the metadata that we are looking for, i.e. the photo is devoid of any data because of PL4 immutability so there is nothing to import into PL5.
Given that in all the scenarios I reviewed only one took anything from the photo and in all the others PL4 data was always there I fail to see what else could be going on (though that could just be me being short sighted)!!
That still doesn’t explain why it works for some users and not others. I used none of the options(Tags etc) when using PL? for my own work, so when my database went missing during the upgrade I didn’t really care but managed to salvage it anyway!
The rules do change once PL5 is working with its “own” database records and DOPs, the Photo becomes the major source of the metadata (though that depends on the Sync option and the manual reads and writes). This data will be missing if the photo is a PL4 photo and that might point to how PL5 starts “losing” data. What triggers a transition from the data being treated as old PL4 to new PL5 I am not sure, in the DOP it is clearly identifiable but in the database? My tests also indicated that the presence of the PL4 DOP is sufficient to cause the data to be taken from the DOP rather than the photo.
I haven’t tested a mix of the scenarios in one directory/sub-directory.
@sgospodarenko Unfortunately I decided to copy the tests that start with a PL4 database so that I can repeat those pertinent with a PL5 database that has already “seen” the PL4 data. Unfortunately, I accidentally killed a copy and was left with a right mess of a pair of directories. I put the data back together (or so I thought) and hung PL5 right from the start. Eventually I restarted the machine and restored the directories but now PL5 is hanging while attempting to read the directories and display the photos.
More work is needed to get things back the way that they were and avoid the following (see snapshots), a job for another day!!
Can you clarify which data you were testing? Rating? And/or other tags? And comparing input and output files for content?
Thanks.
@jch2103 Sorry during the tests I have been looking at ‘Ratings’ (PL4 'Rank) I should have made that clear. Now I have got to clean up the mess (side-line the directories and recreate from scratch) so that I can do the tests again and verify what I thought I saw was correct.