One of the issues with using concurrent programs is concurrent file locks. When that happens, issues are unavoidable.
@jch2103 Agreed but there shouldn’t be any concurrency the way I am doing the tests unless some programs hang onto a file longer than they should, having detected a potential change of interest.
My “acid” test is how many programs seem to run into trouble and if there is the odd one then why has that happened and can I replace that the program in the “mix”.
My current “mix” is PL5 under test, FRV to monitor and set RAW sidecar Ratings, PM to monitor and set RAW embedded Ratings and then I want at least one more program to act as a monitor but added IMatch as both a monitor and an “actor”. I am also using ExifPilot to check on metadata, Xplorer2 to monitor files and DOP contents and File Commander to monitor files and xmp sidecar contents and ExifPro is also there but only as a monitor.
3 monitors (screens) and one aging brain to control 8 programs, most of which need to be prodded (or a wait) to show the latest state, one reason why I love PL5 Sync it is truly a joy to work with (except for the occasional bug lurking beneath the surface!)
Hi Bryan
I think this thread is getting too convoluted because we are discussing too many variables at one time.
Might I suggest we nail down one situation at a time?
Let me start by going back to something you said in your initial post…
How and with what software exactly did you assign the ratings in the first place?
@Joanna I sort of agree, i.e. we have “deviated” into what PL5 could or should be with respect to DB versus DOP versus xmp and also into the risks of the testing that I am doing i.e. concurrency versus contention but that is not different to most posts!!
You are right because it was a long way back and I have forgotten! Looking at the snapshots I was using FRV and ExifPro (ExifPro principally for displaying multiple directories at the same time, it is also fine for setting RAW sidecar metadata but it is still “poison” to PL5 with JPGs, you can force PL5 to import the data set by ExifPro but PL5 will not/cannot then update with its own keywords!!?? ).
FRV is fine for setting JPGs but “only” sets RAW metadata in the sidecar so I added PM in later tests to be able to create the situation where an older sidecar is superseded by newer embedded data and vice versa.
Woah! There you go again, getting complicated 
To start with, this topic was only about ratings and I would like, for the benefit of my own app as well, to know exactly what you were using just for setting the initial rating of an image (we’ll return to the nightmare of keywords later)
Am I right in assuming from what you have written:
- this has always been FRV?
- the rating was written directly to JPEGs and to XMP sidecars for RAWs?
As a small aside, I saw in one of your screenshots a reference to the RatingPercent tag as well as the Rating tag. What created that? Windows?
@Joanna I have attached an annotated snapshot to help identify the “culprits” and will try to stick to this layout where possible. If it is in this post then it is a preview of one of the saved xmp sidecar files displayed by Xplorer 2(squared) preview. With both Xplorer2 and File Commander the highlighted file should be the one shown in the preview.
If it is in another topic then it will come from Beyond Compare (BC) which uses a combination of add-ons including ExifTool.
In the early tests in this topic the answer is yes to both, FRV was used to set embedded in JPGs and in the sidecar for RAW. In later tests PM was added to the mix and used interchangeably with FRV to set embedded in JPGs but in the case of PM also embedded in RAW. In no situation did I set sidecar files for JPGs intentionally.
PS in the ExifPilot display the first Rating is the one you said should be set and the second is the other that PL5 is setting. I believe that none of the software I am using touches the second Rating field so the value remains at whatever PL5 set the last time it made an update.
From what I can tell, writing the Rating to the JPEG file is universally readable, whatever the software.
Even PL4 reads such a rating, marks it in yellow instead of white and displays a tooltip to state where it comes from…
With DxO software, problems start when you change that rating in PL4. At which point, the rating only gets written to the Rank tag in the DOP file and the database. And the appearance of the rating stars and its tooltip changes…
Now, the only software that knows about the new rating is PL4. Even PL5 reads the rating directly from the JPEG file and totally ignores what was written to the DOP and database by PL4.
Now, if these ratings were all applied in PL4; and then the database/DOPs were migrated to PL5 on first run after installation, there should be no problem, but I am unable unwilling to mess up my installation that far back, so cannot test that theory.
But, should you go back to PL4 after installing PL5, that is the point at which all hell breaks loose because nobody at DxO thought that should ever happen. PL5 totally ignores any Rank set in a PL4 DOP and always reads from the JPEG file. Even if I totally remove all trace of the Rating tag from the JPEG, PL5 will then continue to ignore the DOP/database and read the rating as 0.
In theory, this shouldn’t be a problem, but only as long as you never revisit the same file in PL4. But hopefully, such practices should be rare. But you must be meticulous to never export that file from PL4 unless you don’t mind having the (now) wrong rating written to the exported file.
Of course, as soon as PL5 changes the rating, it writes it to:
- the JPEG file
- the DOP file, which deletes the
Ranktag and forces PL4 to read the updated rating from the JPEG - the database
In theory, all should now be well and I am reasonably convinced (unless you can prove me wrong) that the handling of Rating for JPEG files is correct and plays nicely with all other software 
1 Like
Agreed that appears to be the case. FRV can produce sidecar files for JPG and so can Zoner, to name but 2, but most software is not even aware that it is possible and don’t go looking for a sidecar!
Yes but on WIN10 that distinction has vanished in PL5 and there are no yellow stars, at least not in my testing.
Agreed.
I did the equivalent by backing up the database in PL5 (and also changing its name on disk, you really cannot be too careful) and then importing a PL4 database. I need to look up the results. An interesting one would be to import PL4 database into PL5 against an updated PL4 DOP (e.g. Rank=4 should become Rating=4 versus PL4 DOP rank=2).
But DxO should have thought about that because for safety there should be a period of parallel running before a user switches to a new release, that is why the trial period is useful to current users, who may decide not to make the “leap” after all!!
The last release of PL4 just before PL5 should have allowed for such an occurrence because of the step changes with respect to metadata handling. i.e. PL4 at that stage of development being “unaware” of PL5 is a wasted opportunity (just as long as you don’t get me to try to design the code!!)
I was wondering what you meant by this but of course the file export will transfer the PL4 values to the exported JPG etc
It sounds correct but I just tried to verify the statement and things did not go at all as planned in WIN10!?
The test starts in PL4 with its own DOP set Rank=3 but the photo has been changed Rating=5. Unfortunately I also performed an import of the PL4 database into PL5!
I think that PL5 does not take the DOP over the photo but rather the imported database entry over anything!?
Back to PL4 and no change because PL5 didn’t recognise a change so the DOP Rank=3.
Force a read of the photo metadata in PL5, Rating=5.
PL5 changes the DOP values during next DOP update cycle
BUT PL4 continues to believe Rank=3!!
@BHAYT @Joanna -
You should be aware that DxO has made some changes to PL4 (in 4.3.6 Build 32, Windows version). One of these changed how PL4 writes hierarchical keywords to output files. Before, PL4 would simply copy hierarchical keywords from the raw/xmp file to output files. Now, it is writing each portion of the hierarchy as separate keywords in addition to writing the original hierarchical keyword, in the same way as they’re doing in PL5. So, something that was once OK is now broken. I don’t know what other changes were made that might affect PL4 metadata handling.
I sent DxO tech support a complaint about this; the reply was that my information has been sent to the developers. “However, please note that additional changes and updates to DxO PhotoLab are now being included in the latest version of the program, DxO PhotoLab version 5. Other than some minor changes, no additional updates to PhotoLab 4 are planned.”
1 Like
@jch2103 @joanna Can you please do me a favour when you have time, either through this topic or via direct messaging I would like to understand 2 things in particular (other than why am I bothering with all this testing, other than it is a break from being up to my … in … etc etc.).
I stopped testing and checking the forums during some of the crucial metadata discussions (I had become disillusioned by the lack of feedback, I had spent way to long testing and the weather improved so the garden and outside of the house beckoned).
-
DxO put the Sync warning into PL5 because of something I presume was found empirically (by testing) rather than theoretically (a this could happen comment etc.) but either way what was the reason?
-
I am not particularly good at reading standards documents (and wading through really long topics) and actually seeing what is meant, I am way better learning by example. In as short an explanation as you feel necessary what is wrong with the way PL5 has implemented keywords in particular? The example you gave @jch2103 about the PL5 implementation breaking the rules (now followed by the latest PL4) I really don’t understand because I thought that I had read somewhere that including the elements of a hierarchy as simple flat keywords as well as the hierarchy itself was good!? Obviously not but why?
Sorry but I needed this:-
I’m replying here rather than via PM in case some other users find this useful.
Nice to see things in bloom. We have quite a few months to go here before that happens here.
I don’t know the answer to your first question.
For the second, the Metadata Working Group identified two types of keywords. The first, XMP dc:keywords, contains all of the individual keywords listed by the user. The second, hierarchical keywords, as the name implies, defines a hierarchy of keywords, e.g., animal|mammal|bear. Many users spend a lot of time developing a consistent keyword hierarchy system. As you can imagine, biologists, botanists and other scientific users among others are in this group.
One reason to have a ‘flattened’ set of keywords is to simplify searches, although creating a search on hierarchical keywords is just a programming issue. My complaint about what DxO has done by flattening hierarchical keywords and storing them in a hierarchical keyword tag is that it destroys a lot of careful work by users (i.e., by creating ‘animal|mammal|bear’ and also ‘animal’, ‘mammal’, and ‘bear’ in the hierarchical keyword for an image). Instead of
- animal|mammal|bear
the list of hierarchical keywords becomes
- animal|mammal|bear
- animal
- mammal
- bear
What was once a simple arrangement of keywords becomes a clutter in individual keywords all jumbled together. (Imagine the above example in a situation with dozens/hundreds/etc. of hierarchical keywords!) And this mess is unnecessary, because all the disaggregated keywords should be stored in the dc:subject tag!
@jch2103 Thank you I will read this properly a little later but we will have to wait a good few months for that Hebe to flower again but it was in amongst the photos I had been using for testing and couldn’t resist the temptation. The photo was actually taken 06-07-2021 (2021-07-06) but we normally get flowers earlier and it can flower somewhat late into the winter if the temperatures do not get too low.
I’ll respond to this tomorrow. I’ve just got back from teaching a beginners course in exposure.
@jch2103 I had some questions before I did some tests and after the tests I have even more! The Tests were badly executed so I will revoke my own “license” to test right now!! I switched on the test machine and conducted tests across the LAN and wound up with my old friends Virtual Copies, you can never have enough VCs!!
So animal|mammal|bear goes into the ‘dc’ and ‘hr’ fields when input by Zoner for example. Then if that photo is exported from PL5 it puts the flattened components of the hierarchy in both the ‘dc’ and ‘hr’ fields along with the hierarchical keyword.
I disagree with your statement in only one place, not all three of the components are flattened since ‘bear’ is represented by its original hierarchical entry “only” the two levels above are flattened! So after export we (appear to) have
‘dc’ = animal, mammal, animal|mammal|bear
‘hr’ =animal, mammal, animal|mammal|bear
I believe what you are saying is that flattening and including in the ‘dc’ is acceptable but should ‘bear’ also be included in that list. But that the ‘HierarchicalSubject’ field should live up to its name and only hold animal|mammal|bear?
My concern is that Zoner and Bridge and ExifPro are happy with animal|mammal|bear in the ‘dc’ field and they now appear to be showing the keys from both the ‘dc’ and ‘hr’ elements (need to verify) so do we need any flattened keys in either ‘dc’ or ‘hr’, i.e. the program that created the keywords didn’t create the flattened entries so why should PL5 do that when exporting?
The display in PL5 has me really confused because it only appears to show flattened keyword, a trap I fell into during Beta testing, when you hover over ‘bear’ it shows the hierarchy.
What is wrong with showing the whole key (other than really long keys could occupy a ridiculous amount of space)?
So please mark the hierarchy with an icon/colour change etc. to alert “idiots” like me about the nature of that keyword.
The following are outputs from my messy use of two copies of PL5 where I also added Town|Street|House as an additional keyword. The first exported photo was done before the addition of the new keyword and the second done after the addition and was new to PL5 on the M machine (hence no VC).
From PM:-
From ExifPro:-
From ExifPilot:-
The problem you have here is that by putting animal|mammal|bear in the dc:subject tag, you are adding it as one keyword that just happens to include the pipe ‘|’ symbol; you are not declaring three keywords and the one you are declaring is a composition, which can be unsearchable in most well-behaved software if you searching for an exact match on bear.
There are some apps that allow this but it does not agree with the “rules”.
If you want to record bear in its hierarchical context, you need, first of all to add the complete hierarchy to the lr:hierarchicalSubject tag, so it appears in the XMP as…
<lr:hierarchicalSubject>
<rdf:Bag>
<rdf:li>animal</rdf:li>
<rdf:li>animal|mammal</rdf:li>
<rdf:li>animal|mammal|bear</rdf:li>
</rdf:Bag>
</lr:hierarchicalSubject>
This is the “unexpurgated” version that I use in my app, which is compatible with all other software that I have tried. This describes the full hierarchical context for each of the keywords at all levels and, in my experience, allows for more flexibility.
However, some apps write just…
<lr:hierarchicalSubject>
<rdf:Bag>
<rdf:li>animal|mammal|bear</rdf:li>
</rdf:Bag>
</lr:hierarchicalSubject>
… and this also follows the “rules” in that it fully describes the keyword bear in its context but leaves both animal and mammal as strictly non-hierarchical keywords.
The next part of the “rules” states that all keywords mentioned in the lr:hierarchicalSubject tag must be mentioned in the dc:subject tag thus…
<dc:subject>
<rdf:Bag>
<rdf:li>animal</rdf:li>
<rdf:li>mammal</rdf:li>
<rdf:li>bear</rdf:li>
</rdf:Bag>
</dc:subject>
These two tags are now correctly and fully defined and are all that you need to be compatible with all but the most errant of apps.
As I have said before, somewhere, the dc:subject is meant to be the searchable tag and the lr:hierarchicalSubject tag is meant to be the tag used to transmit the hierarchical context of any of the keywords defined in the dc:subject tag.
Arranging the keywords in this manner then allows searching for a keyword, either in one hierarchical context or in several. Take the example of the keyword Orange which happens to be part of three of three hierarchies…
- Colour > Orange
- Fruit > Orange > Mandarin
- Fruit > Orange > Tangerine
- Enterprise > Telecommunications > Orange
Now imagine the scenario. where an advertising agency wants to search for images that portray the theme Orange and they want examples of use of that word in any context available.
With the XMP structure that I recommend, that is simple. Because Orange is going to be mentioned in the dc:subject tag, no matter what its hierarchical context, all images that mention that keyword will be retrieved.
If I want to be more restrictive, all I have to do is search for (e.g.) Fruit AND Orange and any and all images that match Orange as a fruit will be retrieved, no matter which variety of Orange they are.
Or I can specify (Colour OR Fruit) AND Orange to very specifically return images that refer to either the colour orange or the fruit orange.
Unfortunately, PL5’s search is presently pretty abysmal in that it is based on searching for exact hierarchies only and, in addition, can only do an AND combination of search terms.
What is much more important than apparently putting keywords in what an app deems to be appropriate tags, is whether this tags form an effective basis for…
- searching
- transmission of context
It might be interesting for you to manually add something like my test Orange hierarchies to a few files and see what search results you get from various apps.
1 Like
Hello,
The FAQ should be released soon (waitinf for some localizations). Sorry for the delay but it does not depend on me. After your reading if you still have questions/suggestions/complaints we’ll discuss it all together with PO and developers.
Btw, in PL5 metadata values are written in .dop only for virtual copies as we can’t create a separate xmp for virtual copies.
Regards,
Svetlana G.
1 Like
Is this to create virtual copies or as a result of creating them?
@sgospodarenko Thank you for your response I look forward to the FAQ. In the meantime I am concerned about you comment about the keywords in the DOP only being associated with Virtual Copies?
The following snapshots were taken from the DOP of the photo in the above example. Two things are important I feel
-
There are two sets one for the VC and one for the [M]aster image and they are typically there for the [M]aster (typically the only) photo regardless of being able to update the xmp in the sidecar or embedded (JPGs, DNG etc.). They have to be there for the VC because there is no where else to put them!
-
Only the flat keys are present, the context of a hierarchy would be lost!
@John7 it is as a result of a VC coming into existence deliberately, made by the user, or “accidentally”, made by PL5 when there are clashes between DOP Uuid and database Uuid as happened in my example above when I edited on one machine (own PL5db) and the photos were on the other machine so when I opened PL5 on that machine the DOPs did not match the database and the VCs were the result!!!
Joanna thank you for your tutorial. I need to re-read it and get back to you, in the meantime the references to Orange were a cheap shot given I was the Systems Architect for the Orange Voice Mail system (provided then by Unisys, who I worked for) during the last 16 years on my working life!
I would have thought the only time keywords are need temporally saving in a dop is if keywords are added/changed to the VC. Then when the VC is closed the final RAW should have any changes added/changed to the xmp file and those with keywords stored in the file updated with updates to the database and the dop cleared of temporary keywords and left with correction data.