PL4.3.6.32 Ghosting Ratings for Same image in different directories (Win10)

At the risk of prolonging an already too-long discussion, I wanted to point out the implications of the above hierarchical keyword structure for the user(s). You mention below (above?) using something like the standard Foundation list as an example of managing/coping long lists of keywords.

The issue your app currently presents is that it multiplies the number of keywords nodes to manage. As I’ve mentioned elsewhere I use IMatch as my DAM (still Windows only). IMatch presents several different views of files, including Media & Folders, Categories, Timeline, etc. The Categories view includes hierarchical keywords (and a special IMatch group of Categories, which is a hierarchical list of items that don’t need to be stored in the image itself). My keywords list is somewhat detailed in places (e.g., different kinds of animals/plants, etc.) but nothing like a detailed scientific name hierarchy.

But even with my somewhat limited list, I would find it annoying to have ‘animal’, ‘animal|mammal’ and ‘animal|mammal|bear’ all added to the keywords list/hierarchy as it adds more items without (in my case at least) adding any useful functionality. See screenshot.


Although it doesn’t show in the crop, IMatch provides tools to search/filter keywords to very quickly find desired images.

I’m the webmaster for our local photo club, and already have to cope with keywords in images I have to handle for club purposes. Having PL5 add more items to these ‘extraneous’ keywords would not be helpful…

But I recognize everyone’s needs are different. I’d just like options in PL5 that would minimize impacts on my workflow.

@joanna nice idea.

One 22mm olive later but on old 3/4" imperial copper pipes (the 3/4" is the inside measurement of imperial pipes while the 22mm is the outside measurement of metric pipes) so 21.49mm versus 22mm and some very careful tightening and that joint at least seems watertight so now back to re-commissioning the system!

Touch wood all is currently working. I’m going to miss climbing the steep aluminium ladder to the loft many times a day!

Actually, it doesn’t really. My keyword dictionary management window consists of two lists: one that contains every keyword, the other to organise them into any required hierarchies. The hierarchical list is purely a record of the relationships between keywords - none of the keywords are duplicated.

It is very important to make the distinction between:

  • the dictionary, with its organisational structure, which can be as simple or as complicated as a user wants. There is no obligation to even have hierarchical keywords.

  • the presentation of that hierarchical list for the purposes of choosing keywords. My app provides the (optional) possibility of choosing from a tree view or simply typing in part of a keyword and being presented with a small list of contexts that contain a given keyword, no matter where it may be in any hierarchy. See this screenshot of choosing a keyword…

    Capture d’écran 2022-01-14 à 11.20.13

  • the inclusion of hierarchical context in the XMP. There are two purposes for keyword metadata: one is to facilitate searching, the other is to transmit any hierarchical structure to other apps that do not contain the dictionary of the app that wrote the metadata.

    dc:subject is the only tag that is required for searching and must contain all keywords that are mentioned in any hierarchies.

    lr:hierarchicalSubject contains a list of the relationships between any keywords mentioned in dc:subject. If there are no hierarchical keywords, then lr:hierarchicalSubject doesn’t even need to exist. If there are keywords that are hierarchical, the full hierarchy down to any mentioned leaf keyword must be mentioned in order for its relationships to be transmitted.

And this is where confusion starts to rear its ugly head. Some apps put too many keywords in lr:hierarchicalSubject, believing that even standalone keywords constitute a hierarchy of one level. This is nonsense and totally unnecessary for the purpose of transmission of hierarchical structure.

There are two possible ways to record a keyword’s hierarchical structure:

  1. The abbreviated notation
    <lr:hierarchicalSubject>
       <rdf:Bag>
          <rdf:li>Animal|Mammal|Bear</rdf:li>
       </rdf:Bag>
    </lr:hierarchicalSubject>
  • This is perfectly acceptable since it describes the hierarchy if only a single leaf node is being recorded, but it omits the recording of the structure of any intermediate keywords.
  1. The full notation…
    <lr:hierarchicalSubject>
       <rdf:Bag>
          <rdf:li>Animal</rdf:li>
          <rdf:li>Animal|Mammal</rdf:li>
          <rdf:li>Animal|Mammal|Bear</rdf:li>
       </rdf:Bag>
    </lr:hierarchicalSubject>
  • This is what my app writes as it allows for greater compatibility with other apps. As we have seen, C1 uses the same notation.

The biggest source of apparent complexity is not how the metadata is recorded, but how it is presented.

If an app like PL5 attempts to present both structure management and keyword selection in the same small palette, things quickly get very confusing. Which is why my app makes use of a separate management popup window so that, after a dictionary has been constructed, it no longer has to be visible unless a user wants to refer to it.

Which is what my app does, but the presentation of the tree view is optional, but it includes the ability to simply press the spacebar on a selected keyword to see a browser of all images that contain that keyword. Otherwise, a user simply types in the keywords they want to search for and presses the “Search” button. Unfortunately, if you use Windows, my app is only written for Mac.

PL5 doesn’t currently add “extraneous” hierarchy structures but it does add extraneous and unnecessary non-hierarchical keywords to the hierarchy structures. But this appears to be due to how they have organised the search mechanism.

A few comments:

I certainly didn’t intend to attack your app, but rather criticize how PL5 is currently handling keywords. As I believe you agree, PL5’s current processing of raw image/xmp hierarchical keywords into a hierarchical keyword and multiple top-level keywords (i.e., animals|mammals|bear|black bear AND animals AND mammals AND bear) in output files is a problem.

And that’s what PL5 is currently doing in its output images. Doing so loses all hierarchical structure for those isolated keywords (and clutters up my keywords list…!).

I think we have the same criticism of PL5’s current behavior.

Having additional export options in PL5 to keep existing raw image/xmp keywords intact w/o change in output files would be helpful. I.e., I’m content to continue managing my metadata with my own DAM separate from what PL5 decides to implement. I just don’t want PL5 to interfere with what I’m doing.

Regarding your app vis-a-vis IMatch, aside from being Mac v PC, both seem to use generally similar approaches for metadata management, including but not limited to adherence to the MWG guidelines, avoidance of proprietary approaches, multiple views of hierarchical keywords and use of a thesaurus (your ‘dictionary’, I assume). I should mention that IMatch does have additional tools for keyword management beyond those I showed in the screenshot, but that’s extraneous to my issues with PL5.

I didn’t think you did.

The correct, full, definition of the lr:hierarchicalSubject tag for that particular hierarchy should be…

    animals
    animals|mammals
    animals|mammals|bear
    animals|mammals|bear|black bear

… where the top level of the hierarchy should be mentioned, but there is no need for mammals, bear, or black bear to be mentioned.

I don’t know how you are maintaining a keywords list but I would have expected all keywords to be mentioned once; and only “repeated” as the child of a parent in a hierarchy.

I keep two lists - one for all keywords and a second for the relationships between them.

As does my app:

  • addition, removal and editing of keywords to/from the dictionary, with accompanying correction of metadata for images that refer to them
  • preview of images containing selected keyword in the dictionary
  • drag-drop addition of keywords from the list to the combined editing/search field
  • drag-drop of keyworded files to the edit/search field for the purpose of copying their metadata to other images
  • drag-drop of delimited text from a text editor to the edit/search field, which is then tokenised
  • cut/copy/paste of keywords between images
  • copy/paste of keywords from delimited text

@jch2103 I was about to comment on your statement above (some days ago) but fixing and re-lagging all the pipes of the central heating and re-testing all the scenarios got in the way!

I was going to state that the the topic ‘Unwanted Virtual copies’ ran from December 2020 to December 2021 but it actually only “racked up” 92 posts and we have exceeded that “somewhat” in a much shorter space of time, oops! I understand your concerns about the number of posts in this topic (my “central heating” references simply add to the “clutter”).

I understand your concerns and it would be useful to have certain features made optional providing PL5 can then interwork successfully with other products, indeed those options may well be related to such interworking. DxO see Lightroom as their role model and seek to ensure compatibility or so it seems.

@jch2103 and @Joanna

I have redone the whole of the spreadsheet again and added rows for ACDSee, Zoner and ExifPro (which works fine with PL5 for RAW images). All images are now RAW because reviewing ‘xmp’ sidecar files is quicker than getting to the data in JPGs, however I then re-discovered PIE and that simplified things further!

The discussion of the merits of IMatch versus Joanna’s program is sadly academic for those of us using Windows 10 (or 11) systems and vice versa for Mac users, IMatch is Windows only and Joanna’s program is Mac only. However, describing both has merit with respect to others on the forum who are looking for such programs and, possibly/hopefully to DxO with respect to their own (further) aspirations in that direction.

However, I do have two major concerns in this whole issue;

  1. Many of the low/middleweight photo editing packages are still essentially IPTC oriented. For RAW this means that they only look at and populate the ‘dc’ Subject data with their (IPTC) Keywords. They are completely “blind” to ‘hr’; Hierarchical Subject data!
  2. This is related to item 1 and my concern about the “high jacking” of the ‘dc’ Subject field as the repository for only the flat elements of hierarchical keywords and flat keywords themselves. This is effectively “deprecating” an existing capability, namely that the ‘dc’ Subject can be populated with keywords that have an hierarchical architecture but should not be used for that purpose any more!

In the spreadsheet there are entries in a difficult to see, dirty yellow colour which IPTC oriented software such as ACDSee, Zoner, Bridge (when I was populating the IPTC keywords in previous tests) etc. will never be able to see at all.

Others on this forum have stated, in past posts, that they use ACDSee or Zoner, it is possible that they have never even considered using hierarchical keywords with those products (me for one) to describe their images since no major advantage accrued from doing things that way, but it was a feature of the developing standards that has been “conveniently” highjacked!

I apologise for my previous errors in transcribing the data to the spreadsheet and @platypus’s comments about his approach being more accurate was correct but would have led to an even more complicated spreadsheet than the one I have now got.

I highjacked Test 3 for the animals|mammals|bear|“black bear” test (I am still sure that should have been animals|mammals|bears|“black bear” or animal|mammal|bear|“black bear” but life is way too short for that type of discussion!!)

The intention of the test was to investigate how different packages populated the ‘dc’ and ‘hr’ fields in response to a small number of different scenarios and then investigate what happened to that data when passed through the PL5 processing pipeline and exported!

The variants on this theme would be to take PL5 outputs, both keyword updates (synced or explicitly written) plus exported JPGs etc and use that data as input to your favourite editor/DAM etc. The tests are easy and quick but the transcribing is …

In each row of the spreadsheet the output from the package is shown in black and the results from the exported PL5 JPG shown in blue (or dirty yellow for those where the original software will simply be oblivious to its existence), i.e. the baseline outputs are copied to a test directory and PL5 navigated to that directory and used to export JPGs. Many cells contain the same data but the order may be changed.

The attached spreadsheet has jumped from version 04 to version 06 with version 05 retaining the old layout as well as the new as sheet 2; version 06 dispenses with the old layout completely. All the normal caveats apply, I might have left an option off in a program, I might have made an error in transcription etc. Copy of meta data setting _06.zip (13.8 KB)

Please notify me of any errors and if possible add a snapshot of your settings for any given application if you think that they are applicable.

Before we go any further, I think it might be a good idea to see what Phil Harvey writes about IPTC…

IPTC section

In first place, IPTC metadata section was made for archiving purpose. It specifies many “about” (photographer, photo content, etc.) tags, which are ment to be populated by user.
At first, IPTC also allowed using ASCII/ANSI characters only, but now, Unicode/UTF-8 can be officially used as well. Of course, IPTC section has limitations:

1st limitation: Officially, tags defined in IPTC section are length limited. Some tags can only contain 3 characters (i.e. Iptc:Category), while other can contain several hundreds characters (most tags are limited to 32 characters, though).
That’s officially. However, in most cases, more than “allowed” characters can be (and many times are) written into IPTC section, and most software will show them all. But the fact remains: officially, limitation exist.

2nd limitation: Being a bit “old” standard, IPTC section doesn’t specify tags we wish to have and need today. For example, there’s no place, where you could save “rating” of your photo. The same is true for (photographed) people names, etc.

3rd limitation: IPTC metadata specification for IPTC section isn’t maintained anymore -instead, IPTC organisation decided to move IPTC metadata specification into XMP section.
This fact added some confusion among photographers… Anyway, with many software today, when entering “IPTC” metadata, data actually isn’t written into IPTC section only (or not at all) -by most “up to date” software, it is (additionally) saved into XMP section.

To sumarize: Above limitations are only valid for metadata inside IPTC section. That is, in case you use ExifTool command like:
exiftool -Iptc:City=Paris -Iptc:By-line="My name" ...
…values will be written into “old” IPTC section -because metadata section is specified. Ok, By-line tag only exist in Iptc, but you get the idea.

Conclusion: Old IPTC is dead… time to move on.

My conclusion - that being the case, PL5 has no obligation to achieve compatibility with IPTC-driven apps.

See my post quoting Phil Harvey.

Just to make sure I understand…

Delimited hierarchical keywords have no place in the dc:subject tag. Its sole purpose is to record every keyword attributed to an image but “hierarchy delimiters” are not normally allowed or even advised.

So, no, animals|mammals|bears|black bears is only allowed if it is intended to be one “keyword” and not a hierarchy. Some software does, out of good grace to legacy users, when reading such, allow this and translates it into a hierarchy. But I suspect this was intended for those who used to record hierarchy under IPTC tags.

Well behaved modern software can allow this but should then go on to “update” it by separating it out into the dc:subject tag and writing it correctly to the lr:hierarchicalSubject tag. IMO, no attempt should be made to update the IPTC tags.

I will comment on the table later, when we have had a chance to discuss this whole IPTC mess a bit more.

1 Like

Yes. But note the distinction between Legacy IPTC (‘Old IPTC’ as discussed by Phil) and ‘modern’ XMP-based IPTC. The fact that there are two flavors of IPTC causes considerable confusion when talking about ‘IPTC’. So for clarity (for DxO’s benefit) I would revise your statement to refer to Legacy IPTC/‘Old’ IPTC apps.

IPTC has been around for some time, but they adapt their standards every few years. Some interesting info about IPTC metadata use can be found here: Photo Metadata User Guide

IPTC focuses on press and telecommunications industries and pro use. DxO cannot ignore this field, if they think of DPL as a tool for professionals, even though longer standing pros already have a way to handle IPTC metadata by whatever means they use.

Non-pro users can probably get along nicely with keywords and a subset of what IPTC offers.

Nevertheless, DxO needs to get its MD handling right and I’m looking forward to the moment, when this is achieved - for Rating/Rank and all others.

1 Like

As noted in posts above, there are two ‘flavors’ of IPTC metadata, Legacy (IIM/‘Old’) and XMP-based IPTC Extension. IPTC provides more details in a section of the document you cited above: Photo Metadata User Guide
In particular:

After development over two decades IPTC Photo Metadata can be embedded in the following ways:

  • IPTC Core fields can be embedded in the IIM format and/or in the XMP format. A key challenge for metadata embedded in parallel in IIM and XMP is that the values are synchronised - this should be taken care of by the image management software. [Emphasis added]
  • IPTC Extension fields can be embedded only in XMP format.

IPTC would have done everyone a big favor by formally declaring IIM to be deprecated and superseded by the XMP-based IPTC Extension. They chose not to and dumped responsibility for keeping this (duplicate) information synchronized on makers of image management software. All of us, including DxO, would benefit from treating IIM as a read-only source of data and perhaps optionally synchronizing and writing IIM metadata only where requested by the user and needed for backwards compatibility. The XMP-based IPTC Extension metadata tags include all the information in IIM and much more, so no information would be lost by this treatment and things would be much simpler for all of us.

Rating is of course one of the standard XMP-based IPTC Extension fields.

1 Like

@jch2103 I believe that ACDsee and Zoner in particular are not using the old IPTC standard but are using the newer IPTC-XMP standard and they are both up-to-date versions and ACDSee in particular is being marketed as a general purpose data management system as well as a photo editor.

I am no expert on metadata handling, hence my post reaching out to @Joanna and you and anyone else who had something to say to help me understand this particular aspect of managing photos. I did not intend to “stir up the mud at the bottom of the pond to see what comes up…”

However, I am no novice when it comes to testing (just in tabulating the test results - it is extremely tedious - more so than reading it!) so I became concerned when 2 + 2 = 3 in the case of hierarchical keys in the ‘dc’ fields.

I first encountered this almost a year ago and it left me very confused but if you look at the table you will see that all the ‘hr’ savvy programs effectively “disappear” the “interloper” hierarchical key when one is encountered in the ‘dc’ Subject field, i.e. they all follow the pattern of replacing the “hr” keyword by its flat components in the ‘dc’ Subject field and the “hr” keyword finds its way into the ‘hr’ Hierarchical Subject fields in one form or another.

Unfortunately this leaves the ‘dc’ only software bereft of the very keyword that they created!! How can that be right was what I thought at the time and still think now! I’ve been a software engineer since 1964 and it does not add up, regardless of what the standards may say.

It is a clash of the Titans and one definitely comes off the loser, I feel - not particularly passionately but… Zoner I have used for a very long time on and off, off for some time when it went subscriber only but I got a good “family” version deal some years ago. ACDSee I have used since version 2.5 and the last licence was back in 2018 so I took an interesting offer the other day and now have an up-to-date version, it is the software I use to upload my photos from the memory card and does that job exactly the way that I want it done.

These are reasonably respected pieces of software (not as good as PL5 for “developing” photos) and they have their uses but not, it appears, when they “tangle” with the heavier weight metadata handlers like PL5 etc.

PS I “lied” about being a software engineer since 1964, that was when I started my degree course in Computing.

The following shows the original ACDSee information before adding a new keyword in PL5 and the same record after the key has been added.

PPS:- I do understand what is going on and why but the process of “normalizing” the data results in the original photo not visibly carrying the metadata that it started out with. If that program goes on to be processed by an ‘hr’ savvy program all will be well but if it goes on to be processed by another ‘dc’ only program then…!?

If PL5 implemented an option that allowed the “hr” data to be propagated to the ‘dc’ fields that would preserve the data from/for the ‘dc’ oriented programs but what would then happen when/if that data was processed by another ‘hr’ savvy program? I will run some more test, when time permits, processing the ‘dc’ program output from ‘hr’ savvy programs and then also adjust the output to see what would happen if the hybrid solution (‘dc’ = “hr” + and ‘hr’ = ‘hr’ +) is fed back into an ‘hr’ savvy program?

Hi,
only a short marker…we discussed all the things here Database Maintenance - #59 by George also the point with the Metadata User Guide.

So again a step backwards in the future :smiley:

have fun

@Guenterm I followed your link but then wound up on the database synchronisation topic and made the following “rash” post Synchronise PhotoLab - #23 by BHAYT.

Hi Bryan. I have done some research on ACDSee and found an interesting post in one of their forums…

There are two types of keywords in AcDsee

  1. AcDsee Keywords. Image->Properties->Keyword Tab. These are AcDsee keywords own keywords and you can have a Hierarchical keyword tree and a keyword list can be imported and exported. However, this information is only put into the AcDsee metadata and only AcDsee can see them (maybe a few other odd software packages) almost certainly the OS cannot see them. Additionally the Tools->Metadata->Embed AcDsee Metadata should be run to ensure the AcDsee metadata is written to the Image Metadata and not just stored in the database.
  2. IPTC Keywords. Image->Metadata._IPTC->Keywords These are where universally standard keywords are stored. Unfortunately the design standards do not allow a hierarchy (as far as I am aware) so keywords are stored in a flat structure with comma separation. The three little dots next to the keyword box allows you to edit the keywords and full keyword list used in all images. These keywords can be seen by most software including the OS
    Therefore if you export an image, and look at the image properties. The only keywords you will see will be the ones used in the Metadata PTC keywords. The AcDsee keywords are not visible. However if you used the Tools-Metadata-Embed Acdsee keyword option the keywords will be there just hidden and some software may be able to see it

As we already knew, IPTC doesn’t support hierarchical keywords and any such keywords are written to ACDSee tags. But, if we look at the XMP-acdsee tags available, we only find one called Keywords, with nothing explicitly defined for the handling of hierarchical tags.

Then you have this…

ACDSee metadata is proprietary and not viewable in exif (normally it is stored in the database however when you embed it you are writing the proprietary metadata to the files themselves). To get keywords and other acdsee metadata to be readable in exif you need to create a metadata preset to add them to iptc

So, unless you use ACDSee, you are not going to be able to see the “true” hierarchical keywords because they are only written to the ACDSee database, not the metadata. But, I guess, people have instituted the workaround of writing something like animals|mammals|bears|black bears into the IPTC ApplicationRecord Keywords tag because IPTC doesn’t support hierarchical keywords either.

However, this is, to use a technical term, a kludge of the first order and is the reason why we see inconsistencies on interchange of hierarchical keywords so defined in ACDSee.

In summary, ACDSee does not support hierarchical keywords outside of ACDSee, so you are never going to get a perfect solution for compatibility with PL5, or many other apps come to that.

As for Zoner, I couldn’t find anything on their site about hierarchical keywords - they seem to use the concept of categories instead. But, once again, they seem to be more concerned with IPTC, so true hierarchical keywords are not going to be possible without “The Kludge™”

I couldn’t even find help or a manual for ExifPro and it looks more like an open source hobby project that only handles hierarchies by using the same kludge.

@joanna I was not using ACDSee keywording I was writing straight into the ‘xmp’ ‘dc’ fields and that is where the various programs discovered that data and acted upon it, i.e. the programs see the hierarchical data in the ‘dc’ fields and use that data but those programs remove the hierarchical keyword from the ‘dc’ field completely.

Please remember that the test was about using the respective packages to set up the keywords using their own interface and then “feeding” the sidecar files that resulted from that into PL5 and documenting the outcome.

I have since started two more test cycles, one where the package adds a key to the ACDSee 'xmp; sidecar and another where the package imports the sidecar data and is then instructed to write/output or sync etc with the photo (sidecar).

The latter tests tend to track the original tests well but the former were something of a surprise and showed the existing keys being treated as a single keyword to which the data was added (you had suggested that might happen).

So we have the ‘dc’ data being treated in different ways depending on what is being done with/to it. Watch this space …

@BHAYT

This is just an FYI: IMatch comes with a number of ‘apps’, one of which is the Metadata Analyst. “This app analyzes the metadata embedded in image files, XMP sidecar files and the database itself and helps with identifying and fixing common metadata problems .” IMatch Help System

The app can be very helpful in tracking down metadata issues and problems. You could probably use it to ID inconsistent metadata handling in images from different programs.

@jch2103 thank you for that info, I managed to hang IMatch today!! It ius actually hung but it is stuck with “Updating” on the first two of the 4 files!

How do I run MDA!?

Go to the App Manager (in the lower right corner panel). It’s a tab in the same panel as Keywords, Map, Categories.

@jch2103 thank you I finally found it!!