PL5 - Troubles with rating

I still have problems with PL5 indexing folders with my pictures, for some reason it doesn’t read the xmp rating while PL4 works fine.
I deleted both databases of PL4 and PL5, then opened a folder with some pictures:

PL5
pl5_rating

PL4
pl4_rating

ExifToolGUI

Those photos have been developed with PL4, I don’t see any reason for PL5 to display rating=0, I think it should work exactly as PL4.

Metadata synchronization in preference in not enabled, if enable PL5 writes the 0 rating to xmp erasing any existing value

I think there are 2 bugs here:

  1. PL5 should read rating from xmp sidecar regarless of what the dop contains
  2. PL5 should not erase an existing rating when synchronization is active.If the designed behavior is copy everything from dop to xmp, then the option should be renamed. Synchronization is something different in my opinion.

Anyway I don’t see the point in saving rating/keywords to dop sidecar, this only adds confusion

1 Like

Thanks for running those experiments Gianpaulo.

Absolutely. To avoid sync issues, there should be a single canonical source of metadata, i.e. the .xmp sidecar.

Read actions can be stored in the database but any editing of metadata should require reading from the XMP sidecar and writing to the XMP sidecar.

1 Like

For once, we couldn’t be more agreed. In the trade we call it SPOD (single point of definition)

1 Like

I am afraid they decided their “single source” is the dop sidecar :rage:

I am not so sure about that @Gianpaolo64. I think that they are moving to put all standard metadata in XMP files and all PL specific information in DOP files. Standard metadata such as ratings found in DOP files are most likely from older versions of PL. Metadata for virtual copies are in DOP files too.

This is definately what I would do. I remember suggesting in the past that everything should be in XMP files like Lightroom can do but they made a valid point that other software might damage PL data in the XMP files so they will stick with their DOP files.

3 Likes

From what I have seen there is really nothing wrong with the handling of rating in Photolab
…both Lightroom and Photo Mechanic are using color mark up too BUT that seems to be an IPTC-element Photolab completely ignores. Since Photolab seems to have a more limited IPTC-support compared to both Lightroom and PM Plus it doesn´t seem to be a good ideá to activate an autosync between Lightroom and Photolab yet. There are a lot of IPTC elements Photolab 5 doesn´t care about but still PhotoLibrary is quite usable because it do support the most important elements for most people.

I tested just now with updating a number of files in Photo Mechanic both in Title Captio/Description and Rating. Nothing gets updated by itself in Photolab and I don´t think there is any difference there compared to the integration between Lightroom and PM Plus. Even in Lightroom I think you have to make a manual import of metadata to get in sync.

The solution in Photolab 5 is this: Select File - Metadata - Read in the menues of Photolab 5. If you read the metadata you have worked with in PM Plus again it works. It´s really the same way the other way around. In that case you just Select File - Metadata - Write after you have updated for example the star rating in Photolab. If you then open PM Plus the right rating will appear in PM Plus without any other actions needed.

I have lifted this manual sync-problem several times with Camera Bits and asked them to monitor the changes made to the files by polling the file systems changes like the indexing systems in enterprise DAM-systems do. With systems like enterprise DAM it is impossible to live with a situation where metadatachanges on files made by another system isn´t updated automatically by the indexsystem in the DAM, but as long as that hasn´t happened in PM Plus and Photolab for example, we have to remember to update the data we change even in Photolab. They seem to have fixed that with Lightroom (if one activates the sync in Photolab), at least what it looks like in some Photolab 5 demos I have seen.

We also have to remember to index our metadatas topfolder in order to get all keywords imported and Photolab 5 to pupulate the Keyword-lists before we start to use the PhotoLibrary in Photolab.

So the metadata is there in the XMP-files associated with the RAW or written in the XMP-compatible image formats like JPEG, TIFF and DNG. It just needs to be read and written by Photolab and it´s for now a manual process.

Thanks for lifting the problem Alec. It really gave me motivation to test how it works.

1 Like

There is more to this than meets the eye.

My Nikon D850 is capable of rating images in-camera. It writes to the xmp:rating tag in the RAW file.

So, I take some images, open them in PL4 and can see the correct rating…

I then change the rating in all four files from 5* to 3*…

This causes DOP files to be created for each changed image, each of which contains…

			Rank = 3,

So, now I close PL4 and open PL5 on the same directory…

All of the ratings are now showing the original 5* ratings from the RAW files but the stars are in white just like they are in PL4 after changing the rating, not in yellow as in PL4 when they are read from the RAW file.

Accepting the this is purely a cosmetic change, I then change the 5* ratings to 4*…

The DOP files still contain…

			Rank = 3,

The RAW files still contain…

Rating                          : 5

But since no XMP files are written, I have to assume the only place the rating was written to is the PL5 database.

Yes, I can explicitly write the XMP files and they do contain…

         <xmp:Rating>4</xmp:Rating>

So I now have three “sources of truth”:

  1. The RAW files say the images are rated 5*
  2. The DOP files say the images are rated 3*
  3. The XMP files say the images are rated 4*

Imagine I now send all four files to someone else who is still using PL4. They get to see 4*, which is read from the XMP files.

But if I don’t export the metadata, the XMP files never get created and the person receiving the files sees 5* ratings, which is the value originally stored in the RAW files. This is because PL4 doesn’t find the Rank tag in the DOP file, because PL5 has replaced it with a Rating tag.

So the person using PL4 changes the ratings to 2*, which promptly overwrites the PL5 DOP and replaces the Rating tag with the old Rank tag with a value of 2. They can’t generate XMP files so, but, fortunately, when they send the edited files and DOPs back to me, PL5 reads the Rank tag and shows me the most recent value of 2*.

Which still leaves me with RAW files that contain 5* ratings but both versions of PL telling me something different.

And, of course, any other DAM knowing nothing about the change unless I explicitly export the metadata to XMP files, which entails going all the way to the File|Metadata|Export menu item every time I make a change, even though Metadata Synchronisation is checked. But, apparently not for star ratings?

ahhhh this is well hidden :grimacing:
yes, with the command Metadata->Read rating is imported from xmp. So the behavior is by design.
Anyway I think they designed in the wrong way. Keeping rating/keywords (and any other metadata) in dop sidecar in my opinion is wrong. dop should only contain development settings.
I am not using PL library, why should I remember every time to select all files and do a metadata->read command ??
PL5 should work as PL4 and automatically import any changes in xmp sidecar.

2 Likes

I don’t believe you could productively do so: move files from PhotoLab 5 to PhotoLab 4. You’d probably have to leave the .dop files and visual corrections behind, though of course the .xmp files could be moved with ratings.

If it’s possible to do corrections in PhotoLab 5 and move the .dop files to PhotoLab 4 and have them open up properly that would be a great feature. Many photographers are in the position of only being able to run PhotoLab 5 on their desktop or laptop. Cross-compatibility between the two versions (except for Fuji X-Trans) would make photographers lives much easier.

Absolutely. Isolating the metadata in the .xmp is the first step. And as Stenis pointed out the polling should be done automatically:

I have lifted this manual sync-problem several times with Camera Bits and asked them to monitor the changes made to the files by polling the file systems changes like the indexing systems in enterprise DAM-systems do.

Polling is less of an issue in PhotoMechanic (not the Plus catalogue of course but the original metadata tool) as I believe the tool reads all of a folder’s .xmp files every time you open that folder you shouldn’t run into discrepancies.

The only time you’d get in trouble is if actively changing metadata in another application while PhotoMechanic has that catalogue open.

Polling could help here but editing metadata simultaneously in two applications seems a bridge too far for me (as far as reliability is concerned).

With systems like enterprise DAM it is impossible to live with a situation where metadatachanges on files made by another system isn´t updated automatically by the indexsystem in the DAM, but as long as that hasn´t happened in PM Plus and Photolab for example, we have to remember to update the data we change even in Photolab.

The current system is a lot of trouble for what should be simple. PhotoLab should be checking .xmp modified dates on opening a folder. The .xmp files don’t have be read to check modification dates so there should be no slow down.

I have noticed that sometimes PhotoLab 4 sees the changes to .xmp ratings made in an external application while the folder is open, so DxO even has the technology to monitor changes live and update metadata on the fly without requesting a manual sync. @Stenis do you know how PhotoLab 4 does this party trick and what triggers it?

This would be fine but, as in the case of the Rating being written into the Raw file by the camera, this can still lead to confusion if the XMP differs.

I know there seems to be an absolute phobia about touching the RAW file when editing metadata but, surely, in the case of camera generated metadata, this now needs to be reconsidered to avoid this multiple point of definition problem.

I use ExifTool to change metadata in RAW files (via my app) all the time and, in several years of so doing, I have never had a single file corrupt, because ExifTool renames the original file before making any changes, then verifies the changes on a copy of the original before writing to the original file and then, finally, deleting the renamed original. If anything goes wrong, all that is needed is to delete the corrupt copy and re-rename the renamed original.

Is there nothing simple which you can’t make more complicated than it needs to be Joanna? We’re dealing with professional applications which manage metadata here. The convention is to store the metadata in the .xmp. It’s been difficult enough to get DxO to respect the rule to store metadata in .xmp files and now you want to bring in storing metadata in RAW files?

No thanks. DxO has mostly done the right thing which is to respect the convention of storing metadata in .xmp files. There’s some legacy issues with rating in the .dop which should be tracked down and phased out.

If a rating is found in the RAW file, that rating should be migrated over to the .xmp file and subsequently (after a rating is applied elsewhere) the rating in the RAW file should be ignored or changed to follow the .xmp rating (which is the master).


PS. I suggest you stop writing metadata in non-standard places. Reinventing the wheel here would be a very destructive choice. Cross-application compatibility with metadata is very fragile. What’s in the standards is less important than respecting common practice and real world use.

I don´t know why Nikon users want to use their tiny little displays to cull images and rate them in their cameras instead of using software made explicit to carry out tasks like that, Dedicated metadata editors in computers is a much more efficient way than camera interfaces to do that. Why not just do like most of the others that write that data to XMP-sidecars like Alec has suggested in several places now? Wouldn´t it be better if Nikon stopped using their unorthodox methods and wrote to the XMP-files like everybody else?

In Photo Mechanic you can allow RAW to be modified and embed IPTC in the RAW-files too I can see. What ever that might mean in detail. So I don´t know if there is any phobia about that but it is a problem that there are a lot of different ways to handle and exchange metadata between different systems on the market. Still the metadatabusiness seems to be a very immature business.

Even Camera Bits seems to have some home work to do with Photo Mechanic Plus. They claim they support XMP and so do DxO and Adobe with Lightroom, but there is no open interface to other namespaces than IPTC for us in any of these softwares when designing the forms we are able to use. Namespaces like Dublin Core and EXIF in XMP are still hidden. Shouldn´t for example corresponding fields for Title, Headline, Caption/Desceiption and maybee Comment automatically be kept in sync in the EXIF-elements in the files and the corresponding namespaces in the XMP-files (like Alec stresses) or XMP headers in XMP-compatible files like JPEG, TIFF, DNG for compatibilty reasons. They could easily do that by forking from one element to several others and if they don´t we can do it ourselves if necessary with an open XMP-element interface.

Even some PDF-filetypes get XMP written to them i Enterprise DAM-systems because they are XMP-compatible too and have built in headers like the compatible image types. XMP-files are even used in these systems to attach metadata to really any type of files including common Office-files and really all sorts of files companies and organisations like to keep track of. Of that reason I have suggested Camera Bits open their systems for that too. I think that would be a great selling point for both PM Plus and Photolab as a joint offer to the market. Many photographers are entrepreneurs that need to keep track of a lot of other files too. In fact this development has been exactly what has happened to Enterprise DAM-solution like the Norwegian FotoWare-system.

From the beginning that software like many others were developed by photo centric people at Hasselblad in Sweden and it was strictly used for handling images at the newspapers but they were wide open for change when the newspaper business declined heavily 10 years from now. Thats why they were extremely on their toes to help the City Museum of Stockholm to develop a DAM-implementation that supported all types of documents the museum wanted to throw at it instead of just images.

In the example below you can see the result of a search for “Kungsgatan” or “King Street” in "The Digital City Museum of Stockholm. Under every tab on the top there are really a separate index and there are a different data types indexed like historical images, documents and publications, art and artifact e.t.c. I don´t expect a fancy interface for all types of material but it would be nice just to be able to tag documents too and be able to open them in corresponding software in Windows or Mac. It can be a lot better than today with really very little extra work.

Historiska fotografier - Digitala Stadsmuseet (stockholm.se)

Since DxO have worked for quite a while to integrate with Lightroom they now have a hot link working with Lightroom (if you turn on the sync in “Preferences”). The problem is that not all of us would like to have it like that. Personally I don´t like two way sync solutions and I think it would be good to be able to configure in “Preferences” if Photolab shall read or write data from or too the XMP-files associated with RAW or the ones that have the data embeddad in the image files.

When I decide to let Photo Mechanic own the metadatamasterdata it ment in my world that Photolab only shall read the XMP-data wether it is stored in XMP-files or embedded in XMP-headers in JPEG, TIFF or DNG. I haven´t really figured out how my PM Plus metadata initially got read by Photolab because it is not updated when I have changed it in PM Plus and open the pictures again in Photolab but it seems to always get read when I export a JPEG from the corresponding RAW.

What I think they have to do is that they always read the XMP-data outside Photolab automatically when we open a image-folder and have told Photolab always to read XMP from outside automatically. It gave a short lag of a couple of seconds when I did that manually by selecting all 220 images in a folder I had opened and I activated the “File - Metadata - Read” manually.

If we stick with the principle of “single direction flow” we also need to be able to select the opposit configuration in Photolab “Preferences” where we instead tell Photolab always to write to the external XMP. This is the case when you instead use Photolab and PhotoLibrary as your metadata owner. Already today PM Plus directly amd instantly see the changes Photolab makes to the XMP-data without any further manual interaction needed.

In both cases Photolab has to update it’s database. Despite I strictly use PM Plus as a metadata owner I see a value of a PhotoLibrary database in sync. I think it´s valuable to be able to see what metadata will be exported from my RAW to my JPEG derivates. If I configure Photolab to use another XMP-data owner than Photolab all the IPTC-elements in Photolab ought to be inactivated to block metadata editing. If I chose to have Photolab as owner they should be open. All just to prevent two way flows and avoiding duplicate “truth”.

This is of course just a suggestion but to make this work better DXO can NOT leave responsibility to the users to manually keep the systems in sync. That will definitely lead unnesessary to unsync.

Stenis, Nikon does not write .xmp files with their images. That’s okay as Nikon controls the whole format.

What matters is that whatever triage tool a photographer uses reads that embedded rating and then moves it to .xmp where it belongs. PhotoMechanic in fast does have settings addressing this specific issue, whether to write into the file or the .xmp sidecar.

The most important is “Always create and/or update XMP sidecar file”. CameraBits recommend that almost everyone choose this option. When this is selected, non-standard images with embedded metadata will have their metadata brought into line with standard practice.

PhotoMechanic writes back out new data to all the locations as far as I know just to be sure.

The treatment of metadata and metadata consistency are a work-in-progress. Metadata between applications mostly works but it’s very fragile. Which should make every software developer who touches metadata very cautious about deviating from the limited common ground there now.

In terms of sync, avoiding sync is the best sync solution. Hence all applications reading from and writing to the common .xmp file is by far the best way to keep data consistent.

Agree! That´s even my configuration in Photo Mechanic. :slight_smile:

1 Like

When you have spent two years trying to fathom the depths of metadata, it is impossible to think of it as “simple”. So many different standards, some of which are now incorporated in other standards, some of which have been dropped. No, if there’s one thing you cannot say about metadata, it is not “simple”.

Maybe for you, but there are many photographers out there that do not require expensive, enterprise level, metadata managers.

Or, more correctly, the convention that you wish to adhere to.

macOS provides a free, efficient DAM. It provides Finder Tags, which can be added to files, no matter what type or even if they are folders. The Spotlight mechanism efficiently and unnoticeably indexes them in the background and can search for files that match hundreds of different attributes in an instant. The tag metadata even transfers with the file from one computer to another.

Unfortunately, the world doesn’t run entirely on Macs. It has only been in the last couple of years that Windows even started to get an integrated metadata indexing and search functionality. And, of course, it had to be implemented differently.

I don’t know much about the Windows metadata capabilities but the Mac system modifies the metadata in the attributes part of a file and has no need of an external sidecar.

The problem with XMP sidecars is that it requires a management app to maintain the link between the sidecar and the original file, otherwise, you have to search for stored metadata held in sidecars and then further search for the related file. This is highly unwieldy and inefficient, and is the reason why most DAMs use a database to maintain some sense of sanity and speed when searching.

Unfortunately, this now makes for multiple sources of truth, where it is possible for the metadata in XMP files to become out of sync with that held in the database of whichever DAM was used to originally index it.

All it takes is for someone to change the metadata in the XMP file outside of the DAM that created it and you now end up with four possible “sources of truth”: the originating XMP, the originating database, the modifying database and the modified XMP. But which “truth” is “the truth”?

If the whole world were to read and write metadata directly from/to files, this reconciliation nightmare would disappear overnight.

If I were conspiracy theorist, I would postulate that Adobe deliberately invented sidecar files for the sole purpose of ensuring it was impossible to manage metadata without using their DAM, and the whole world fell down and worshipped the words spoken by the great god Adobe and, dutifully, fell in line - thus creating the mess we have now, where even Adobe themselves have changed the standards over the years, including now having two different ways of laying out the contents of XMP files.

Adobe themselves allow for options as to how hierarchical keywords can be written to metadata tags, for which, the default is different between Bridge and Lr.

No. metadata is not “simple”. Which is why my app (which you dare to call “hobbyist” without even having used it) doesn’t use a database, because it relies on the single source of truth of what gets written to files and users have a choice of writing to RAW files, if they wish, or XMP if that suits them better. Either way, what the app sees and edits is always the original metadata - never a cached version from a database. It relies on the superb metadata engine that Apple has perfected over many years for storage, editing and searching. As long as other apps comply with the XMP metadata domain, my app can read their metadata and any app can read and correctly interpret, and search for, the metadata my app writes.

I am not writing them to non-standard places. My app complies 100% with XMP, MWG and Dublin Core standards. Of course, they may not be standard to your definition, but every other app out there seems to cope perfectly, even if the original metadata was written directly to RAW files.

1 Like

Dear @Joanna,

I’m only a reader of all the stuff, but one question I would like to ask.
From time to time I can read some statements her in the forum or on other wesites, that isn’t a good idea(not to say forbidden) to write anything into the raw file. Could you explain this statement to us, or give us a link to a good article.

thanks a lot and enjoy the week

Guenter

Some Nikon cameras allow you to rate your shots in camera. It is not something I do but, as the author of a DAM, it is something I have to be aware of. As a photographer though, I tend to cull on the camera, those shots which I know are rubbish, if I have the time. But then, I don’t take thousands of shots, preferring to take a more considered approach and getting it right in camera before even pressing the shutter. I then use my DAM to further assess on a larger screen and PhotoLab to experiment with those shots that might warrant “rescuing” or reworking.

I don’t know of a camera out here that writes anything to XMP files.

So, despite some people asserting that XMP files are the only place to keep metadata, even Photo Mechanic allows direct writing - interesting.

I won’t quote it but your next comments only serve to confirm my assertion that metadata is hard and, whatever DxO have managed with PL5 is nothing short of miraculous even if it is not yet as complete as other DAMs.

That would be nice and it is something that my DAM app allows for. You are right, it is very little work to implement. Maybe you should add a post in the Which feature do you need? section of these forums.

I would agree that the XMP file should always take precedence of anything else that PL5 might store internally.

Absolutely.

Hi Guenter

Your message prompted me to find something that might explain why some people seem so reluctant to write metadata to RAW files ,and why this is a false premise.

A certain Bogdan Hrastnik has written a very interesting article on the ExifTool website, which is horrendously technical but effectively states that there is no reason, apart from unreasoned fear, why you should not write directly to RAW files. As I have stated on numerous occasions, it is something I have always done for years and have never had a single file corrupt.

1 Like