Some advice regarding metadata needed

[VMB]

Hi there!

I’m new to Photolab 5 (running 5.2 on Windows 10) and often get in troble with metadata. I usually use an outdates Lightroom 6.14 to cull images and to set metadata. While doing that I convert all images to DNG files. The advantage of that (at least for me): All metadata is stored inside the DNG file instead of having an xmp sidecar file, too.

When I edit these DNG files in PL5 everything works according to plan. PL5 does see all metdata I previously entered in Lightroom. But if I alter the metadata in PL5, nothing gets written to the metadata section in the DNG file (despite having “always sync” switched on). Also, no xmp sidecar file is created by PL5. All edits are just stored in PL5’s database. In the end, all metadata edits are wasted as Lightroom can’t see any of them. Is this expected behaviour?

In contrast, If I do not convert my images to DNG, PL5 reads and writes metadata in xmp sidecars. And thereby exchanging metadata between the two apps becomes possible. That is nice, but im my case DNG files are preferred over NEF files. Is there something I am missing? Any option in PL5’s settings that allows such a DNG based workflow?

Best,
Volker

[Joanna (Joanna Carter)]

I just ran a test.

I setup a DNG file with hierarchical keywords, using my own external software and then opened it in PL5.

If this checkbox is ticked, then metadata should automatically be updated when you make changes in PL5.

If it is not ticked, in order to see the metadata, you will need to

1. read them manually from the file…

   

   or respond to the out of sync icon…

   

   

2. write them manually after having changed it in PL5

   

This will then make them visible to other apps.

[VMB]

Hi Joanna!

Thanks for taking the time to run a test. But your result leaves me puzzled.

Here is what happens on my side: With “always sync metadata” switched on in PL5 the app does read all metadata in my DNG files written by Lightroom. But any changes done in PL5 are not carried over to the DNG file (and therefore not in LR) despite having “always sync metadata” switched. Only - and only then - when I manually force PL5 to write the altered metadata to the DNG file the changes are reflected in LR.

By your screenshots I assume you are on Mac. Perhaps another difference to the PL5 app on Windows?

[Joanna (Joanna Carter)]

@VMB

Only - and only then - when I manually force PL5 to write the altered metadata to the DNG file the changes are reflected in LR.

Hmmm.

@VMB

By your screenshots I assume you are on Mac. Perhaps another difference to the PL5 app on Windows?

@StevenL could you investigate this?

[platypus (Canon & Mac (Intel and M1))]

I get the impression that DPL5 does indeed read changed metadata if the reading is launched manually. It may also sync automatically…but the issue seems to be that DPL is not sure where it should read from - and that a few IPTC sections are not written correctly by DPL5 as far as I can see looking at files that have returned to LrC from DPL5.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@VMB and @Joanna I am running on Win 10 and have been testing PL5 reading metadata put into images by other software today and what PL5.2.0 does with that metadata in the exported JPG.

So with ‘Always Sync’ OFF, AS(OFF), I exported 4 of the RAW files from PL5 I have been using files as DNG to a new directory from PL5 and checked with PIE (a useful free utility) and they were all exported with the expected keywords.

I then navigated to the new directory just created [AS(OFF)] and PL5 discovered the new images and imported the metadata; please note that PL5 automatically executes the ‘Metadata’/‘Read from image’ command (or as least an equivalent) whenever it encounters a new image, regardless of the AS (setting) (ON or OFF). If the setting is OFF then that will be the last automatic metadata operation that PL5 will perform, i.e. with the image.

I then set AS(ON) and added a new keyword to the 4th. photo and the results were that PIE saw the newly added keyword

I opened LR on my Test machine and accessed the directory across the LAN and it also found the keywords, including the new one, intact!

So I used Adobe DNG convertor on the original files and they were added to the same directory that PL5 has the RAWs . I added a new keyword in PL5 with AS(ON) and nothing new appeared according to PIE! However, the DNG and the RAW and its sidecar were all in the same directory!

PIE and FastRawViewer seem to get into trouble when there are sidecar files (in this case for the RAW) and TIFF or DNG files with the same name as the sidecar file and the RAW file etc. all in the same directory; an xmp sidecar file should not be used with a DNG or TIFF and, if present’ should be ignored.

So I moved the DNG files to a separate directory and PIE found the new keyword successfully!

So I copied the DNG file back to the original directory and checked with LR and it also found the added key successfully in spite of the sidecar belonging to the original RAW!

Sorry it could have been shorter but If I don’t write it down as I go then …?

So we are back to not being able to reproduce the problem on Windows 10 and …!?

PS:-

I then tried to reproduce the original workflow so:-

1. Copy only the original RAW files to a new directory
2. Use Adobe DNG convertor to create DNG files.
3. Import into LR
4. Set a keyword and check it is set
5. Open the new directory in PL5 and add a new keyword
6. Keyword added successfully
7. Needed to be read in LR but new PL5 keyword present.

Next test:-

1. Copy RAWS to a new directory
2. Add metadata in LR
3. Export as DNG
4. Open in PL5
5. Add keyword in PL5
6. Keyword O.K. in LR

PPS Everything appeared to be O.K. but I seem to remember LR flagging when the image metadata had changed and when I started to investigate I discovered that PIE had changed and was no longer showing the data that had been added by PL5!?

It is too late to continue tonight so will try again tomorrow.

[sgospodarenko (Svetlana G.)]

Good morning,

@VMB

By your screenshots I assume you are on Mac. Perhaps another difference to the PL5 app on Windows?

• Nope, same on Windows:
  

Regards,
Svetlana G.

[sgospodarenko (Svetlana G.)]

Good morning!

@VMB

When I edit these DNG files in PL5 everything works according to plan. PL5 does see all metdata I previously entered in Lightroom. But if I alter the metadata in PL5, nothing gets written to the metadata section in the DNG file (despite having “always sync” switched on). Also, no xmp sidecar file is created by PL5. All edits are just stored in PL5’s database. In the end, all metadata edits are wasted as Lightroom can’t see any of them. Is this expected behaviour?

• Have you tried this option in LR?
  
  

As it seems the problem is that LR does not update it automatically.

Regards,
Svetlana G.

[platypus (Canon & Mac (Intel and M1))]

@sgospodarenko

As it seems the problem is that LR does not update it automatically.

Lightroom does not automatically update, but it should show a badge for the change, unless that badge has been switched off by the user.

[sgospodarenko (Svetlana G.)]

Yes, thank you. I meant exactly that badge - in my case it does not appear but if I manually trigger ‘Read metadata…’ all metadata gets updated.

Regards,
Svetlana G.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@platypus and @sgospodarenko The badge has been driving me mad because when testing last year I was sure that there was such a badge but it has not appeared in testing today!

@platypus where do you believe the badge can be turned on and off?

What does happen is that if you attempt to write to an external image from LR a warning is given that the image data has changed and do you wish to ‘Overwrite Settings’ or ‘Cancel’. If you ‘Cancel’ at that point then the badge is definitely set!

[platypus (Canon & Mac (Intel and M1))]

@BHAYT

where do you believe the badge can be turned on and off?

Open Lightroom’s grid (Library) view
Type command-J
Check the box(es)

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@platypus Thank you.

Just using ‘J’ toggles the display options and ‘Ctrl-J’ on Win 10 opens the screen you included. I just changed a keyword on PL5.1.4 (on my test machine) and LR showed the ‘Metadata has change externally icon’,

Now back to the problem!!

@sgospodarenko, @platypus, @Joanna and @VMB we need a plan to evaluate this and other incidents being reported because it is doing no good to to the reputation of DxPL and shaking user’s confidence.

All the tests I ran late last night and early into the morning showed that PL5.2.0 on my Main machine was behaving perfectly until I ran into a test where the data seemed to be changing under its own steam!!

Either I was hallucinating (tiredness from age/to much DIY) or more likely I had switched between directories with similar titles because of tiredness and trying to re-purpose existing tests instead of copying to new Test directories plus I was using LR across my LAN (LR on Test machine and PL5.2.0 on Main machine)!

I do not want to update my Test machine to PL5.2.0 yet because I want to be able to continue to compare the changes that have been made between PL5.1.4 and PL5.2.0.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@VMB Please can you give me a step by step process to follow to try to reproduce the fault.

For example

1. Copy RAWS to a new directory (probably unnecessary except as a Test step when repeating the test, again and again and …!)

2. Add metadata in LR (please show me the nature of the keywords. e.g. DNG-01, DNG-02, L1|L2|L3 etc. + Rating? + Rotation? )

3. Export as DNG

4. Open in PL5 - AS(ON)

5. Add keyword in PL5 what other metadata changes might you make in PL5, e.g. changing ‘Rating’

6. Keyword (O.K.) ? in LR

Please change the order to reflect your workflow as closely as possible and add items e.g. do you export the RAWs to DNG immediately and then add keywords to the DNG?

The closer I can get to your exact workflow the better?

Many thanks

Bryan

[VMB]

Dear all! @sgospodarenko @platypus @BHAYT @Joanna

Thanks for all the input and your efforts to look for the root of the issue! I’ve run some tests here on my side in the meantime as well. Good news is: PL5.2 seems to do a flawless job!

I clicked, imported and tagged a couple of photos today in Lightroom. And then altered the tags in Photolab. The result is: LR doesn’t seem to recognize altered metadata reliablably. Sometimes LR shows me a flag that the metadata of an image has been changed, sometimes it doesn’t. I was not able to see a pattern here. It is just pure luck.

That means: I have to force LR every time I start the app to search for altered metadata. That is doable, but not convenient. My conclusion: PL needs to get better as a DAM in order to get rid of LR…

Again: Thanks a lot for your help. Much appreciated!

Best,
Volker

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@VMB I am glad that things are O.K…

To be honest apart from some niggles and the odd bug I actually prefer PL5’s handling of metadata to most of the other software I have tested. I have made various suggestions, some of which would raise PL5 way above the other DAM software for ease of use but the issues of “potential” data loss need to be put to rest as soon as possible.

Plus, of course, they need to implement some or preferably all of my suggestions plus the advice of @Joanna of course!! Just think metadata nirvana could be just around the corner (I can dream can’t I? @sgospodarenko )

@VMB

I clicked, imported and tagged a couple of photos today in Lightroom. And then altered the tags in Photolab. The result is: LR doesn’t seem to recognize altered metadata reliably. Sometimes LR shows me a flag that the metadata of an image has been changed, sometimes it doesn’t. I was not able to see a pattern here. It is just pure luck.

Agreed! After @platypus kindly put me straight about the ‘Ctrl-J’ option all the missing icons appeared on that occasion but when I repeated the test with LR the same thing seemed to happen, no icon until I tried a metadata write from LR, then the warning, then the icon.

For my tests I was mostly using PIE (free software - the purchase buys the option to change the metadata) to check what had changed, a very useful piece of software.

Please note that when I am testing I use at least one other program to double check what is going on (an onlooker that is not actively part of the test, to corroborate or otherwise what appears to be reported by the actual “protagonists”) but with the following caveats;

1. With TIFF and DNG any sidecar with the same name (if you have a RAW with sidecars also in the same directory - unlikely for real but likely if you are taking shortcuts during testing) then PIE will use the sidecar for the DNG!? This is actually shown in the display with an icon that shows it is using the xmp sidecar data instead of the image.

2. PIE does not automatically refresh and there seems to be a problem with the ‘F5’ not refreshing the contents of a directory so you have to move to an adjacent directory and back and all is O.K.!

These are snapshots for RAWs with sidecar files, except photo 3 which has no metadata (and no sidecar) included to show the icon that designates the origin of the data being show.

Like much of the software around a sidecar file seems to result in the software using the data from it (if available) rather than the embedded data from the image regardless of timestamps (I say seems because I have run tests that seem to show this property even when I have deliberately created a situation where there is embedded data with a timestamp later than the sidecar file).

or is this clearer

[Joanna (Joanna Carter)]

Before responding to anything else, I just had to try something so basic, we might have possibly missed it.

If we start with three non-hierarchical keywords, One, Two, and Three, what should be written to the XMP?

The correct answer is…

         …
         <dc:subject>
            <rdf:Bag>
               <rdf:li>One</rdf:li>
               <rdf:li>Three</rdf:li>
               <rdf:li>Two</rdf:li>
            </rdf:Bag>
         </dc:subject>
         …

Nothing more, nothing less.

Unfortunately, due to DxO having decided to base what they do solely on the lr:hierarchicalSubject tag, they do this…

         …
         <dc:subject>
            <rdf:Bag>
               <rdf:li>One</rdf:li>
               <rdf:li>Three</rdf:li>
               <rdf:li>Two</rdf:li>
            </rdf:Bag>
         </dc:subject>
         …
         <lr:hierarchicalSubject>
            <rdf:Bag>
               <rdf:li>One</rdf:li>
               <rdf:li>Three</rdf:li>
               <rdf:li>Two</rdf:li>
            </rdf:Bag>
         </lr:hierarchicalSubject>
         …

… essentially duplicating, unnecessarily, three keywords and, in so doing, transmitting to other software that there is hierarchical context to these non-hierarchical keywords.

It is totally unnecessary to add anything to the lr:hierarchicalSubject tag if there are no hierarchical keywords and, thus, nothing to communicate.

Unfortunately, DxO is not the only one to do this and I get the feeling that they have done it just because everybody else does it, without reasoning as to whether it is necessary or not.

I have just keyworded a file, using ExifTool, with just the dc:subject containing these three keywords and every possible search mechanism I have, in various apps, as well as the in the OS itself, was perfectly able to both read and search for them.

My reason for bringing this up is because, if I use my own software or other external DAM, which does not write the lr:hierarchicalSubject tag, I don’t want PL adding it to the metadata that gets written to an exported file. As it stands, if I export this file with only dc:subject, PL goes ahead and adds the lr:hierarchicalSubject tag to the exported file and there is nothing I can do to stop it.

It might not be obvious with such a simple example as this but, with more complex situations, this could end up corrupting metadata where the same keyword appears at different levels in different hierarchies.

[Sigi]

@Joanna

It might not be obvious with such a simple example as this but, with more complex situations, this could end up corrupting metadata where the same keyword appears at different levels in different hierarchies.

This needs to be fixed asap

[Guenterm (Win11, MAC OS, DPL 5, FP, VP, Nik4, Olympus)]

This nightmare was one of the reasons not to use DXO as a DAM, and stay for this with LR, and for quick searches I haven’t search criterias in LR to use Excire Foto even though there are sometimes funny search results using the automatic tagging here.

[JoJu (Joachim)]

@Guenterm

…to use Excire Foto even though there are sometimes funny search results using the automatic tagging here.

At least it’s entertaining. And I use it as a reminder that artificial intelligence bascially are two words promising more than they can deliver. But it’s a very good tool to find faces and together with statistics module it delivers information beyond a simple DAM. Pity is only, it can’t read the “final edits” of RAWs which makes the keywords “unsaturated” and “low contrast” the most used.