PL5 Tag Field not read from .dop file

[jch2103]

@Musashi

As for now here are some explanations

I hope the ‘full’ FAQ goes into more detail about ‘All other Metadata’. Currently, there are some serious bugs/issues with PL5 metadata handing that cause serious problems with users of 3rd party metadata products (as noted by comments to this thread). Unfortunately, turning off metadata synchronization in PL5 doesn’t solve all of these issues because of what PL5 writes to output files.

PL4.x simply copied metadata from the raw/XMP file to output files. It did a good job with this, and was consistent with 3rd party metadata products. There needs to be an option in PL5 to restore this behavior.

[Johnny]

What I do not understand: when there is no entry in the database file and no xmp file for a RAW file, BUT there is a dop file containing a star rating. Why will it not be read?

I do not want to use a database file, especially if I cannot edit it and there is no error message when files cannot be found anymore because the are moved to a different folder (what I did).

Does anyone know a possibility to edit the database file content?

[KeithRJ (Keith - Perth, Australia - Win11, PL6, VP3, Canon, DJI)]

You can use any tool that can edit SQLite databases but you can destroy the database so be careful and operate on a copy to test your edits. If you are familiar with relational databases then you should be OK because it it not too complicated.

I use FlySpeed SQL Query (there is a free version).

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

I have played with the database (but while PL5 was running) to change a timestamp of a file and crashed PL5 totally and irreparably!! With PL5 shut down changing some things might be possible!? On Win10 I use DB Browser for SQLite version 3.12.2 which is free but beware any changing of the database is tedious and potentially “dangerous” but rating changes should be easy enough etc.

I am testing my Synchronisation strategy right now so I changed the DOP Ratings field from 5* to 0* and requested a ‘Sidecar’ ‘Import’ but PL5 changed the DOP value back to 5*!

I am not sure this has previously been the case with PL5 releases because many of my tests have been by “hacking” the DOP so it seems that they may be closing down certain “windows of opportunity” for “testing”.

If you are dealing with RAWs then you have the ‘xmp’ sidecar but I believe that Photo Supreme can also change the embedded ‘xmp’ for RAW files. You effectively need to be able to take the Rating from the DOP and create an ‘xmp’ sidecar with just that data which PL5 will import in preference to other sources iff timestamps are right (still unclear about timestamps in this context).

It might be possible to reduce this further but the minimum I seemed to be able to get away with for an xmp with “just” ‘Rating’ is

P1074396.xmp (740 Bytes)

[Joanna (Joanna Carter)]

@BHAYT

You effectively need to be able to take the Rating from the DOP and create an ‘xmp’ sidecar with just that data which PL5 will import in preference to other sources iff timestamps are right (still unclear about timestamps in this context).

From my tests, it is apparent that PL5 never reads the Rank from a PL4 DOP file, even if there is no Rating in either the RAW or XMP files. It would seem that there is no forwards transfer of PL4 Rank to PL5 Rating. Even importing metadata from an XMP file doesn’t change the Rank until you make another change to the image, at which point, PL5 changes the DOP to PL5, deletes Rank and writes whatever rating it finds in the XMP file, which is usually 0. Thus losing any Rank/Rating set in PL4.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

Joanna (@joanna) you know how I hate to contradict you but PL5 ingesting a PL4 DOP does work. I just used the data I used in the post above and in PL4 I set the ‘Rank’ to 3. PL5 happily read the DOP and created a new Virtual Copy (it had already “seen” the photo in my previous test and so a VC was inevitable) and the VC has a ‘Rating’ of 3.

I panicked that the data had been passed by an ‘xmp’ sidecar so I made a change to another photo in PL4 and then checked the directories which showed ‘xmp’ sidecar for first (the subject for the @John7 reply post above) but none for the other two photos. So if PL5 ignores PL4 DOPs how have I got the snapshots that I have? (remember I am on Windows 10 and PL5 has already seen this directory and the database was not clean before the test and this was an adhoc test rather than a meticulously executed, clean-room test and …

PL5 does read PL4 DOPs and convert ‘Rank’ to ‘Ratings’ according to my test (if I haven’t made a mistake!?) and PL5 has left the PL4.3.6 alone, at least up to the point of the snapshot!

PS:- In spite of creating VCs which normally create a larger DOP, PL5 made no changes to the DOP on closedown, because I had made no changes in PL5. This actually maintains some co-existence providing you do not make any changes in PL5 that would result in a material change to the status of the data. It means that if I clear out the PL5 database I can re-discover the photo and there will be no VC!

PPS:- This Topic was originally about Tags so I set tags on all four photos in PL4 red, green, green, red and those all made it into the PL5 VC[1] images.

[Joanna (Joanna Carter)]

OK. I’m editing this in an text editor and then pasting it here.

Test of Rating from PL4 to PL5

Delete databases for both PL4 and PL5
Delete DOP and XMP files for test image
Check Rating was not set in RAW file by camera
Open image in PL4
Set ‘“Rank” to 3

Close PL4

DOP shows…

Sidecar = {
	Date = "2022-01-19T13:11:28Z",
	Software = "DxO PhotoLab 4.3.6 72",
	Source = {
		CafID = "C30510d",
		Items = {
			{
			Albums = "",
			CreationDate = "2022-01-19T13:11:00Z",
			ModificationDate = "2022-01-19T13:11:14Z",
			Name = "_JNA0004.NEF",
			OutputItems = {
			},
			ProcessingStatus = 1,
			Rank = 3,
			Rotation = 0,

---

Open PL5

PL5 does not display a Rating for the file…

Make an image edit without touching the Rating stars

Close PL5

DOP file…

Sidecar = {
	Date = "2022-01-19T13:11:28Z",
	Software = "DxO PhotoLab 4.3.6 72",
	Source = {
		CafID = "C30510d",
		Items = {
			{
			Albums = "",
			CreationDate = "2022-01-19T13:11:00Z",
			ModificationDate = "2022-01-19T13:11:14Z",
			Name = "_JNA0004.NEF",
			OutputItems = {
			},
			ProcessingStatus = 1,
			Rank = 3,
			Rotation = 0,

Nothing has changed

---

Reopen PL5

Do nothing

Close PL5

DOP file…

Sidecar = {
	Date = "2022-01-19T13:17:04.9440000Z",
	Software = "DxO PhotoLab 5.1.1 52",
	Source = {
		CafID = "C30510d",
		Items = {
			{
			Albums = "",
			CreationDate = "2022-01-19T13:15:02.0840000Z",
			IPTC = {
				contactCreator = "Joanna Carter                       ",
				statusCopyrightNotice = "Grandes Images 2017",
			},
			Keywords = {
				{
					"Abeille Bourbon",
				},
			},
			ModificationDate = "2022-01-19T13:15:29.0140000Z",
			Name = "_JNA0004.NEF",
			OutputItems = {
			},
			ProcessingStatus = 1,
			Rating = 0,
			Rotation = 3,

Rank of 3 has been deleted and Rating of 0 has been added

Image edit has also been added to Overrides section…

				Overrides = {
					ExposureActive = true,
					ExposureAutoMode = "Manual",
					ExposureBias = 1.7370986238532113,
				},

No XMP file created.

---

Reopen PL5

Set Rating to 4…

Close PL5

DOP file…

Sidecar = {
	Date = "2022-01-19T13:21:51.4780000Z",
	Software = "DxO PhotoLab 5.1.1 52",
	Source = {
		CafID = "C30510d",
		Items = {
			{
			Albums = "",
			CreationDate = "2022-01-19T13:21:24.4940000Z",
			IPTC = {
				contactCreator = "Joanna Carter                       ",
				statusCopyrightNotice = "Grandes Images 2017",
			},
			Keywords = {
				{
					"Abeille Bourbon",
				},
			},
			ModificationDate = "2022-01-19T13:21:46.1930000Z",
			Name = "_JNA0004.NEF",
			OutputItems = {
			},
			ProcessingStatus = 1,
			Rating = 4,
			Rotation = 3,

Rating now set to 4

XMP file created…

         <xmp:Rating>4</xmp:Rating>

---

Delete XMP file

Reopen PL5

Rating is now 0, proving that the DOP file does not get read in the absence of an XMP file

---

I noticed, at least, two things:

1. PL5 doesn’t write either the rating change or the image edit until the second time it was closed. If I select another thumbnail before closing the first time, the changes are written.
2. If you delete the XMP file, the Rating in the DOP file gets ignored.

Well, Bryan, what do you think of that? And can you reproduce it?

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@joanna

@Joanna

Well, Bryan, what do you think of that? And can you reproduce it?

What do I think of that well

1. You are trying to outdo me with the longest post!

2. I cannot reproduce it because it does not happen on Win 10, or rather on Win 10 on my Main machine. PL4 and PL5 play happily together with the DOP making its way to PL5 and PL5 leaving it alone so I can pick it up happily in PL4 until I make an edit in PL5.

I will rerun the tests on my test machine after clearing the databases etc. but the base photos I used have not been used in any previous PL5 tests of mine (deliberately avoided, hence the use of the term base and baseline, against which later incarnations can be compared). I will repeat the tests later today on my machine where I will clear the databases and follow your tests to the letter but using my way more attractive pictures from the garden (taken last July)!

It is possible that PL4 and PL5 prefer such pictures but I somehow doubt that!!!

Edit 01:

1. Set up new directory with photos that have no rating.
2. Open in PL4 and add ‘Ratings’ (Rank) and Tags.
3. Close PL4 and open PL5
4. Navigate to directory in PL5 and BINGO it works!

[Joanna (Joanna Carter)]

@BHAYT

• Open in PL4 and add ‘Ratings’ (Rank) and Tags.

I haven’t tested with adding tags as well. Since this counts as an extra edit, over and above just setting the rating, it might? make a difference. My tests are only about the Rating and, if you follow my “procedure”, you mustn’t even click on another thumbnail, as this can stimulate writing the Rating to the DOP.

I have been extremely OCD over my testing, trying to isolate one thing at a time

---

What happens for you if you delete the XMP file after setting the rating in PL5? Does closing and reopening PL5 still read the Rating from the DOP? This is starting to look like a very subtle platform difference.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@joanna I understand your caution over clean-room/clean slate testing however I did the rating test on its own and it also worked fine! There is no evidence of any difference between the two situations because in both PL5 leaves the DOP intact, i.e. it either finds the PL4 DOP and acts upon it or it doesn’t.

Just because it was you I repeated the test, new directory, all clean, this time 3 JPGs and 1 RAW, and when I opened the directory in PL5 it found and displayed the images with rating correctly and then hung @sgospodarenko so the hangs are related to PL4 DOP processing with JPG images perhaps.

So repeat again with RAWs.

1. Terminate PL5 with prejudice!
2. PL4 process RAWs
3. Close PL4
4. Open PL5 and perfection all ratings present and correct

Either the software is more afraid of me than it is of you or it works (unless I use JPGs from my collection) on Win 10 on my Main machine!

There are no ‘xmp’ files involved in this test. The first test was with a directory that I had already created for another test and I used photos 2 and 4 (1 was used for the previous test).

In my test above I went from 4 down to 2 stars and in the latest test I went from 1 to 4. Given my concern about ghosting, a new but rather different issue has occurred on PL5 with my execution of my synchronisation procedure which means that I effectively change the name of a directory to the name of one that was itself renamed (conceptually deleted) and PL5 showed the images for the previous incarnation of the directory!! This was then resolved on a restart so the software is confusing itself and using from a cache perhaps @sgospodarenko!!?

Please note that all work is done in the PhotoLibrary and the Customise section is not used at all in conducting these tests!

I agree with you about a potential platform difference but I am not sure how many others have complained about PL5 versus PL4 during the upgrade process.

This topic was not actually started because of a PL4 to PL5 issue but because of a PL 5.0.? to PL5.1.? issue.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Johnny I must apologise for my original post in response to your query, i.e. I attempted to address the question

@Johnny

Does anyone know a possibility to edit the database file content?

but I should have probed the original issue

@Johnny

What I do not understand: when there is no entry in the database file and no xmp file for a RAW file, BUT there is a dop file containing a star rating. Why will it not be read?

I should have asked for details of

1. The operating system of the machine you are using, i.e. Win or Mac.
2. The generation of the DOPs that are “not being used”, e.g. PL4, PL5.0.?, PL5.1.? or even earlier or even ALL.
3. Is this a problem that you have actually experienced or is it solely in response to the table that has been published in an earlier post or both? and again this response should also align with the issues item 2.

Throwing out potential solutions is wrong when I actually need more information on the exact nature of the problem.

Arguably if there is an entry in the database then the data in the DOP “should” be redundant. I am not sure if forcing a ‘Sidecar’ ‘Import’ will actually force PL5 to read the DOP and update the ‘Ratings’ or ‘Tag’ fields or simply ignore the request or only action the request if a timestamp related to the DOP file date/timestamps or one of the timestamps held within the DOP actually shows that the DOP data is of a later generation to the database.

For any of this to happen the DOP Uuid must match the database Uuid for the image or VCs will result and these will certainly read the DOP!

EDIT 01:-

One of my favourite “pastimes” has been “hacking” DOPs with “hacking” ‘xmp’ sidecar files running a close second! So I just “hacked” a DOP to test if PL5 will pick up a ‘Rating’ change; in fact the DOP that was “hacked” was a 4.3.6 DOP and I change the ‘Rank’ from 1 to 3. PL5 was running and there was no automatic reaction but a ‘Sidecar’/‘Import’ request sorted that out and the ‘Rating’ changed from 1* to 3*.

I then forced a DOP write ‘Sidecar’/‘Export’ and a 5.1.2 DOP was created so I “hacked” the ‘Rating’ field to 5* and PL5 ignored the change, I requested an ‘Import’ but no change, so the chart is correct PL5 will not read a PL5 DOP ‘Rating’ if there is a database entry it is ignored and changed back to 3* in my case on the next DOP update cycle.

However, perhaps the reason is because the DOP isn’t the only sidecar in the game, the export to the DOP also created an ‘xmp’ sidecar with Rating=3. So I removed the ‘xmp’ sidecar and tried to “persuade” PL5 to read the DOP sidecar file with Rating=5* but that did not succeed!

So I restarted PL5 and now I have a rating of 0 because although there is a DOP present PL5 is taking the data from the photo itself i.e. Rating=0.

At last I think I have finally walked into the exact scenario that you @Johnny reported in your original post!!

So on my system 4.3.4 DOPs get a preferential treatment and 5.1.2 DOPs are stonewalled!! While I sort of understand the logic I am interested in situations were users believe that this is going to disrupt their workflow.

Currently @sgospodarenko there is no way of removing a database entry from PL5 without deleting the database and the disk data. We really do need the ability to delete/expunge database entries sooner rather than later.

A “trick” that I have been testing with the procedure I proposed for executing synchronisation on PL5 systems is to use renaming to push one set of data to one side and introduce the other set of data. In the tests I have been renaming directories but the same could be used for individual photos but…

EDIT2:- This is a change in behaviour compared with PL4. Removing the xmp file and changing the DOP actually changes the PL4 entry when a ‘Sidecar’/‘Import’ is executed!

Changing while PL4 is shut down causes no automatic reaction on restart but ‘Sidecar’/‘Import’ seemed to fail once and succeed on the next test so more work necessary!

[Sandersoni]

Thank for the table.
Looking forward to the FAQ.
Can the .dop‘s „Processing Status“ tag be passed on to the xmp sidecar?
Would greatly enhance transparency when working with third party file viewers.

[sgospodarenko (Svetlana G.)]

@BHAYT

I opened the directory in PL5 it found and displayed the images with rating correctly and then hung @sgospodarenko so the hangs are related to PL4 DOP processing with JPG images perhaps.

• Yes it could be, you will be able to test it with the next release where this issue was fixed.

@BHAYT

I effectively change the name of a directory to the name of one that was itself renamed (conceptually deleted) and PL5 showed the images for the previous incarnation of the directory!! This was then resolved on a restart so the software is confusing itself and using from a cache perhaps @sgospodarenko!!?

• Was it a folder or a project in this case? It should not happen with folders.

Regards,
Svetlana G.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@sgospodarenko thank you for your response. I welcome the news about the fixes to the hangs they have been a real nuisance with respect to testing.

@sgospodarenko

Was it a folder or a project in this case? It should not happen with folders.

It was a folder as shown in the following incident

The test was actually “flawed” because rather than having two copies one on each machine I actually ran with T (the Test machine) accessing the files on M (Main machine) across the LAN but I decided to continue and revisit the actual “design” another day. Two things happened

1. The “ghost” images of the previous incarnation of the directory (the name of which had been changed to “delete” the files safely while the operation was executed) apparently appeared as the images of the directory whose name was changed to “switch” the replacement images into place!! They vanished after a restart!!!

2. The Tags appeared to get lost in the process.

I am getting way, way, way, … too old for this type of testing (mainly for not keeping notes so that I can unravel the snapshots I take at every step!!!).

Can I have a donation for the disk space I am consuming for these tests, only joking!

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Sandersoni not least because I appear to have lost the tags in the Synchronisation test I undertook where I reported another fault (a “possible” variant on my PL4 complaint about Ghosting or a completely different problem) in the post above this one.

[Johnny]

Hi Keith,
thank you very much for your reply! I did not expect that the database access is that easy and not secured or encrypted. I simply used HeidiSQL which I normally use to connect to a MariaDB Server. I had never worked with SQLite before but I really like the concept.

What is really interesting: I expected that the Item properties like rating are not processed because I changed the drive where my photos are stored. But the target folder entry is correct and all Rating entries for my PL 5.x edited Photos have a ‘NULL’ entry. So it was definitly a bug which obviously overwrote all rating entries in the database (they are still available in the PL 5.x dop files but not read)

What I could do is to write a programm which goes through all entries of the Item table and checks if a dop file exists for them, parses the dop file for the rating and then update the referring column in the Item table. But actually this should be done by PL since I payed for the Software

[Johnny]

Hi Bryan,
no need to apologise at all, thanks for your replies!
I use Windows 10 and I also experienced that PL4 dop file ratings are available in PL5.

[BHAYT (Bryan [WIN10, PL7, VP4, FP7, G9 & EM1 Mkii, RTX 3060])]

@Johnny In my tests I think that I discovered that PL5 takes the the PL4 DOP ‘Rating’ in preference to the photo (embedded or sidecar) but the photo ‘Rating’ in preference to the PL5 DOP. However, while that is acceptable in my test because I am hacking a DOP which should be identical to the database (the Uuid test will be passed) in the case of you presenting effectively new files to PL5 then the DOP should be taken into account in some way, even if that is “gated” by a date/timestamp check etc.

This is actually a bug @sgospodarenko and I will try a test to verify this situation - most of the data is already present on my system and just needs moving to a new folder and “ingested” by PL5!

[Joanna (Joanna Carter)]

@BHAYT

1. You are trying to outdo me with the longest post!

Would I?

@BHAYT

I cannot reproduce it because it does not happen on Win 10, or rather on Win 10 on my Main machine.

If you followed my procedure, to the letter, doing all the intermediate tests, and not getting the same results, this indicates a difference between Mac and Win and, instead of being a purely UI issue, this is a matter of data integrity.

@BHAYT

PL4 and PL5 play happily together with the DOP making its way to PL5 and PL5 leaving it alone so I can pick it up happily in PL4 until I make an edit in PL5.

Just to make sure, I reran my test

Preparation

1. Delete databases for both PL4 and PL5
2. Delete DOP and XMP files for test image
3. Check Rating was not set in RAW file by camera

Set Rank in PL4

1. Open image in PL4
2. Set ‘“Rank” to 3
3. Close PL4

DOP file after setting in PL4

	Software = "DxO PhotoLab 4.3.6 72",
        …
			Rank = 3,

Read Rating in PL5

1. Open PL5
2. Notice no stars for Rating
3. Close PL5

Unless I make a change to the image adjustments, PL5 doesn’t update the DOP file!!!

Write Rating in PL5

1. Reopen PL5
2. Set Rating to 2
3. Close PL5

DOP file after setting in PL5

	Software = "DxO PhotoLab 5.1.1 52",
        …
			Rating = 0,

Rank of 3 has been deleted and Rating of 0 has been added!!!

Reopen PL5

DOP file now contains…

	Software = "DxO PhotoLab 5.1.1 52",
        …
			Rating = 2,

XMP file has been created…

         <xmp:Rating>2</xmp:Rating>

So, it looks like neither the DOP file is updated, nor the XMP file is created until PL5 is reopened for a second time after the Rating was set.

However, if after setting the Rating in PL5, instead of closing PL5, I switch to Finder, to look at the folder that contains the file, Whilst I am switching, the DOP file gets updated and the XMP file gets created and I see it appear in Finder.

Crucial difference

• closing PL5 without changing focus to another app does not write either the DOP file or the XMP file
• changing focus to another app does write both the DOP file and the XMP file

Curiouser and curiouser, as Alice would have said

[platypus (Canon & Mac (Intel and M1))]

I set DPL to NOT automatically import/export/sync sidecar files.

When I …

• change a rating in DPL4 to ***
• manually export the settings
• manually import .dop sidecars in DPL5

…the changed rating comes along correctly.

When I then …

• read .xmp sidecars (no rating present), rating is set to 0
• read .dop sidecars from DPL4, rating is set to *** again

Summary:

• DPL5 does what I expect with .dop sidecars from DPL4
• DPL5 does NOT the same as above with DPL5 .dop sidecars

Conclusion

• DxO needs to fix this bug