Thanks for confirming the finding. It’s independent of OS then.
Yes it is a consequence of not writing the xmp data
- Ever
- When a change has been made in PL5 but not written back to photo
- When a change to a photo has not been re-imported into PL5
etc. etc.
I am afraid that my preference is now for using the ‘Sync’ option because it avoids so many problems!
So:-
It is not a bug it is a consequence of the changes in metadata handling that have come about with DxO “abandoning” the immutability constraints of the pre-PL5 era and the impact on users who were used to that way of working (because that was all that was on offer - then).
It is the result of DxO not making it clear, scenario by scenario what was going to change and how old workflows could or should be changed.
It comes from DxO not interacting with their UserBase to understand what those workflows might look like @sgospodarenko (sorry I keep using your @ too often, please find me another DxO @ to “annoy”).
It comes from us (that’s you, me and others not always understanding properly and communicating to other forum members) but that could be helped by some (any) assistance from DxO personnel whose help might shorten the long learning and realisation process! In fact we typically work alone on tasks that would be better handled as a team but that has a number of issues and is not easy to organise so lets continue to battle it out as we have been!
It comes from a slightly dysfunctional forum where the same ground is being covered again and again and the message is still not getting across.
It also comes from forum members not always explaining the reasoning behind the recommendations they make etc. etc.
And sometimes forum members actually not describing the problem clearly enough for others to recognise it is the same problem as …
and so on!
Activating the sync of .xmp sidecars might solve the rotation issue, but you risk e.g. the integrity of metadata that you added in other apps. Sync cannot be set one way as with .dop files and we’ve seen the database ironing over info in sidecars, which does not make me trust in DPL in this area just yet.
@platypus I understand your concerns and reticence over adopting the ‘Sync’ option and currently we have a choice between that option and using the two manual options and in the context of ‘Rotation’ (and ‘Ratings’ etc.) I am concerned that the manually controlled writes will simply not take place, i.e. the user will simply forget to force the writes or will get most but not all!
One problem with PL5 is the lack of icon on the thumbnails like those present with Lightroom and the additional icon that indicates a mismatch of keywords. However, essentially LR can ‘Save Metadata to File’ and ‘Read Metadata from file’ as PL5 can. However, there is a ‘Catalog Setting’ under the 'Metadata, Tab which is identified as ‘Automatically write changes into XMP’ which appears to be an automatic ‘Save to file’ option.
Capture 1 offers preferences for ‘Auto Sync sidecar XMP’ and the choice is ‘None’, ‘Load’ or ‘Full Sync’ and under the ‘Image’ menu item there is ‘Load Metadata’ and ‘Sync Metadata’.
Photo Supreme offers in the ‘Preferences’/‘Synchronize settings’ ‘Only update out-of-sync images (uncheck to force)’ and ‘Store metadata to the database only (uncheck to allow metadata to be written to files)’. For an individual file a right click on the thumbnail offers ‘Save Metadata to File’ and ‘Read Metadata from File’. However, in the ‘Tools’ submenu there are additional commands ‘Save Metadada to File for all Out-of-Sync Images’ and 'Read Metadata to Catalog for all Out-of-Sync Images. I have just installed Photo Supreme Lite on my Main machine which appears to have all of the features of the full product but with a limit of 5,000 images to a Catalog, which is plenty for testing!
Comparing with other products does not help right now but may help guide future DxO development in the “best” direction (if any such development is actually going to to take place)?
There is a question that you or @Joanna can answer for me with respect to database (re-)location on the Mac, can the database be located on a specific directory on a specific drive or can it only be located on/at the default location? The snapshot below shows my current PL5DB location on a USB3 connected SSD designated as T:.
I’m sorry but I am fine of those who uses my simple keywording/rating app to write directly to RAW files. I use DOP files and have absolutely no need of the database and regularly destroy it to avoid any conflicts that PL might throw up.
Nonetheless, as a nosey software developer, I went digging and found that it is indeed possible to change the location of the database on a Mac, but it’s not something I would advise any non-nerdy type to do without a safety belt and crash hat 
The location is buried in a .plist file and I just changed it to my desktop and PL5 seemed to work fine.
I don’t use the .xmp files, because I don’t need them.
And likewise I don’t use the internal database for searching.
I need a solution where I can use the OS commands
for copying an moving files, else it becomes impossible
to make a backup, or move a whole image directory tree
to a new device.
The loss of the rotation info, and maybe also the rating info, in
PL5 is simply a bug, not a feature or a new algorithm.
@ups How could the features you want be implemented? If you export from PL5 to JPG, TIFF (TIF) or DNG then the ‘Rotation’ and ’ Rating’ are preserved with the image and available to other display/editing/DAM software. With RAW image files PL5 adopts a common strategy of storing the data in an xmp sidecar file which makes those fields, which are not unique to PL5, available to all software that follows the rules to be able to access it and to use it.
Prior to PL5 this data remained alongside the DxPL editing data in the DOP, as such it remained “hidden” to programs that could usefully use that particular data. With PL5 it has been “liberated” from the confines of the DOP and made available to a wider range of software. Unfortunately the change has caught out the users who were unaware that they need to preserve the xmp sidecar files and not just the DOPs.
While I agree with you about databases restricting the fluid movement and renaming of files, directories etc., one reason that I have avoided a DAM for many years. To be honest PL5 fairly easily forgets about a directory if you change its name or move to another device entirely but …
Now back to my original question how do you intend to use the data that was in the DOP and is now in xmp files externally, ie. via the OS? In fact on Win 10 I have configured my file manager display to include ‘Rating’ and ‘keywords’ which work fine for those files with embedded xmp (JPG, TIFF etc.) but less well for RAWs (albeit all software built to handle RAWs can also handle an xmp sidecar file at the same time - but not a DOP).
Sorry I beg to differ, it is a change and arguably one that opens that data up to a wider audience than existed before. If you have built software that accessed that data in the DOP then those fields are still retained in the DOP but as “read only fields” so that software will continue to work.
I don’t consider it a bug, the bug was the lack of communication about what the changes might mean to current users workflow or back up strategies and your post does not indicate what the actual impact is to your work flow or backup strategy.
There is more software around that understands keeping RAW and xmp sidecar files together than understands keeping RAW (JPG, TIFF & DNG) and DOP files together (albeit the better software allows for customisation to treat .DOP sidecars in the same way as 'xmp sidecars for moving, copying. deleting etc as a group).
I just installed PL 5.1.2 and noticed, the bug concerning rotation
is still there, and it’s still a bug, not a new feature.
@BHAYT said in the last post the Rotation field in .dop files
is still there and it’s a read-only field. No it’s not a read-only field,
it’s a write-only field.
When taking a copy of an image and the .dot sidecar file, e.g.
with the finder app or in a terminal, in the copied .dot there is the right value,
but when opening the directory containing the copied image with PL
the PL Library does not read the Rotation value in the .dop file, it immediately
overwrites it with 0 and the image shows up with the wrong rotation.
Concerning the implementation: The source of the trouble with the rotation (and maybe the rating)
is the introduction of redundancy by using .dop files, .xmp files and
the internal database to access and change the same data.
With early versions of DxO there was the single source for editing and metadata info, the .dop
file. Now the rotation and the rating and maybe other info are stored within .xmp, .dop
and maybe also the hidden internal database.
All of us, who worked a few years in the software industry, know that such a kind of redundancy is a source of ugly errors (not bugs) and a source of a lot of trouble for users and developers.
Here are a few rules, which maybe useful for solving these problems in PL:
Rule 1: If PL finds a never before seen image file and there are no .dop and .xmp files,
the only source for metadata is the EXIF data in the image file.
Rule 2: If PL finds a never before seen image file together with just a .dot sidecar file, no .xmp, use the data in the .dop file and nothing else, nothing from the internal database.
Rule 3: If PL finds a never before seen image file together with just a .xmp file, use the data in the
.xmp file and nothing else.
Rule 4: If PL finds a never before seen image file together a .dop and .xmp file, ask the user,
which one contains the up-to-date info.
Rule 5 (concerning the internal database): Use the internal database only as a cache for meta info,
and assure that the cache is consistent with the sidecar files. Check this e.g. by timestamps of the
sidecar files or by checksums (MD5 ore something else). But never use the internal database
for more than a cache, because there isn’t a chance for the user to make a backup or move
to another machine and edit the images there (At home i work on a 27" iMac, when travelling on a MacBook with an external disk, and I would like to do this in the future).
@BHAYT: I hope, you see there is a (not too complex) solution for these kinds of bugs. But by introducing redundancy, one can mess up a well running system very quickly.
I hope to see the bug to be fixed in the next version of PL.
@ups From the tone of your post you appear to be under the misapprehension that I work for DxO, I am a user of the product like you and also had a career in I.T. starting with touching my first computer in 1964 when I started my degree in Computing and finishing with 36 years with a mainframe manufacturer where my speciality was database design for 24/7 real time systems.
I have been testing PL5 and reporting those results back to users who are experiencing problems and to DxO and I am not an apologist for DxO mistakes.
The issue of replicate versus replicake versus … have been discussed in the forum since the release of PL5 of the apparent WOM (write only memory) nature of certain fields in the DOP. My mistake in the response to your post was in error when I wrote “read only memory” instead of “write only memory” for which I apologise.
I once believed that the DOP was the principle source of data until I was investigating the “unwanted virtual copy” issue when it became clear that the database is considered to be the master and the DOP entry the “copy” if the Uuid’s do not tally, the [M]aster and the [1] etc. (virtual) copy.
While testing I frequently throw away the database and so DOPs are very important to me and once described the whole of the sphere of influence of DxPL but that changed starting with the first release of PL5. When DxPL changed to externalise the data to other software via xmp data (embedded or sidecar) then the DOP data effectively became “redundant” for the [M]aster but continues to serve its original purpose for the Virtual Copy entries.
DxPL continued to maintain the [M]aster entries but effectively as WOM fields, hence all copies including the [M]aster continue to have the same structure but for the [M]aster the data used is actually that from the xmp (wherever that is held - embedded for JPG, TIFF, DNG & RAW - sidecar for RAW only) and for the VCs, which have no external presence, in the DOP.
The biggest failure was that DxO did not “make a fuss” about the changes and draw attention to them in the documentation which meant that users were not updating the xmp data but only preserving the DOP along with the photo which is insufficient if the xmp has not been written and/or not been preserved!
That should be extended to include the xmp data which is the file, even in RAWs it is possible to put ‘Rating’ etc. in the xmp embedded in the RAW
If the file is new then there is no entry in the database to ignore however, in PL5 the rules have changed and the DOP data will be ignored and PL5 will go searching for the data in any xmp sidecar and in the embedded xmp data of the image, including the embedded xmp data of the image file itself. If it finds no data for ‘Rating’ and ‘Rotation’ in that xmp data it will assign a value of 0 and write that back to the WOM fields in the DOP.
The requesting the date would be an acceptable solution, I have previously suggested that “metadata timestamps” and DOP timestamps could be used to automate this process.
Currently for pre PL5 DOPs, PL5 automatically uses the DOP and ignores and xmp data, this is also wrong and I suggested the date fix for that situation.
I am afraid, like most other editing systems with databases, that DxPL consider the database as the main source of data and the DOPs are a useful, more portable option for users. At least it provides DOPs. Currently when a DOP is presented to a database there is a check that takes place and that is the Uuid (which I have discussed elsewhere many, many … times). If the test fails the Virtual Copies result and I have requested Uuid override and the ability to promote a Copy to become a [M]aster and but I am a user and they will act upon or ignore my suggestions just as much as yours!!
@ups I have discussed and tested various strategies for this and documented them elsewhere, I need to make the most of the fine weather but I will provide references to my other posts in another post.
In the meantime the changes are inconvenient but open DxPL to receiving and transmitting data to other products and for that data to travel with the image in an industry standard form, whereas the DOP is ultra useful but proprietary and downgraded with PL5 to being the source of editing data for the [M]aster (+ Tags which have a bug as well) and the source of everything that it used to be for all the Virtual copies, for the [M]aster the source of that data is now xmp data (sidecar or embedded) which must be written from PL5 and preserved with the DOP!
Regards
Bryan (just a user of DxPL - not a developer)
This give’s headache …
@BHAYT
@ups
Don’t yet have this kind of problem, and i even don’t completly understand what you’re talking about.
My question is : is it needed to read all this topic to be sure not having problems later ?
1 Like
Yes, but would not like to see later my datas bugged for a reason I don’t know.
And i don’t uderstand in detail what it is about. But I understand something can lead to bug.
So … What to do to have no problem ? And what is it about exactly ?
What I understand is that it is about using several softwares. Right ?
And then it become more confuse …
I even don’t know why some delete the database on a regular basis (which des not involve several software, if i’m right …
Can someone help ?
@JoPoV I’m sorry the document was not supposed to give you a headache and thank you for translating it, my French goes back to my school days and that was almost 60 years ago!
If you use ‘Rating’ in PL4 or earlier you will have been used to being able to restore the values with just the DOP, i.e. it held the your edits and certain other bits of data, ‘Rating’, ‘Rotation’, ‘Tag’ and ‘Keywords’ etc. in the DOP. Hence, all you needed to restore everything back into DxPL was the photo and the DOP.
But on PL5 this data is now held in the xmp data where it can be shared with a host of other packages that understand the same protocol. This is a change that has caught some users out. If you use none of those items then don’t worry.
Please note that there is currently a problem with the Tag (accept/reject etc.) because this is an item unique to PxPL but PL5 is actually not handling it in the DOP correctly (it writes it to the DOP but does not read it back!)
So on PL5 if you are using ‘Rating’ etc. you need to set the options that update the xmp data and secure the sidecar file that will be created for RAW photos as well as the photo and the DOP (one extra file per RAW photo). Provided the options are set then the data will be updated back into the photo for JPG, TIFF and DNG so you only need to continue to keep the photo and the DOP.
Is that any better?
@BHAYT
I don’t really know what I use.
I thought database and dop are used even if I do nothing. And about xmp it’s less clear for me. I think to understand it is an option that as to be activated ?
what is PxPL ?
The sidecar file isn’t the dop file ?
terminolgy is not yet clear for me …
Maybe before knowing if I have to understand all this, I should know when this has an impact.
What situation can lead to bug ? Maybe I’m not concerned yet, and just have to know when I’ll have to be concerned ?
Can you tell me ?
PxPL is a typo I am sorry it should have been DxPL (currently PL5 etc.)
Please answer the following questions
1,. Do you use ‘Rating’?
2. Do you use ‘Rotation’?
3. Do you use ‘keywords’?
If the answer to these is No then don’t worry just keep the DOP and photo and don’t bother to read the rest of this post!
The DOP is a proprietary sidecar file that is unique to DxPL and has been for a very long time and is read by no other program. The DOP is the sidecar file described in the following ‘Preferences’ setting
If you want to look inside a DOP you can open the file with a text editor and will find something like this:-
The xmp sidecar is created for RAW files and contains the ‘Rating’ etc. in a way that can be read by other programs. Typically (not always for some programs but always for DxPL) the RAW file will not be changed and an xmp sidecar file will be created. ‘Rating’ data will now be written back directly to JPG, TIFF and DNG files so an xmp sidecar is not required.
This xmp sidecar file can also be opened in a text editor and looks something like this;
Before PL5 (i.e. PL4, PL3 etc.) the data was held in the DOP but now we apparently have two competing sidecar sources of data the DOP and the xmp sidecar and it is this which is causing the confusion amongst users. Both the above were taken from my test data for the same file.
@BHAYT
First, I thank you for the time you spend to explain it to me.
I use all.
And I’ve well understood what you’ve just tell me (didn’t know dop was an ascii file).
So now, when all this can bug ? Doing what ? Because it seems that some people have bug ? So what use of photolab can lead them to bug ? (or way of moving datas, or … I don’t know).
What is dangerous ?
This is what worry me. Hearing some people have bug, and not understanding when. I wouldn’t like this happen to me.
Or is the reason unknown, and you’re just trying to repair something broken no one knows why ?
I am lost in fog.
@JoPov I am sorry to hear that you are using all of those items!! Incidentally what operating system are you using Win 10/11 or Mac?
So my “argument” with @ups is whether this is a bug or a feature.
The truth is that it is a “feature” and a direct consequence of the changes within PL5 where this data is to be made available to other programs. e.g. Lightroom, ACDSee, Adobe Bridge, Photo Mechanic etc. etc. rather than be “locked away” in the DOP.
Even if you are not using any of these programs the changes that have been made to “interface” with them means that things have changed for PL5 users. @ups was suggesting a way whereby the “old” way could co-exist with the new way and I have made similar comments in other posts of mine.
However, while there may be a change it is not coming anytime soon so what needs to be done is to set the highlighted option below ;
This will then update the xmp data and create an xmp sidecar file for any RAW images when you assign values to the fields I identified. That xmp sidecar will only be created for RAW photos and must be treated in the same way that you currently treat DOPs when backing up your data. Now for recovery purposes you must present the photo + DOP sidecar + xmp sidecar for all RAW photos that have been assigned these values with PL5 software.
Please tell me if this value is already set, and when you think it might have been set!?
For all photos that were assigned those values pre-PL5 only the photo + DOP sidecar is necessary.
Unfortunately there may be issues with this “cunning” plan and your data may already have been damaged!?
Issue:- The Sync option takes any changes made to the values externally and updates the PL5 database and takes any values assigned in PL5 and externalises them via changes to JPG, TIFF or DNG photos or by the creation and adjustment to an xmp sidecar file, but at the time that the change is made!
In testing I believe I have seen cases where PL5 did not keep up with changes I made through an external program but … it was not predictable and worked a lot more times than it failed!
Potential problem: If you have allocated values to any of the fields post the installation of PL5 then there is a risk that the values in the DOP (and the database) could become 0 if you discover the data afresh, e.g you used the photo + DOP to recover a lost database or you are carrying data between two machines running PL5 etc.
It also means that you need to play “catch up”, i.e. if the ‘Sync’ option was not set then any changes made will only be held in the database. The DOP is now being treated as Write only for these fields!
The catch-up means that you need to “flush” any changes made to those fields out into the xmp data (sidecar etc.) and to do that you need to use the following command, the ‘Metadata’/‘Write to disk’ command;
Unfortunately DxO has not provided a simple “flush all” command, or even a directory “flush” command (both of which I have asked for and @ups’s idea of working with DOPs alone would have made all this writing on my behalf unnecessary!)
I am sorry that it has suddenly become complicated but if I only tell you half of the story you won’t thank me for not telling you the rest!
Please query anything that you want to and I will try to help all that I can.
1 Like
@JoPoV if the above post gets too complicated too quickly then the following might walk through the issues in a better organised manner (but it is still a long post) What is the best way to move a folder of photos with their respective *.DOP and *.XMP files? - #15 by BHAYT.
It might also be of interest to @ups and contains pointers to other posts where I try to explain the issues of moving data between systems. In these and the cache posts below the issue of retaining and moving the xmp sidecar file with the DOP and photo is missing because we had not become fully aware of the significance of writing and preserving the meta data when those posts were written.
Please remember that all that has been written has only been determined by testing and re-testing and not by any code inspection and with limited information from DxO.
Yet another set of posts that relate to moving data are here Cache file - #20 by BHAYT, Cache file - #22 by BHAYT, Cache file - #23 by John-M and Cache file - #29 by BHAYT.
Typically all the posts are long which stems from my I.T. days where I learned to explain the problem and the underlying reason that the customer was suffering that problem, then describe the potential outcomes of encountering the problem, then describe any ways of avoiding the problem preferably ones that had been tested and then “promise” a date by which it will all be fixed. This is not my product so I cannot make any such promises and arguably we are looking for additional features that make certain tasks easier and the risks of encountering problems much less!
At least I was earning a salary for that work and the customers were paying millions for their kit and for its support (hardware and software).
1 Like
Not time to look at it now. Because have to check several things on several external drives. I will tell you later how I organised things and it will be clearer for you.
Thanks.
Will check all this later.
And again, thank you very much for your time.