PL5: Moving files outside PL loses corrections and more

Without going into all your extensive detail - - I can confirm (with confidence from long-time dependence ONLY on sidecar/.dop files - instead of the database) that ALL correction details are held in the sidecar file associated with each RAW/Image file, and that PL correctly reads and applies all of it.

PL does NOT restore metadata (projects, keywords, ratings) from sidecars - because this information is not stored in sidecar files … but, personally, I don’t care about that because I don’t use those features.

John M

1 Like

In another post somewhere in the forums I complained that when a PL4 DOP is present then PL5 will only take the metadata from that source when it first discovers an image with a PL4 DOP!

This statement was correct with respect to the data I was testing at the time, namely ‘Rating’ and ‘Rotation’ but was actually incorrect when other metadata is factored into the equation as I finally discovered.

I have run a series of tests adjusting timestamps to see where PL5 will take the metadata from and whether timestamps have any impact.

I have done a number of tests and then decided “to get a life” but the situations tested were essentially

  1. Image with a PL4 DOP introduced to PL5
  2. Image with a PL4 DOP + xmp sidecar introduced to PL5
  3. Image with a PL4 DOP + embedded xmp data introduced to PL5
  4. Image with a PL4 DOP + xmp sidecar + embedded xmp data introduced to PL5
  5. Image with a PL5 DOP introduced to PL5
  6. Image with a PL5 DOP + xmp sidecar introduced to PL5
  7. Image with a PL5 DOP + embedded xmp data introduced to PL5
  8. Image with a PL5 DOP + xmp sidecar + embedded xmp data introduced to PL5

However, with the PL4 DOP in particular my original results changed when ‘Keyword’ and ‘GPS’ data was available in the xmp sidecar and/or the embedded xmp data.

So briefly, PL5 takes the data that used to be saved in the PL4 and earlier DOPs from the “older” DOPs but takes the keyword and GPS data from the xmp sidecar or the embedded data!

I believe in all cases that PL5 takes the sidecar in preference to the embedded data, this is common amongst most of the other DAMs etc. with the exception of Bridge I believe and I am not sure about LR, which favours the embedded data (or so it seemed in my tests!)

Timestamps appeared to be ignored by all the software I tested I believe (again needs verifying). Believe because contriving such a test is mind numbing in the extreme and a simple slip means going back to the beginning re-evaluating the apparent results and going back to the beginning again.

The main takeaway is that if you have an image with ‘Rating’ or ‘Rotation’ data that has been added after you last used PL4 and discover it in PL5 then, for these items of metadata, PL5 will (probably) take the data from the DOP but

  1. If your database was taken over successfully from PL4 into PL5 that data will already be in the database anyway.

  2. I need to verify what happens if there is a PL4 DOP with ‘Rating’ set to 0, i.e. does that “block” a value in the sidecar or the embedded xmp data (I suspect that it will).

Thank you John.

Do you really think I like going into such excruciating detail! My first job after graduating was teaching students the same age as me and older, first as a Research Assistant and then as a full time lecturer, they were one of the harshest audiences ever! I then left to see what computing was like in the “real world”!?

@platypus as you will know from other posts I have made my speciality was Mainframe Database design and support but involved being a Project Manager, Projects Manager, pre-sales analyst, finally ending as a Systems Architect for the last 16 years working on what was then the Orange Voice Mail system for the manufacture/supplier of the product.

I have been adding summaries to my posts recently but wrote this some days ago and finally decided to post it as is! I will create the summary you suggest in the next few days.

The truth is that PL5 is certainly not perfect, simply that it is not necessarily suffering some of the ills documented in this topic.

In truth some of its failings are also present in other products but PL5 should have sought to go beyond what they do not just try to emulate them and fall short in some areas because of the size of the task.

It has such promise but it is falling short and tests like the ones I was trying to run in the list above should be unnecessary!

It should be easy to ask the question and receive a response from DxO, not necessarily immediately (they have a job to do after all) but with some certainty that there will eventually be an answer forthcoming! The FAQ promised has never materialised!

Testing certain scenarios is truly a nightmare but when they have the code to hand!?

This is a sample of the test Directories I was setting up

The longer the statements the smaller the chance they are read & taken noticed.

2 Likes

Agreed >=10

1 Like

Summary of my mega post above:-

  1. PL5 Reads ‘Rotation’ & ‘Ratings’ (and the Tag) from PL4 DOP, ‘keywords’ & ‘GPS’ from xmp sidecar or xmp embedded when first discovering a “new” image.

  2. PL5 never reads ‘Rating’ and ‘Rotation’ from PL5 DOP only from xmp sidecar or embedded xmp image metadata. The details are stored in the DOP but not read back!

  3. Tag also not read from PL5 DOP - a bug!

  4. PL5 not much changed from PL4 database so why should we suddenly be concerned that the database is “not fit for purpose”. Not much more fragile (possibly no more fragile) that the competition. All (except ACDSee use SQLlite as the database system of choice).

  5. The role of the PL4 DOP for ‘Rating’ and ‘Rotation’ now taken by the image metadata so that must be written and secured and retained as part of the DOP + xmp sidecar(where appropriate) + image package for “back-up” purposes! If it is never written back to the image any PL5 entered metadata will be “lost” (i.e. in the database only).

  6. PL5 does re-format the metadata under certain circumstances see XMP-files gets F*cked up due hierachical mismanagement in dual management - #39 by BHAYT for the most excruciating detail! Some caution with options etc. required.

  7. What do I have on my current bucket list for DxO when they have the time of course? Here are some of the items PL5.2.0.4730 Conflict Resolution only detects externally generated conflicts.

2 Likes

This stance is quite rude. Bryan went to a lot of trouble to run a lot of real world tests on this data and to even more trouble to share them with us. The reason both @John-M and especially @BHAYT posts are so long is that PhotoLab’s handling of metadata is too complicated.

John seems to be right (in my experience) that image processing data does travel reliably in .dop sidecars. On the other hand, where the metadata is stored and whether it is all stored there is extremely unclear. While Bryan seeks to shed some light via his tests, his findings are far more positive than my own. My own experience with PhotoLab 5 metadata is extremely worrying. Moving images with the .dop and without the database most of it disappears.

I’d love to see someone like @Stenis who really deeply understands image metadata standards and has substantial experience managing said metadata at scale in the real world test metadata portability and cross-application compatibility rigorously.

What PhotoLab 5.x or PhotoLab 6 should do is clear:

  • keep image processing data in a .dop
  • share image metadata (IPTC, ratings) in a .xmp

Why?

Image processing data is more or less proprietary (crop could usefully be shared with PhotoMechanic but that’s another more complex topic, what would be most useful here is if PhotoLab would read the PhotoMechanic crop data if it doesn’t have its own crop data, i.e. one-way sync) hence no need to share.

On the other hand, metadata includes a number of open standards and is very usefully shared among photo applications in .xmp format.

DxO’s mad rush to a database only world is a real step backwards and erodes a substantial competitive advantage. The kind of photographers who will invest in an expensive tool¹ like DxO PhotoLab treasure image portability. It’s time for us as the people who fund PhotoLab and DxO to step up and demand that DxO maintain and improve image portability.

Mailer’s limitation is that it does not work on multidomain and it’s old enough to be finicky and obsolete.


Footnotes

1. If DxO seriously hopes to chase the Adobe Lightroom masses, the first step would be to cut prices in half. That won’t (and probably shouldn’t) happen. Even at half-price, the “I’ve heard of Adobe, it’s really pro, everyone uses Lightroom” crowd won’t come. Thirty percent new users with three times as many support tickets (complex tools in the hands of less expert users enough generate many times more support tickets than expert users, we are a software publisher as well, this is first hand field experience) will not even start to make up for a 50% decline in revenues. Plus with DxO showing a stiff finger to smart phone owners for the last five years, thinking of mass adoption of PhotoLab is risible. The lack of reliable support for my iPhone DNG sometimes makes even me regret my choice of RAW developer.

2 Likes

@uncoy Thank you for your response and support, I hope you have recovered from reading my post!

The points that both @platypus and @Wolfgang made are valid and the risk is that without a summary and with a long post the potential audience may simply ignore the post as “too much” and the post will then fail to get the point(s) across and to impart any potentially useful information because it will not be read.

Is that my “fault” for writing a long post or the audiences “fault” for not reading it?

The answer is neither and both!! It takes way too long to write up the test results and the conclusions (with all the buts, maybes, caveats, disclaimers etc.) than it does to execute the tests even though it might take many attempts to actually get the test data into the correct format and to actually capture the moment that things go "wrong " etc…

My conscience is clear. I can only do my best to “torture” test the software until it breaks or, preferably, does not and publish what I have “discovered” along the way!

My biggest concern is that the software is within striking distance of being a useful product for handling metadata (not a full blown DAM, but I bought one of those at Christmas - IMatch) but I am seriously afraid that those who I really want to listen (DxO) are actually not taking any notice of the concerns expressed by me and others and are not engaging with us in any meaningful way.

I really only want a product that allows me to handle my limited metadata easily and reliably and continues to allow me to make the modest “fixes” that I apply to my photos.

Just tested the concept of using .dop sidecars as a transport medium.

  1. Modified all test images and engaged all corrections, aded star rating, GPS and IPTC metadata, which made the screen look like this
  2. Told DPL to export settings files, which it did, then quit DPL
  3. Duplicated the test image folder and renamed all files in Finder
  4. Opened DPL again, navigated to the new folder copy
    and imported settings manually, which made the screen look like this

I had expected to see the same screen as after step 1,
but as you can see, it did not happen.

I then renamed the sidecars with the Finder and let DPL export settings again and compared the old and new .dop files with BBEdit. Apart from different creation dates, the files were identical.

Observation 1:

  • DPL 5.2 imports settings and metadata, but does not apply them.

I then copied the .dop sidecars from the original folder to the new folder and renamed the sidecars so that they matched the filenames again.

On manual import of the .dop sidecars, DPL changed the previews to match the original (modified) images.

Observation 2:

  • It needed extra effort to convince DPL to actually apply the settings stored in the sidecars.
  • Even under these conditions, DPL did NOT show the following
    a) Star rating
    b) GPS coordinates
    c) IPTC fields
  • Actually, the missing metadata had been lost

All of the above done with DPL 5.2.0.56 on macOS 11.6.5 on Intel iMac 2019.

Can someone please try to reproduce the effect?

3 Likes

Thank you for your efforts, Platypus, systematically testing metadata portability. My own real world field experience supports exactly these results, although occasionally and mysteriously star ratings will change when modified in an external application, even while PhotoLab is open.

I repeated the test, but this time, I renamed the sidecars at a different time than the raw files. When I imported sidecars manually this time, the settings were applied. Nevertheless and again, Stars, GPS and IPTC were not read.

1 Like

A quick question @platypus, did you also copy xmp files? Non-PL data is stored in xmp files - data like IPTC, etc that you say is not copied.

It seems to me that DxO are seperating metadata and edit data into seperate sidecar files. Metadata goes into xmp files and edit data goes into dop files. Those is what I suggested way back before EA for v5.

The test was done without any xmp sidecars. As you can see in the text file screenshot above, dop sidecars contain the IPTC, GPS and rating metadata, but they seem to get lost when DPL reads the dop files.

If I remember correctly, DxO have duplicated that info in both sidecar files for writing but only used the xmp file for reading.

Yes, it looks like it. But why write stuff into files if the stuff is discarded anyway? If that stuff were kept, the .dop files were an almost perfect backup and transport medium. To me, the current decision feels like wantingly shooting one’s own foot :woozy_face:

1 Like

Yes, and if I remember well, someone from DxO team explained that metadata in .dop files were there to be used with Virtual copy…

So to be able to import metadata in DPL we must do it from xmp.

1 Like

Yes, I must admit (to be honest) that that’s the case for me … I have a short attention span !
At the same time, I do respect your good intentions …

John M

No need; that’s how I work all the time (with sidecar/.dop files only - having deleted the database before I invoke PhotoLab), and that’s how it works for me … Everytime … No problem (other than loss of metadata that you noted … and which I don’t care about).

John M

Why not just try/test the method we have suggested ?
There’s no danger in doing so, provided you keep your “from” system in place.

John M

1 Like

Thanks John. It will probably be a long time, if ever, that I get a clear, concise recommendation from DXO. So I have been thinking the last couple of days that I will start doing some experimenting.

I am very concerned about losing keywords though with your system. Some of these are many years old and are helpful to us. In fact that is actually more of a concern than losing edits, although I don’t want either to happen.

I will keep a copy of my entire library in a safe place and will start off working with a few copies. To be honest I am really bummed at DXO that any of this should be necessary. I’m also disappointed that they have not been participating in this and other related threads.

On the other hand this ‘Community’ is wonderful with the help of so many of you experts that share freely. Thanks! :slight_smile: Rod

1 Like

That’s the way to go, I reckon - you’ll learn more from trying it yourself than reading here.

Yes, with my proposal (that you let PL build a new database, on your new PC, from info in the sidecar/.dop files) you will lose any keywords held in the database on your old PC - as they are not stored in the sidecar files. Similarly, metadata (ratings, GPS coordinates, IPTC fields, etc) will not be transferred.

John M

1 Like