PL 5.2 Resolve Metadata Conflicts: Sidecar vs Image File

Well, yes and no. It is strictly limited to keywords, ratings and description via ExifTool. But, for searching it leans heavily on Apple’s own metadata framework and Spotlight search engine for ultimate speed.

Indeed. I have come to trust it and rely on it and have never, ever been let down. But then I do use the part of ExifTool that renames the original file before writing any changes to a copy and, should such a write fail, the original file copy its always available. Only when everything has been verified does the renamed original get overwritten by the edited copy.

Understood. I think the thrust of what proponents of ExifTool are saying is that there is an amazingly reliable tool that does write directly to RAW files with no measurable risk. There are also users who have a non-XMP sidecar workflow, who currently cannot use PL as a DAM due to its lack of support for for direct writing to and synchronising of RAW files.

If everybody used ExifTool under the hood, that risk is minimised to practically zero. Which is why I chose to use it in my app.

We are only talking about metadata, not image data. But then Adobe write all their changes to XMP files, mixed in with the metadata as well. Now that is what I call a mess.

Indeed but that only applies to the image data and the EXIF metadata written by the camera. Separating image edits into sidecars, whether they be DOP or XMP can also lead to wholesale loss of edits and subsequent loss of time having to re-process hundreds of files.

Mainly because nobody seems to be interested in adhering to a single standard for metadata tags. What we are talking about here is whether writing a metadata tag, which is in a separate part of the file, can corrupt the image data part of the file.

After all, for certain types of photography, RAW files are not feasible and we don’t seem to have a nervous breakdown writing to original camera JPEG or TIFF files.

If we are really that worried about protecting the integrity of the image data in an image file, why do we allow it for some formats but not for others?

Some cameras use DNG as their RAW format. PL happily writes metadata directly to them.

To be (RAW) or not to be (RAW) that seems to be the question :hugs:

1 Like

…I simply would like DxO to read from files and sidecars and offer a way for us to choose, which entry is the one to stick to in those cases, where we have conflicting information. This could mean that info in original files might need to be changed. Having a choice would allow us to do so - or not.

2 Likes

I don’t care where the metadata sits and consider the notion that it must sit only in the RAW or only in the xmp sidecar file an interesting academic/philosophical point. It is less about where is is placed, manipulated and updated that matters as much as the rules with which it is accessed, manipulated and updated!

I agree with @platypus when he wants options but the first thing I want before that is clarity @CaptainPO about what DxPL actually does. Ignorance of what rules DxPL applies is not bliss it is simply confusion which leads to anger which …

I have run tests trying to determine what the various bits of software do to access the metadata, which for DxPL can exist in the following places

  1. A PL4 DOP
  2. A PL4 DOP + xmp sidecar file
  3. A PL4 DOP + xmp sidecar file + xmp embedded RAW

It also exists in the PL5 DOP but the change in metadata strategy with the PL5 release leaves it as WOM (Write Only Memory). I do not disagree with the strategy that DxO has adopted but have spent the last couple of months trying to explain it to irate users who have “lost” their data!

Communication is everything and in this particular case there has been a single table which came too late for some!!

The PL4 DOP can hold the ‘Rating’, ‘Rotation’ and DxPL’s own ‘Tag’. The sidecar can hold somewhat more, i.e. the data placed there by any editing software or DAM (keywords and GPS in particular) and the RAW holds even more since it contains all the data put there by the camera in addition to any placed there by software packages that are ‘RAW update enabled’.

The “nirvana” that users are seeking is unobtainable until all software developers are “forced” to adopt the same set of rules in both the handling of that data and its subsequent placement.

So please ignore the pleas from even @joanna and concentrate on the reality that not all software developers are going to move to RAW updating and DxPL will have to live with data in the RAW and in the sidecar and, potentially, in both depending on the mix of software the user chooses for their workflow!

Concentrate on a scheme which has a sensible set of rules for “normal” use with the addition of options suggested by @platypus for the more experienced users.

I need to revisit my tests which I conducted after (erroneously) stating that PL5 only takes the metadata from the PL4 DOP (if present) and ignores all metadata from either an xmp sidecar or embedded xmp metadata. I don’t think the actual rules have been published.

The tests I conducted showed my statement to be true for ‘Rating’ and ‘Rotation’ but PL5 sought ‘keywords’ and ‘GPS’ data from other sources! I believe the test showed that it would take them from the sidecar file in preference to the embedded metadata in the RAW but if there was no sidecar it would use any data it could find in the RAW.

I would respectfully ask that the actual rules used are published! I guessed that timestamps might figure in the equation but the very presence of the sidecar seemed to be the end of the search. That was true for most of the packages I tested. One or two (Bridge certainly) favoured embedded even if a sidecar was present but I would need to try to repeat a nightmare set of tests again but publishing DxPL rules would at least clarify the situation for that product.

To the other users who have responded to this topic I would agree that if there was only one source for the data then the confusion would vanish but why spend time “normalizing” DxPL when many of the other packages do not offer the capability or do not offer it for all cameras, and DxPL must continue to co-exist with them!

EDIT

PS:

Most of the software that offer to update the RAW do so as an option and also offer the alternative of adding/updating the data in a sidecar file, one at least seems to offer an option to do both at the same time!

I personally do not agree that the raw file should be updating. After all that is your master file. I agree with @Joanna, the dop file should only contain photolab edits and its updates and the XMP file should only contain the exif data and its updates. The database should read both the updated dop file and XMP file and update itself.

If the database file gets deleted. The new database file should read both the DOP and XMP files. If XMP file gets deleted. It should rewrite a new XMP file. The same with the DOP file. Hopefully nothing gets corrupted this way and even though there are 3 files involved, it still becomes a SPOD.

If any outside dam programs have updated the XMP file. The XMP file should be re-read by the database, keeping everything in synchronisation.

Actually, I’m definitely not pleading for everybody to start putting metadata in RAW files. It’s just what I do because, for me, it makes life a lot simpler

Personally, I wouldn’t touch PL’s DAM with a bargepole, apart from beta testing it to try and help make it better.

When it comes to metadata, I exclusively use my own app, which manages keywords, descriptions, ratings and macOS Finder Tags. It uses macOS own metadata reading and search engine in combination with ExifTool. It views my images in a flattened hierarchy, which is so much easier than having to keep on switching folders. It cost me over two years hard work to write, but it does exactly what I want, no more, no less.

However, given my now extensive background in metadata handling, I find myself hoping for all sorts of things in how PL handles things, sharing how these things are easily possible, but constantly frustrated by the lack of response from DxO to what several of us are talking about in these forums.

Where I stand (and as @Prem mentions)…

I feel that DOP files are there for image editing data - only - no metadata.

People should have the choice, at their own risk, to write and synchronise metadata from RAW files, which, for them, should be the only place that metadata lives.

If people are not happy with writing to RAW files, then metadata should be written and synchronised only from XMP files.

Metadata should never be written to either DOP files or the database and the database should be used only as a temporary cache to speed up searching and possibly to save previous search results.

Any external changes to metadata should be detected, either from RAW files, if they have been changed and if the user so chooses, or from XMP files as a general rule for everybody else.

For those who use a RAW file workflow, if DxO are not comfortable with writing to them, then maybe metadata should be read only (discuss)

I agree and that is exactly why PL5 uses an xmp sidecar file in all circumstances where the metada is written back to a RAW image. JPG, TIFF and DNG get the embedded metadata updated directly.

And where does the metadata for the Virtual Copies go!?

Sorry but I see no special reason for changing the way that it currently works, except to provide an emergency option to use the DOP metadata from the [M]aster image in emergencies! I do not have an aversion to data being in more than one place but do have an aversion to SPOF (single point of failure).

The dangers of replicake (sorry replicate) data (if I use that one more time in a post I should be fined) I fully understand but the dangers are more about strict coding standards than anything else!

That is exactly what happens. The fiasco I referred to above is that PL5 “abandoned” the use of the metadata for the [M]aster from the DOP (which it continues to maintain, correctly in my opinion) and now takes that data from the image metadata but failed to make that obvious to all users at the release of PL5 and many users fell into a “trap” by assuming that things worked the same way as PL4!

PL5 will read the metadata from the xmp sidecar file and from the xmp embedded in the RAW (I believe that the existence of the sidecar file automatically ends the search for metadata, whereas I would prefer a mechanism controlled by timestamps, i.e. the data with the latest timestamp “wins”), just as you stated.

But what do you mean by “rewrite the xmp file”, I understand you are referring to the ‘xmp’ sidecar file but what scenario are you referring to in this case.

Currently if the xmp sidecar file is lost and the PL5 database is intact it is in a position to be able to recreate the sidecar file but all of this then depends on the options you have selected for metadata handling in PL5, i.e. AS(ON) or AS(OFF) and the use of ‘Read from image’ and ‘Write to Image’!?

I haven’t actually tested this exact scenario (I will add that to my list after some DIY) to know exactly where PL5 will choose to take the data from, i.e. will the database always win out, or will the embedded metadata win out (if any exists) or will timestamps play a part (extremely unlikely!).

You get to choose the options AS(ON) (‘Always Synchonize’ = ON) or AS(OFF) in the ‘Preferences’ and regardless of the option setting you can always use ‘Files’/‘Metadata’/‘Read from Image’ or ‘Files’/‘Metadata’/‘Write to Image’ but these operations replace the database data or the image data completely.

Please see my post at PL5.2.0.4730 Conflict Resolution only detects externally generated conflicts for the options I would like to see added to PL5 to further improve its metadata handling.

So, why don’t let the choice to the user in Preferences.
Like that you will be happy, and those who want to have all included in Raw will be happy.

Again, putting metadata directly in Raw is the way I work since 20 years and I’ve never had an issue with that.

To my mind, this is where DxO could get clever.

XMP stands for eXtensible Metadata Platform.

There is nothing in the world to stop DxO from providing their own metadata sections in an XMP file, in order to contain metadata related to virtual copies.

Surely it’s not such a crazy idea, given that Adobe writes its image edits to an XMP file?

@Pathal I am glad you have been successfully using metadata added directly to the RAW for so many years and if DxO ever add it to DxPL it should be added to the product as a ‘Preferences’ option as it is with those products that support it.

But I simply consider that to be an unnecessary step for DxO at this time. i.e. it might well be a useful feature but there are other faults and features that I consider way more important aright now.

The sidecar file works just fine and when I started PL5 Beta testing I came across all the lamentations from DNG users to Adobe. They wanted the data to be in a sidecar file because the sidecars are way smaller and more “agile” than the DNG files but Adobe just ignored the requests.

Plus I can “hack” sidecar files and can’t “hack” the embedded xmp data. I can create a sidecar file for one scenario and place it with an image representing another scenario and see how the product handles it. I can hack the file timestamps and …

Again, @BHAYT , why not let the user choice?

I simply consider that to be a necessary step for DxO at this time, as we’re still at development phase.

Something (actually there are many things) we can (and do) agree on.

I feel that the “obsession” with keeping metadata out of the DOP is just that, an “obsession” and a potential diversion from the real problems.

The same goes for the issue of removing the database, there I might be biased by having a career based upon database design!? But why bother to remove it?

Your product, Photo Mechanic (not the Plus version), and Fast Raw Viewer work by discovering the data of interest from scratch every time, all the rest that I have tested use databases. That doesn’t make a database right it just means to me that DxO are certainly not wrong. Plus they open up all sorts of possibilities which the developer may or may not exploit.

DxPL’s problems are not caused, in my opinion, by having metadata in the DOP (incidentally where are they going to put the Virtual Copy metadata without the DOP) which they only use for VC’s not for the [M]aster anyway and not by having a database. Where there are actual problems it is the coding, and possibly the design which is at fault but mainly it is that DxPL simply haven’t finished what was a large undertaking.

I actually believe that so many of the criticisms of PL5 metadata handling are caused by the misunderstanding about the change in the use of the DOP, the Tag bug (still present I believe) and the fact that DAM created metadata is “bent out of shape” by PL5!

The result has been a self fuelled, self perpetuating belief that the product is unreliable, cannot be trusted, it must be the database, it must be the DOP (partly true but not for the obvious reasons), DxO should never have undertaken the work, it has become an easy target for the forums and from my perspective once I understood what happened with the DOP many of the issues simply vanished.

There are bugs but nothing on the scale that is gossiped about in the forum and few are doing anything like the amount of testing I have been doing and yet they are complaining about the state of the product - based on what, their own testing!?

So the problems are caused by lack of communication, lack of engaging with their users, lack of communication, a somewhat blinkered approach, lack of communication, bad to very bad release procedures (PL5 abandoned the sensible approach of including the release id in the file name), next to no useful patch release documentation (do you have any idea what was actually in the last two PL5 releases), i.e. communication and two way engagement with the user base, nice to see someone from DxO “breaking cover” in this topic!

So I feel that the long-standing discussions about databases, DOPs and what belongs where actually achieves little and will have no real positive impact on the product in the short or even the long-term.

Given the amount of complaints made when the PL4 DOP strategy changed and ‘Ratings’ were lost etc. many have been happily using DxPL for that job for some time. My very first post suggested that users might not want to use Metadata or might actually want to keep the metadata separate (I wondered if some might use PL5 for developing their clients photos and could use PL5 for progressing the dialogue with the customers without “tarnishing” the image metadata with that information).

With AS(OFF) the user is in complete control of what comes into PL5 and what, if anything escapes from it! Plus if you have followed my posts you will know what kind of controls I want in place to keep me and, hopefully, some of the other users happy.

I am afraid that I consider the whole set of discussions about the data and its storage an “obsession” that diverts the attention away from what needs to be done to actually make the product more productive, e.g. tracking all conflicts (not just externally generated), improving change detection and providing a compare function and providing a pass through capability.

Yes I also want my beloved ‘Merge’ capability and its young siblings ‘Add to database’ and ‘Add to image’ and way more accurate and timely documentation and communication from DxO! Have I said anything about communication…?

@Pathal and I simply think it is an unnecessary diversion.

If it was a zero effort “project” then I would be all for it but I think it would divert resources from more worthwhile changes so my vote would be ‘no’ right now, finish what is there already and add that capability to the finished product to round it off!

Dear @platypus ; @Joanna ; @Pathal ; @m-photo ; @BHAYT and others,

Thanks for your ideas shared within this thread. Even if this opens discussions and ideas for future decisions, as described by @CaptainPO , DxO as for now will stay on the current position of respecting the original RAW file without writing datas in it.

The idea of reading the MDs from the original file, even though we won’t be writing in them was heard nonetheless and we’ll continue discussion with our teams to see if and how it could fit in future developments.

This topic will be closed soon to focus on other more important feedback and discussions.
Thanks again for your participation.
Best regards,

1 Like

Thanks for hearing us…and please provide some choice to improve the “choix de vivre” :innocent:

2 Likes

As @platypus says “thanks for hearing us.” All we wish to do is make photolab the best of the best. For me and a lot of other uses. Please bring the Mac and Windows version closer together. And for me, the Windows version. Please retain the advance history between sessions. Otherwise it is not history.

Regards to all.

To summarise, if the user only uses the database, then this has priority.
If he uses *.dop files, it must save ALL the changes made and has priority.
If he uses *.XMP files, it must save the IPTC, MD and keyword… and is the priority.
*.DOP file only concerns changes specific to PL and *.XMP which may be modified by other software

1 Like

@Franky Could not agree more :+1:t2:

And here is the problem.

If it weren’t for the need to record, store and transmit metadata, the DOP file would be sufficient, as it has been before DxO decided to enter the DAM market.

If it weren’t for the need to record, store and recall image changes, the XMP file would be sufficient for handling metadata.

But we now have an app that people want to use for both purposes and DxO have decided to use the DOP file to store metadata, as well as putting it in XMP files, thus providing a need for conflict resolution.

If DOP files were used solely to record image changes and XMP files were used solely to record metadata, there would be no conflict to resolve.

Yes, I am well aware of the need to store metadata for virtual copies but, as I have mentioned previously, there is absolutely no reason why that couldn’t be put into the XMP files, as is other proprietary metadata from other software.

I understand that DxO feels the need to put more and more “stuff” into the database and, for those who do not want to use sidecars at all, that may need to remain an option for image editing changes. But for metadata, it is a whole different matter.

In order to be able to co-exist with other metadata software, metadata has to be stored in one of two places

  • for JPEG, TIFF and DNG files, it can be stored in the image files themselves
  • for RAW files, it is usually stored in XMP sidecar files.

Either way, the only reason I can see for storing metadata in the database as well as in XMP files is to be able to index it and thus facilitate searching. But it comes at the price of having to institute conflict resolution procedures, which can get complicated - and contentious if this thread is anything to go by.

As for storing it in DOP files as well, this is compounding the need for good conflict resolution by adding a third location to resolve conflicts with.

Let me take my desire to see writing metadata to RAW files out of the mix as it is obvious that DxO has no intention to go there.

I personally intend to never touch metadata via PL (apart from beta testing), due to this unwillingness; and because I have my own app, which handles writing metadata to RAW files perfectly. It is also obvious that other contributors are of a similar mind, in that they are confining their metadata handling solely to other, far more complete and competent, external DAMs.

Anyhow, it seems that this particular public discussion is soon to be curtailed - but at least we have had an acknowledgement that DxO have heard our concerns. It just remains to be seen how much influence our opinions carry.

After 39 posts I just know that “metadata” is a complete mess, not only in DxO PL but in general it’s a bad idea to trust a “standard” which turns out to be only valid for parts of certain apps which give hilarious examples for incompatibility. DxO with it’s lack of colour labels, with it’s messy mixture of DOP, XMP, and a database which apparently is only good enough to be deleted a.s.a.p. is a very good example how messy things get when a company wants to include all needs of users but already fails in developing a clear and communicated concept. I’m put off that rubbish.

Dear @CaptainPO you are totally right.
I made here a genuine and humble remark, nothing more, nothing less :man_bowing:t4:

Sounds good.
A great DAM should be able to do this.
The one I use does this. It is able to read MD from the original files but it will not write back to the original files (except .dng and .jpg).

Anyway, I know DxO is very busy and we are here to test, share and comment.
Personally I am fine using an external tool today for my workflow.
The perfect software to make everybody happy at the same time is yet to be created on Earth :joy: