PL5.2.0.4730 Conflict Resolution only detects externally generated conflicts

I was responding to @John7’s post at 5.2 What on earth to do with Metadata Conflict Resolution about conflict resolution when I set data in PL5 and no conflict was flagged [AS(OFF)].

Please note that I am a user, not a developer! My only access to the product is by creating scenarios and testing to see if I can empirically determine how the product works, i.e. I have no access to the code, or design documents and what is written here has either been determined by running such tests and/or by looking for solutions to problems reported by other or postulated by me.

This is another long read and the chances that any of what I suggest will be implemented is unlikely, but if you ask you probably won’t get but if you don’t ask you definitely won’t get!

So to re-iterate what I wrote to @John7

I have given an example using Photo Supreme Lite below in the hope that is might make the process a little less opaque! I “imported” the images into PL5 with no ‘Rating’ assigned and assigned a ‘Rating’ to the first photo in PS and that was detected by PL5 and the ‘S’ icon set on the thumbnail.

But then I started to write

OOPS! @sgospodarenko

I then set a rating in PL5 but there is no change in the thumbnail to indicate that the PL5 “copy” of the image metadata now differs from that held by the image! I am afraid that (now) I consider that to be wrong, when there is a mismatch detected by PL5 or created by PL5 then the ‘S’ icon should be set! Without such an indication it is not obvious which images have had their metadata changed in PL5 and should be flushed back to the image (if/when it is the user’s intention to keep the metadata in step).

In addition to flagging that the metadata is ‘out of sync’ (‘Requires Synchronising’) because there is a change in metadata externally or a change in metadata internally PL5 could/should indicate in which direction the mismatch exists.

There is potentially a situation where there may be multiple changes made externally and also made in PL5 and flagging up that situation would at least indicate to the user that a simple ‘Read metadata’ or ‘Write metadata’ command is going to potentially destroy data on one side or the other and make for an “interesting” display with icons indicating that synchronization is required in both directions!

Can I make my plea for a “merge” option yet again at this point!?

In the past I have suggested that a metadata “merge” is simply using existing code used for ‘Always Sync’, however, I finally woke from my “dream” and realised that with AS(ON) what was actually happening was that ‘Read from disk’ and ‘Write to disk’ were being executed sequentially, i.e. metadata read from image, followed by metadata written from PL5, followed by … etc. etc., which gives the impression of the metadata being “merged”.

This is “simply using the existing ‘Read from’ and ‘Write to’ ‘Files’/’Metadata’ commands which is not the same as taking the “excess” of metadata in PL5 and the “excess” of metadata in the image which then need to be amalgamated (and duplicates removed) to create a merged situation where no metadata is lost from either side!

Complexities exist with respect to “uni-valued” data such as ‘Rating’, ‘Rotation’, GPS etc. as to which data actually takes precedence over the other which is why AS(ON) is “arguably” the better strategy if the user wants to be using the metadata settings facilities in PL5. However, In spite of this complexity I still think it is an option well worth having!

Please note that the potential complexity of changes on both sides, in PL5 and in the image, will only occur iff (if and only if) this type of situation can actually occur with a users workflow, simpler or more controlled workflows will typically cause mismatches in one direction only (he says hopefully!) and can be ignored if the user chooses not to keep the metadata in step!

So I feel that the ‘Conflict Resolution’ in PL5 is not yet complete and that some or all of the following are, in my opinion, needed to finish this feature;

1. Flag All Metadata “Conflict” situations:-

Add a ‘Sync required’ icon when PL5 metadata changes result in a metadata mismatch with AS(OFF)

2. Flag the Direction of the Metadata Mismatch:-

Additions to ‘Sync required’ icons that indicate in which direction metadata needs to “flow” to resolve the ‘out of sync’ situation, i.e. a more stylish equivalent to ‘>’, ‘<’, ‘<>’.

3. Add a ‘Merge’ function:-

Add a ‘Merge’ function to the ‘Files’\’Metadata’ options that avoids the need to keep AS(ON) but results in the same (most of the time) equivalent to that feature and ensures that neither set of metadata is overwritten and lost.

With this function metadata is taken from the image and from PL5, combined, de-duplicated and written back to the both PL5 and the image.

The complication of “uni-valued” data such as ‘Rating’, ‘Rotation’, GPS etc needs to be addressed and probably the only way to handle that is via Timestamps. This still poses a problem because each metadata element may have been changed at different times and typically only a single timestamp will exist for all PL5 metadata and another for the image metadata!

4. Add a Metadata ‘Compare’ function:-

Add a ‘Metadata Compare’ function that can exercised at any time to detect when the ‘S’ situation has occurred but not detected and which results in both the ‘S’ icon being set and the direction that metadata must flow to resolve the situation.

5. Add ‘Preferences’ option equivalents to the DOP ‘Load’ and ‘Save’ options for ‘Metadata’:-

PL5 provides the ability to handle DOP data by selecting ‘Save settings in sidecar file (.dop) automatically’ and ‘Load settings from sidecar file (.dop) automatically’ in the ‘Preferences’.

Could this type of feature be added to PL5 to supply the same commands for handling metadata, i.e. ‘Read Metadata from Image’ and ‘Write Metadata to Image?

Arguably setting both of these new options equates to the current ‘Always Synchronize’ option.

However, some users want the metadata set (and updated) by their DAM to be the only source of data and would be able to use the ‘Auto Read Metadata option’ useful to keep the PL5 metadata inline with the DAM output to the image files. This function could have a ‘warning’ attached when the first such situation is detected in a session with an ‘All’ option to accept for the session etc. to prevent potential “corruption” to the PL5 data!?

6. Add an ‘Add from’ and ‘Add to’ feature to the current ‘Read from’ and ‘Write to’ ‘Metadata’ Commands:-

The 'merge function that I described above is effectively a combination of an ‘Add to’ command, i.e. ‘Add to database (from image)’ and ‘Add to image (from database)’ and potentially useful additions to the current ‘Read from’, and ‘Write to’ and (of course) the suggested the ‘Merge’ functions.

Hence, the ‘Files’/’Metadata’ commands would become

  1. Read from Image
  2. Write to Image
  3. Add from Image
  4. Add to Image
  5. Merge
  6. Compare

Would it then also be useful to add the ‘Add to’ commands to the automatic synchronisation options, i.e. item 5 above describes automatic ‘Read from’ and ‘Write to’, both potentially destructive operations where the ‘Add to’ is non-destructive. This would enable users to have the external metadata contents automatically added to or overwrite the PL6 database automatically with no additional intervention required.

The current ‘Always Synchronize’ ‘Preferences’ ‘Metadata Synchronization’ option would effectively become

Read from Image Automatically Add from Image Automatically
Write to Image Automatically Add to Image Automatically

Selecting ‘Read from Automatically’ and ‘Write to Automatically’ will provide the current ‘Always Synchronize’ option but using the ‘Add from Automatically’ and ‘Add to Automatically’ will provide a more secure option with respect to the preservation of metadata.

The caveat about uni-valued metadata still applies, ‘Rating’ etc. are not “additive” items whereas keywords arguably are!

7. Options for Users that are not interested in PL5 handling metadata:-

Are there users who would rather not know about metadata mismatches at all and would like an option that simply ignores what is going on, i.e. not only turn off all the read and/or write metadata options but also the detection and display of the ‘S’ icon?

Whether the ‘Compare’ function at item 4 above (which currently does not exist) should still be available if the user selects to suppress the display of the ‘S’ icon etc. is open to debate?!

8. PL5 versus DAM users:-

Many users of DxPL use the product for its RAW photo editing capabilities rather than any DAM functions.

They typically want the metadata set by their DAM to be taken into DxPL and then transferred unchanged to the exported file. The metadata will currently be captured by PL5 when the file is first ‘discovered’ and to protect the image data from any changes that PL5 might choose to make to that data AS(OFF) will have been selected (i.e. there will have been no selection made for the ‘Always Synchronize’ option).

However, if that image metadata is subsequently changed via the DAM any such changes will not be reflected in the PL5 database and not be included in any ‘Exports’.

Currently a one way import can only be accomplished by manually executing the ‘Metadata’/‘Read from image’ command on selected images.

If either the suggestions at 5 or (better in my opinion) 6 are implemented then there can be an automatic ‘Read of metadata’ from the Image whenever the DAM makes metadata changes to the Image (provided PL5 detects them).

The ‘Compare’ option suggested at 4 above would allow the user to check that the PL5 metadata is up to date before making an ‘Export’ and make a manual ‘Read from’ in the event that something untoward has occurred!

That should keep even the pickiest of users (me) happy!

9. Automatic Metadata Pass through:-

That leaves the thorny subject of PL5 applying its own ‘Reformatting’ rules to the metadata. This has gained more than a few column inches in the Forum from irate users who do not want the keywords that have come from their DAM re-structured in any way.

This effectively requires an option to control whether PL5 applies its inbuilt rules when formatting keywords either to write back to the image, if allowed or blocked by the judicious use of the options currently in place and/or those suggested above (if any are implemented), and when exporting the image.

If the suggestions above were adopted then automatic one way transfer would be possible but PL5 will still apply its reformatting rules as currently implemented, particularly when exporting the image.

A Global ‘No keyword re-structuring’ Option:-

So there needs to be an option somewhere that allows the user to override/inhibit the current PL5 (re-)structuring rules for the keyword metadata. Other packages do include options in the preferences to control how the package handles the metadata in line with the user’s wishes.

The obvious locations for such an option are ‘Globally’ or on ‘Export’ or both!?

A global ‘Keep Image Metadata format’ option would ensure that no restructuring of metadata takes place, what is currently in the ‘dc’ and ‘hr’ fields remains exactly as they are on import and (by implication) on export or on export via an explicit option

That still poses a problem if any metadata is added in PL5 to the imported metadata and what rules should be applied to metadata created in PL5. For those users that have made complaints about the re-formatting it is almost certain that no metadata will be added via PL5 and, if coupled with the automatic ‘Read from Image’ global option suggested in item 5 (and item 6) then any changes made externally via a DAM will be captured and updated and the output format will remain.

Control what metadata is read from an image:-

The metadata is currently read from the image when the image is first “discovered” by PL5 using the ‘Read from Image’ or equivalent code (a guess). Should that function be controlled or inhibited? The problem with either case is that the data will not be available for output unless…

Export of Metadata Direct from the Image (‘Include from’ option added to the Export Options):-

One additional feature that could be considered is that NO Metadata is actually taken into PL5 or it is taken in but simply ignored for export when a particular export option is selected.

This export option would identify the metadata source, i.e. an ‘Include from’ option which indicates that the metadata should be taken from PL5 (the default) or from the original image or a template or ….

This option would precede the current ‘Include’ option so that the items to be included are not controlled by the ‘Include from’ option but are applied regardless of the data source.

An actual Image protection Option:-

Effectively, if users wanted the metadata inserted by a DAM to be fully protected (from DxPL) there could be an ‘Image Write protection’ feature somewhere to prevent DxPL from writing metadata back to the photo as well as a metadata protection feature to ensure that no changes are made to the exported photo meta data “layout” as mentioned above.

3 Likes

YES, YES, YES! Please DXO.

It seems to this non-programmer that it should be relatively easy to implement the above.
I think it’s going to be a long while until DXO PL has a fully developed and SAFE DAM. In the meantime, just as now, there will be a lot of problems, frustrated users, and most likely users giving up on DXO. If this option is added much of this can be avoided during the interim period.

1 Like

@BHAYT

Bryan Thomas, I must say this is very very well written. Since I use Photo Mechanic as an Image Library and metadata editor I am afraid that DXO has changed things that worked before.

Since I don´t use “structured keywords” in order to keep it as simple as possible everything has so far worked beautifully for me. If I update XMP-metadata in Photo Mechanic and then just open that RAW-file in Photolab without tampering with the metadata at all in Photolab 5 all the metadata in the RAW has got transferred to JPEG-, TIFF- or DNG-files without any losses. It has looked fine when I have checked my JPEG-derivates in PM Plus. So that circular flow has been perfect so far for me.

I really hope they haven´t screwed up anything in this workflow that has worked without any extra work needed when they now starts to check for conflicts. That would decrease efficiency substantially. What I want is that they give us a switch that defines my external datasource as the owner of the metadata and another that defines Photolab as the owner if that would be the case. If I choose to use an external metadata owner I also think that they should disable the possibilities to updata the metadata in Photolab.

I don´t believe in biderectional flows at all because that is to ask for problems and it increases the complexibility tremendously. PM Plus is for example forking data depending on how it is configured and pushing the same data the other way might not respect that, which definitely will create discrepancies.

I guess I have to start testing these tools too ASAP.

Thanks for your great empiric efforts and your thoughts on this very complex subject. I learned a few things too :slight_smile: I hope DXO listens to these discussions too. They also has to solve the issues with metadata migration between Photolab and other platforms using structured keywords. There has to be possibilities to import vocabularies and to export and import them as for example TAB-separated text in order to make maintenance of structured keywords more efficient for the people who needs that.

2 Likes

@nwboater Thank you for your response.

The feature would provide the “firewall” some feel is required but I believe that turning the ‘Preferences’/‘Metadata Synchronize’/‘Always Sync’ OFF (the box unchecked) means that PL5 will not automatically import and export (Read and Write) any keywords except on the initial discovery of the image file when the metadata found in the image will be imported.

With the new ‘Conflict Resolution’ mode then any changes made to the image metadata should be detected and flagged with the ‘S’ icon. At that point the user can user the ‘File’/‘Metadata’/‘Read from Image’ command to refresh the PL5 metadata, if desired or simply ignore the flag. However, that command overwrites any PL5 data, hence, the other options I am asking for in the post.

I did not prioritise the items that I included in the post but believe that they should be

  1. Fix the lack of ‘S’ flagging when changes have been made to PL5 metadata in an AS(OFF) situation. This is not particularly urgent for those using DAM software because they will not be making changes in PL5 but I consider the current lack of an icon a “bug”.

  2. When I tested this new feature earlier in the year PL5 kept up with 889 settings done with a PM batch job with no problem but I had a couple of changes missed by PL5 when testing with all the image thumbnails on the screen!? This was not something that I could repeat in tests which then leaves me a “bit wary” so my next most important item would be the ‘Compare’ facility. It would help ensure that no DAM changes could slip paste PL5.

  3. The ability to control what PL5 does to the keywords to ensure that what is put there by a DAM stays exactly as the user wants (for those users who want that feature) would help make a number of PL5 users way more happy about PL5 keywording.

I believe that many of the horror stories about PL5 have stemmed from the unfortunate misunderstanding over the role of the DOP in PL5 and an expectation that ‘Rating’ and ‘Rotation’ set in PL5 (possibly after being successfully imported from PL4 via a PL4 DOP) would be preserved by just storing the image and the PL5 DOP without writing the PL5 metadata back to the image or xmp sidecar file and preserving all of the parts of the puzzle. These parts for a RAW file are the image, the DOP and the xmp sidecar file not just the DOP and the image file as was the case pre-PL5.

The other and perhaps the biggest complaint was from those who use a DAM to set the keywords only to have PL5 set about that data re-arranging it based upon the PL5 “standards” for handling the data.

My lengthy post and wish list was born out of trying to help users who lost settings because of the DOP misunderstanding and help the acceptance of the product by sceptical users.

I use a number of other packages when testing PL5 and much prefer the PL5 interface for ease of use. I currently use IMatch and have a copy of Photo Supreme Lite which are rammed with features but way harder to wrap my old brain around than PL5 and more cumbersome (for me) to use when I just want to set up some test data in a hurry!

1 Like

Well, your “old brain” appears to do way better at getting into the minutia of this stuff that my 80 year old brain! :slight_smile: And I really appreciate yours and others efforts to help DXO make a more usable PL.

As you can probably tell I’m in the camp of telling PL to leave my metadata alone. I have been using Photo Supreme and probably will switch to IMatch (although I am a little concerned about the learning curve).

I want continue using a separate DAM for a variety of reasons:

  1. I believe it’s a huge project to develop a rock solid and fully featured DAM. This can take a long time to get right for a company that does nothing else, so for DXO I fear it’s a long way off.
  2. I may change editors in the future.
  3. I do a lot of jpg editing outside of PL. If I have to bring those files from the other editors into PL for DAM function I may as well use a single real DAM for everything.

So again, I really hope that DXO can soon implement a simple selectable setting to allow it to not touch my metadata. PL is a great RAW editor and I would like to be able to use it as ONLY that.

2 Likes

@nwboater Yes I had guessed as much! You are a little older than me, about 6 years! I feel that there is a big learning curve with IMatch although there are a number of users of that product on the forum @jch2103 for one who may be able to help.

@jch2103 has been fairly vocal on the forum with respect to PL5 “playing” with the keyword format output from IMatch and would also welcome a source image protection facility, I suspect. However, the latest release of PL5.2.0 has made changes which might be more acceptable to users who have complained. The complaint is as much about what happens with the exported JPGs (or TIFFs or DNG files) as it is about PL5 damaging the source metadata!

I was going to post the following XMP-files gets F*cked up due hierachical mismanagement in dual management - #39 by BHAYT here since it closely related but finally added it to the the post indicated which helps identify the changes that have been made.

I purchased IMatch when it was on offer at Christmas and it is a very well featured product but with a steep learning curve for someone who has not used a full blown DAM before, i.e. me!

If you can send me your normal Photo Supreme preferences/settings and the types of keywords and other metadata that you use I can try to configure Photo Supreme to replicate the outputs that you expect to get.

Once we have a baseline I can then attempt to reproduce those in IMatch and also subject both to PL5 to see what might be happening with any and all of the data that you use!

In the meantime I would welcome some advice with respect to Photo Supreme. I want to be able to add animal|mammal|bear for example without PS adding ‘Miscellaneous’ (a category I believe) to the keyword! What setting stops the automatic assignment of a category so that I get the keyword I entered with no additions?

Brian, thanks very much for your kind offer to help with my IMatch setup. My present Photo Supreme configuration is very simple and I’m using it at a Flat Database (non-hierarchal). Since I planned from the time I purchased it to later switch to Imatch I have spent little time doing much with it other than entering keywords. At that time my PC was too old to run IMatch. I have just completed building a good photo editing PC (Ryzen 5900x & soon a 3070TI GPU).

At this point I have installed PL5 but have not put any of my RAW photos on the PC. There is another issue of concern and I made a post about it https://feedback.dxo.com/t/pl5-moving-files-outside-pl-loses-corrections-and-more/25664/23 Once I get an answer from my DXO Support Request and can feel more comfortable with the issues in both threads I will hopefully be able to put my RAW’s on the new PC.

As you can tell I’m a ways off from getting involved in any database work because I need to first get the PL5 issues settled. Once that happens I will probably buy Imatch and start setting it up as best I see fil which will probably be more complex than my PS presently is. So thanks again for your offer of help. I’m actually still struggling over the use of Hierarchal keywording or not.

I did check the Photo Supreme Quick Start re your concern of it adding the ‘Miscellaneous’ Category. It seems that it does that because it wants you to always have a ‘Category’. If you don’t enter one with your new Keyword it puts the ‘Miscellaneous’ one there and suggests you later assign the proper one. Thus I doubt that you will be able to set PS to ignore Category when you enter a new Keyword.

From Photo Supreme Quick Start:
“Miscellaneous During import, pre-existing keywords will be read, and Catalog Labels are
created for every keyword. When a keyword can’t be related to an
existing top-level Category, for instance if there is no hierarchical
keyword information, then the keyword will be imported as a Catalog
Label in this Miscellaneous Category. After importing you should then
cleanup this Category by dragging the labels to a better matching
Category.”

Rod

Bryan, you are making this far too complicated. Properly handled, there is no reason to “protect metadata from PhotoLab”.

To keep metadata handling simple, there’s a one path: a single canonical source of data. In this case the XMP file.

If a PhotoLab user changes any metadata, it should be reflected immediately in the XMP file. Any changes to the XMP file should be reflected immediately in PhotoLab. In this case, even with two applications changing metadata more or less at the same time (not recommended to push one’s luck, there can be OS level file control issues though sharing a txt file has long worked reliably across well-written macOS apps like BBEdit), the metadata will stay in sync. Why? Because there is a single canonical source. This is the way to avoid all the issues with sync.¹

What I could understand is a single preference to disable active handling of metadata application-wide. Your menu of write/read instructions for metadata is so complex that no one could use it reliably or the programmers maintain it. I know PhotoMechanic does have such tools but PhotoMechanic only does metadata and has been doing it for twenty plus years. Trying to add these kind of metadata manipulation tools into PhotoLab is like turning a convertible sports car into a dump truck or a bulldozer.

I would not like to see 80% of DxO’s programming team (relatively futilely) working on two-way sync for the next five years.


Notes

1. You may or may not be aware of it, but two-way sync when added to an application often requires double the programming time just for sync than the rest of the application requires. Even then reliability is spotty, particularly across OS versions. Reliable sync was the genius of Dropbox but sync got away from even Dropbox whose raison d’être is sync. Total nightmare in terms of managing an IT project. Avoid two-way sync at all costs.

1 Like

While I think I understand what you say I don’t think I agree with the “logical” conclusion that you arrive at!?

EDIT:1
Summary:-

To achieve a “single canonical source” source you would not only need to have a single source of data but also a single source of updates to ensure uniform compliance from all update “agents”.

In the case of DxPL to obtain that single source you would have to remove large quantities of code, much of which has been there for some time.

That would hardly be a productive endeavour and would consume more of the precious resources than it would ever save!

EDIT End (including removing some extraneous elements)

Firstly the fact that PL5 possesses a keyword structure in the database is arguably a “legacy” from the days when the “immutability” constraint did just that, it constrained DXPL from accessing and updating the xmp data directly and the keywording structure in PL5 was essentially a closed system with an input flow from the external image metadata.

Whenever DxPL detected a change in the image it would read and decode the data and automatically overwrite the database fields with the new settings.

As for preventing DxO expending quantities of resource on this area, that “horse has already bolted” but in fact the basic metadata handling code has been in the product for as long as keywords have existed in DxPL!!

Removing all the code to handle the database might make DxPL appear to be a “purer” product but the structures are not a major part of the problem! They have existed for a long time, as has the code to handle them and most other products also have their own database!

I would rather turn the metadata handling into something every user can use or at least no-one “fears”, than ignore the flaws and leave in its current state and removing the database will help no-one, in any practical sense!

When a change is made to the metadata in PL5 it will be made to the DXPL database and then the changes made to the external metadata depending on the AS(ON) or AS(OFF) parameters. All this code already exists and mostly works, I say mostly because I believe that during my tests DxPL failed to notice 3 updates to the extent that the records were not even in the database.

With AS(ON) DxPL automatically makes the changes that you are suggesting, it just happens to have a local copy of the data at the same time! Incidentally I have conducted tests with multiple bits of software all having access to a photo at the same time and rarely noticed any hiccups, i.e. the various product programmers know how to get in and out of the file quickly (as they should) and/or programmed the retry strategy!

Even if all software providers agreed to only ever store in the image metadata and not “risk” the issues that can occur with replicake, sorry replicate, data we now have the issue of the standards for populating the fields in that one copy of the data on offer.

So I believe that my “over-elaborate” scheme to prevent PL5 from “changing” the data is not replaced by having a single source of that data unless there is also single set of rules applied (and enforced) for processing that data!

Hence, you not only need a single source of the data but you also need a single gateway to that data through which all programs must pass, which exerts a single unified set of rules to the metadata requests and updates, which is never going to happen!

A simple example of the problem I was trying to address is if I set keywords of “animal”, “mammal” and “bear” in ACDSee then those keywords will occupy three ‘dc’ keywords and any data that finds its way into the ‘hr’ keywords will be ignored by ACDSee and Zoner, and ExifPro etc. @joanna would suggest that the ‘hr’ fields have no right to be there in this case!

Therein lies a problem;

  1. PL5 is “obsessed” with the ‘hr’ keywords. @joanna complains about this and I started complaining early in PL5 beta testing even though I was only learning about metadata at the time!

  2. PL5 applies its own “standards” to the metadata, in the case of the simple example I used above it will create ‘hr’ entries for the 3 keywords and these will find their way into any exports and, potentially, back into the image metadata!

Many users do not want this kind of “corruption” of the data that has been put into the image by their DAM, hence the suggestions that I have made:-

  1. A global “no-go” parameter to stop PL5 “corrupting” the image metadata.

  2. A global “no-go” and/or export parameter to control what finds it’s way into the exports.

I really don’t believe that anything I am suggesting is particularly difficult to develop, all the heavyweight coding is already in place, effectively the options I am suggesting (in this particular case) simplifies the update process, it is a straight mapping from the input metadata to the database and beyond (to the exports).

There are complications however and these come if the user is allowed to make changes to the metadata in PL5! How should this be managed, should this even be allowed?

With respect to the note, the code is there already mostly working correctly!

My system is simple, transparent and works. Your system is complex and will fail repeatedly, even when years have been spent developing it.

There’s good reasons that old hands at maintaining data integrity like @Stenis and I (and many others) insist on maintaining the primary source of data 1. in a flat portable file which travels with the image and 2. preferably in a standards compliant format.

Standard were created for a reason. If you would like to interact with a program which does some of what you demand from DxO, there is PhotoMechanic. Metadata management/curation/translation is all PhotoMechanic does, it does it well, it’s in development for twenty+ years and it’s full of bugs and exceptions like any program of this kind (check the release logs).

If you would like PhotoLab to be the auditor of metadata (throwing away what PhotoLab doesn’t like) and then the hoarder of the metadata (not sharing that data in a standards compliant and portable way by default), then what you are trying to build is a dysfunctional prison for your/our images. From which all the data will eventually be lost and/or corrupted.

No thank you.

But I don’t want to have to work both sides of the street, I want to work just one and get the basic metadata handling I need at the same time as a RAW developer!

Most of the doubters wrote off DxO attempting to do what they have done before they started!

Had DxO consulted with the users, listened more and avoided the fiasco of the DOP change by actually producing some timely documentation, there would be even less credibility in your statement.

Yes there is more than a grain of truth in what you say but some of the errors are potentially in one line or a very small sections of code.

To provide what you want, the particular feature that I have suggested is still required, i.e. a straight pass through of metadata from Photo Mechanics or the utility or DAM of a user’s choice.

Then DxPL can do what they want, I might get a lightweight metadata editor and RAW developer in one and you can continue to use Photo Mechanics or any other product you desire in the knowledge that the metadata so carefully curated by your chosen software is safe and will make it into any exports from DxO in one (unsullied) piece

The only worry that leaves is whether DxO development resources are spread too thin.

But DxO is not breaking standards much more than a number of other products and those aberrations would not take long to fix, if DxO can be convinced that they need fixing!

Elsewhere in the topics we have SPOD cropping up again, I don’t see SPOD I see SPOF (Single Point of Failure), I see “all my eggs in one basket”.

I would prefer DxO and other software to have a database that they can practice on and when they have got it right (and I have convinced myself that they have got it right) I might let them loose on my metadata!

1 Like

The first problem with DXO and Photolab when it comes to metadata is the legacy and that they couldn’t resist copying the Lightroom metafor. Now they have a mess with metadata in several diffferent places instead of one. The second problem is that they have opened up for a two way flow like Capture One but are not at all as sofisticated as CO yet. The third problem has been that they have locked their users in because they have not yet all features in place to make ut possible to migrate to and from PL in a controlled manner.

There is a reason that all serious DAM systems use XMP and only XMP. There is also a reason why serious integrators avoid bidirectional data flows as Alec just have stated. To provide a tool for conflict resolution to solve these problems is a joke. This might make a few hobbyists happy but is nothing to have for people looking for efficiency in their workflows.

1 Like

@stenis Thank you for your input but I am not sure I understand much of it! While I am no metadata expert I spent a long time in the IT industry and well understand files and databases but do not really understand your comments.

I understand the gist of what you and many others are saying and have been saying from before DxO embarked on the work to update DxPL’s metadata handling capabilities. I am not sure that I have seen more negativity in a long, long time. I do understand the concern that a well respected RAW Developer product was not going to make the advances desired on the photo editing side because resources were being wasted on a “vanity” project, i.e. the metadata handling.

So can I ask for a longer post from you to explain what you have written!

Given that DxO was built to interact with Lightroom is that so surprising? But more importantly why is “borrowing” from Lightroom actually a bad idea, there are only so many ways that you can “skin” the metadata handling “cat”?

Not really, except for the importation of some of of the metadata from where PL4 used to maintain it, i.e. the DOP, PL5 gets the metadata from the image. Yes it stores its own copy in a database, like most of the other products (albeit ones with DAM “pretensions” or DAM “credentials”) and continues with the portability “paradigm” by keeping a copy in the DOP, but that is principally for storing the metadata for Virtual Copies (plus the editing data).

I am not sure how many users will actually want to assign independent metadata values to different Virtual Copies, except perhaps a quick description of the edit, but it was a simple extension to include keywords in the DOP!

The “sniping” at the database should be aimed at Capture 1, IMatch, Photo Supreme, DigiKam, Photo Mechanic Plus, XnView in the image editing/image management arena and countless more in the
field of general cataloguing.

The DOP is a DxPL unique I believe and “belongs” to DxO and they can do exactly what they want with that “metadata” store as they please and as long as they do it well I really don’t care, the key (to me) is the quality of the care they take over that particular item that makes it a valuable resource or a liability.

Is it really not getting close to the sophistication of Capture 1 yet. I don’t see the manual ‘Sync’ command in DxPL but when I read up about that in C1 I am concerned that they both have flaws, nor necessarily the same ones but neither might be perfect in to my mind!?

More explanation please!

DxPL uses xmp, is this another concern over the DOP? The main mechanism for DxPL communicating to other software is xmp, embedded for JPG, TIFF and TNG and sidecar only for RAW. The DOP storage is a convenience for DxPL with respect to Virtual Copies and their metadata. Please stop using its existence as some excuse to dismiss DxPL metadata handling!

By all means use errors in handling of xmp, bugs or lack of facilities but “sniping” at the DOP or the database is just a “cheap shot” to discredit the DxPL metadata handling!

Turn off ‘Auto Sync’ (C1 appears to have that as well) and you have a a uni-directional system, it takes seconds to do and doesn’t require a restart like C1 and you can happily use PL5 to ingest your metadata.

However, at that point the system becomes entirely manual and I would like DxO to add the automatic ‘Sync’ ‘Load’ facility of C1 to DxPL and one of my posts suggests a change to the shape of the single AS(ON) or AS(OFF) option into a series of automatic options that includes my own favourites of ‘Add to image’, ‘Add to database’ and ‘Merge’ plus the obvious ‘Read from’ and ‘Write to’. If DxO added those features it would raise DxPL above even C1 for metadata handling.

It then needs to add multiple database management etc. etc. but …

However, the area where I sympathise with comments that have been made is DxPL putting its own “stamp” on the ingested metadata. I understand why others have complained and there should be a straight through path to leave the metadata exactly as it was and then carry that through to any exports!

Then that makes me a “hobbyist”.

Without an ‘Auto Sync’ ‘Load’ command it becomes necessary to alert the user that data has changed. They may well know that it has changed because they are almost certainly the person that changed it! However, it will become less useful when DxO add the ‘Auto Sync’ ‘Load’ command to the preferences.

This topic was started by me because I also want that icon added to changes made in DxPL, does that make my “hobbyist” credentials even more obvious. I don’t want to have to remember what I have changed externally or internally, that is why I use a computer, it is after all “infallible” (when coded correctly) I am not!

Please stop sniping, start testing, make constructive comments, preferably not one’s like “scrap the database” or “change the DOP” because to do that would change the direction of the product and simply waste time.

I am as “cynical” and as “sceptical” as “doubting Thomas” himself. There are things wrong will DxPL metadata handling but so many of the issues raised are “philosophical” points where the product fails to meet some “ideal”!

I worked in I.T. in one way or another from touching my first computer in 1964 (buttoning in a program via an ICT 1301 console) until I retired in 2009 (after being the project ‘Systems Architect’ for the manufacturer who supplied the Orange Voice Mail system, a project I worked on for the last 16 years of my career) and wrote or pseudo-coded only about 750,000 lines of code albeit COBOL while working on multi-million pound 24/7 real-time, on-line database systems.

Does that make me a “hobbyist”!?

I will admit that photography metadata is relatively new to me but fixing the ills of DxPL will only come from constructive criticism and careful testing and reporting, not some of what I read in the forums.

Thank you if you read this far.

I’m sorry Bryan but @Stenis is definitely right. DxO stores the same metadata in…

  1. the DOP
  2. the database
  3. an XMP sidecar.

As I have explained elsewhere, there is absolutely no reason why metadata for virtual copies cannot be stored in a custom section of the XMP sidecar for the original file.

The only reason for storing it in the database as well has to be for reasons of caching and efficiency of searching.

I didn’t read it as sniping, possibly just difficulty in translating out of @Stenis native language

Personally, I don’t feel getting into the heartache that is conflict resolution is going to do anything other than make the user experience any more enjoyable for the majority of users - especially given that interactive DAM functionality isn’t something that most people seem interested in.

My feeling is that folks who started out with Lightroom and other DAM solutions are hardly going to be swayed by the lack of interaction in PL with their favourite DAM. Most seem to have resigned themselves to ignoring PL’s offering as to do otherwise just means pain - I am certainly staying well clear of PL’s DAM (apart from beta testing), much preferring my own software, which has a much easier to use UI.

I get the feeling that some folks have tried constructive criticism but, seeing that there was no response from DxO, have now got to the stage of, understandably, “venting” their frustration.

@joanna you know that I know that PL5 uses a database, a DOP and the xmp data (embedded or sidecar) my point is that I consider most of the discussions about how many copies it cares to keep are academic (and I was one once!), its principal source is the image data. The discussions over the niceties or otherwise of multiple copies is also academic.

What is in place right now is potentially good (notwithstanding the total lack of clarity with respect to the implications of the DOP usage changes and the potential impact on on workflows and expectations)! I believe that the drains-up commenting is totally pointless and counter-productive.

The bugs that remain and the additional features that should be present are not rocket science and would make a perfectly usable product if added. Instead we get endless debates about whether DxO have done a good enough job, why do we have DOPs, why don’t the DOPs do what they used to, who needs a database anyway, why is the DxPL database not better, why…

The database is the same that it has always been, the DOP hasn’t changed except to take keywords and GPS onboard and that isn’t used as the primary source of metadata anyway (which is often the main reason for all the disquiet).

Basically DxO made one simple mistake, they didn’t spell out in words 10ft. (reverting to Imperial oops) high what the implications of the changes were and “geniuses” like you and I didn’t spot what had gone “wrong” until too late so that all the old complainers could wade in with all the old “tropes” and do a thorough hatchet job on the product.

You know that I have a high regard for your knowledge and the way you try to help DxO and me and anyone that asks but you also know that I don’t agree with you that the database and the DOP are a real issue.

Replicate data is only really an issue when multiple coders are let loose with multiple programs on multiple structures and I’ve been there and got the tee shirt and wear it for DIY when the weather is right!. The database and DOP are under the control of one group of coders and handling those two structures is genuinely not a hard task.

Many folks tried to block DxO going down the path and are now saying “I told you so”, some are genuinely impressed with how far they managed to get (albeit elements of that development may have been the reason for the flaky start for PL5).

You know my views on comm… when you get any it isn’t complete and it is all driven by a marketing plan (as it should be) but one that has an element of pragmatism would be good. The danger is that DxO will try to park the metadata project at some convenient point and concentrate on wooing new purchasers and the current users to upgrade with new, improved photo development features.

I would welcome an automatic colour correction (with manual tweaks) that rivals the best that I have seen, a ‘repair’ function that can handle something more than lens or sensor spots, what it did when I tried to remove a flagpole was beyond a joke, an ability to add a preset without it resetting back to the beginning (that is beyond annoying), the ability to remove entries from the database and leave the photo, DOP and sidecar intact (that would help fix a lot of issues and simplify a host of procedures with respect to moving data) to name but a few.

However, the items I have mentioned are not headline grabbers for the next release to pull in new “punters”!

Take care!

1 Like

I never understood the sync solution betwwen Photolab and Lightroom really. Who will live with a clumsy solution like that and a very expensive one too but I guess that´s why DXO made Pure Raw as a more lightweight offer.

The metadata standard these days is XMP period. From what I remember it was an active step to turn on the XMP-export from the Lightroom database to XMP-sidecars for RAW. If you forgot that and your database got corrupt you were in real trubbel because as a default it was turned off and the database was the metadata master - a classic exampe of a single point of failure that in the early days of Lightroom even hit myself a couple of times.

If you hadn´t taken care of your backup in a correct manner then you had a severe problem - I learned that the hard way. The problem with Lightroom is and was it´s database centric design. Every DAM worth the name shall always have the XMP-metadata in RAW-sidecars or in the XMP-headers of the XMP compatible file formats like DNG, JPEG, TIFF and ever PDF-textfiles. In Enterprise DAM systems the files and only the files are the owners of the XMP-metadata, which makes it a very rubust distributed system. If an image- or sidecar file gets corrupted you loose one imagefile or the metadata to an image compared to loosing it all when a Lightroom database file gets corrupted.

If you place all your image folders of yours under a top folder to begin with a future migration to another system and in fact you will make yourself a good service for the future if you do that. I have a few catalogs/XMP databases like that in Photo Mechanic that can manage as many databases I like (Lightroom can just manage one at the time).

If I have a structure like this and that your system uses XMP as the only source of metadata it´s relatively easy to migrate from one system to another.

Topfolder
…Imagefolder 1
----------------RAW
----------------Full size JPEG
----------------4K JPEG (or say HD if you prefer that for monitors)
…Imagefolder 2
----------------RAW
----------------Full size JPEG
----------------4K JPEG

(Sorry for the dots but this editor doesn´t keep the spaces and doesn´t support Tab)

When I for example made Photolab aware of my XMP-metadata I just pointed at my topfolder where all my developed image files were stored and after a couple of hours they were all in Photolab including all my keywords (non structured of course to make it simple) from my files) that Photolab also aggregated in a sorted list ready to use in Photolab. I haven´t tried to migrate from Photolab but that will not happen really for me since Photo Mechanic is my metadata owner.

The day I want to migrate I will just take another DAM-software and index all my images and their metadata by just pointing at my topfolders. Of course I will have to design a few forms to match the XMP/IPTC elements of my data but after that I will be in the air again.

In the case one of my catalogs/XMP-databases in Photo Mechanic Plus gets corrupt there will be nothing to worry about at all since these systems like real Enterprise DAM are not subjects of the single point of failure problems that Lightroom has for example. I just reindex by pointing at the topfolder of the files for the corrupted catalog and in a couple of hours I will be able to work again without loosing any data at all.

Since the “almost” autosync of Photolab/Lightroom is Lightroom-centric we will need to remember to “Read” the metadata from the files we open manually in order to keep the database in sync. It would be very nice if Photolab always read the metadata automatically from the files in the folders it opens too.

With that said I´m not negative at all that Photolab now have got a Photo Library and I was surprised how good it really is already and how well it integrates with Photo Mechanic Plus (except when it comes to structured keyword) but I wonder what will happen in the future. I have a feeling that DXO is putting more focus now on the database because it is a very convenient way of handling things but when it comes to the metadata the database can never be allowed to be the metadata master of Photolab. If the Photo Library database will get the same metadata role as in Lightroom I think they make a big mistake.

DXO also have to make it possible to import and export keyword vocabularies manually too - both structured and flat and not just the vocabulary from Lightroom. There are lots of other vocabularies out there than the one in Lightroom. This function can also be useful for people who wants to maintain tab separated text vocabularies in a far more efficient way than the Photolabs own interface admits. When I tested to use structured keywords in Photo Mechanic Plus I used to export the tab file and edit it in a text editor instead and then import it back after maintenance. As you might understand by that is that I am not a very big fan of the maintenance interface for structured keywords in Photo Mechanic Plus :slight_smile:

1 Like

@Stenis I need to read the rest of your post more thoroughly but when PL5 encounters a photo that it has never seen before (i.e. which is not in the current database) it effectively executes the ‘Files’/‘Metadata’/‘Read from image’ command (or at least does the equivalent).

This always happens except if the image file has a PL4 DOP when things are a bit different and the DOP is used in preference for at least some of the metadata!

So just encountering an image file for the first time results in the metadata being read and added to the database and the PL5 DOP (but never read back for the [M[aster image) regardless of any settings!

Thereafter any automatic action will only occur iff the ‘Always Synchronize’ option is set. Manual action can be instigated at any time by using the ‘Read from image’ and ‘Write to image’ which can be used regardless of the AS setting, i.e. AS(ON) or AS(OFF).

If AS(OFF) is set (actually AS(ON) is not set to be more accurate) then no automatic action will take place with respect to metadata except that the ‘Conflict Resolution’ icon(‘S’) will be set when a change occurs to the metadata of the image but no change will occur even to that icon if the change is made in PL5, hence the reason for my post!

So there is an automatic process for the first discovery of an image and for every discovery that is detected with AS(ON) or only for the first discovery with AS(OFF) when any further discoveries are instigated by the manual execution of ‘Read from image’.

BUT the problem is that some users are not happy with AS(ON) because it gives PL5 permission to write the metadata back to the image whenever it changes and those changes can be as “simple” as PL5 deciding to add ‘hr’ entries to non-hierarchical ‘dc’ keywords immediately after it has read that metadata into PL5.

The AS option looks to be really useful at first sight and I certainly enjoy testing with AS(ON) but because of this “fiddling” with the metadata it would be good to have an automatic sync option that only goes one way, i.e. a ‘Read from image’ whenever a change to image metadata is detected but does not go in the opposite direction, “what happens in PL5 stays in PL5”!

Unfortunately the problem does not stop there because the “fiddling” then manifests itself in any exported image.

Hence the options I have asked for! On paper the AS option looks to be really good but it is arguably “too clever” for its own and the users good it “fails” because

  1. PL5 will insist on “fiddling” with user metadata. The latest release has attempted to reduce some of that for hierarchical keywords but more work is necessary or actually “no work” is ideal for some users, i.e. DO NOT TOUCH THE METADATA!

  2. I have detected instances where PL5 has missed a change, to give the users (or at least me) confidence in the AS option a ‘compare’ function is essential but once a sync opportunity is missed the ‘Read from’ and ‘Write to’ options cannot be guaranteed to give the same results as the AS(ON) option when it is capturing every situation accurately. Hence, my request for the other options!

Correctly - I know about “the first time auto” and I´m very glad for that since it´s very useful in my own work flow but would very much have the feature active all the time (one way) too. Auto sync is nothing I will have two ways. That seems very scary and I can´t see myself taking care of the conflicts that will occur manually.

Indeed that is why I suggested the options that I did, namely

The option that you would want is ‘Read from Image Automatically’ I would always favour ‘Add from …’ and once PL5 remove or control the “fiddling” I would probably set 'Add to ’ as well!

But the current AS(ON) is potentially risky, mostly because of the “fiddling” with the metadata format!

@Stenis The way that I wrote the options was not particularly clear so does this make it any clearer

It is actually harder to design the proper screen because of “localization” issues but there is actually space if the following items are moved down to provide a better layout. The ‘Always Read’ and ‘Always add’ are mutually exclusive, i.e. one or the other. The ‘Maintain keyword format’ should be available with any of the other options. If none of the ‘Always’ options is selected then the current automatic load from external on initial discovery could still be in operation as it currently is but the ‘Disable’ option then puts a stop to that.

Adding features to an existing program is always harder than building them into the initial code but options like these that effectively control the execution of sections of existing code and add new ‘Add’ features aren’t excessively complicated, if there is a will to make the program more “obedient” to the users wishes.

Splitting the code between the current (re-)formatting of the metadata by PL5 or simply passing it through unscathed is a bit more complicated but essential if DxO want the existing userbase to use these features of the product @sgospodarenko.

PPS:-

But none of these will control the behaviour of PL5 that if I add “A”, “B” and “C” to the keywords in PL5 then I am going to get BOTH ‘dc’ and ‘hr’ keywords for “A”, “B” and “C” when I should only get ‘dc’ keywords!