Unwanted Virtual Copies and Moving DxPL edited images (DOPs) between System - Revisited

I just responded to post on the ‘Unwanted Virtual Copies’ topic but had intended to start a new topic until I found things were not as I seemed when I undertook some testing recently.

So I will go ahead with the topic revisiting some “old chestnuts” and issues that might still “plagued” some users!

@danwdoo Its funny you should ask that!

The issue has not gone away and what I wrote here is correct with respect the how and whey the problem occurs, i.e. if an image with a DOP is presented to a copy of DxPL that already has that “EXACT” copy of that image in it then a Virtual Copy will be created as part of the “re-absorption” process, i,e, as I stated as was demonstrated then if this Uuid does not match the database entry Uuid then the “new”, non-matching DOP will result in a Virtual Copy and frequently that copy will be the latest and the one you want to become the [M]aster!

The Uuid as shown in the following table (taken from a document I have not published that needs to be updated for PL6)

image

But as time has passed more things have become apparent when forum members have posted issues.

I was about to start a topic requesting that DxO provided some options when this situation arises, e.g.

  1. Warn that the issue has arisen

  2. Offer the following options
    1 Ignore - make a VC from the new DOP data, this retains the current situation
    2 Overwrite - use only the new DOP data, no VC is created
    3 Make Master - Use new DOP data as the Master but retain old as the VC, the new DOP is the [M]aster and the existing DB DOP becomes the Virtual Copy

This requires some UI development but the code already exists and needs to be re-purposed @Musashi but implications need to be evaluated and checks and balances made to ensure users know and understand the implications.

The advantages of doing things this way is to leave all the internal references alone so that existing projects are not disrupted.

The procedure that some users have developed for themselves and the version I wrote for

  1. changing the original directory name in DxPL to -old,
  2. introducing the new versions as -new and
  3. when happy changing the name in DxPL to and
  4. finally deleting -old

is controlled, checkable and reversible right up to the point of deleting -old but no database pointers (references) are preserved and projects (if used) are “damaged”.

At the same time I started to investigate writing a Python script to swap the Uuids between the existing DOPs and the returning DOPs, effectively a programmatic way of of the above procedure.

To test the principle I recreated the problem and copied the DOP from the original DOP to the new revised DOP and opened DxPL (PL5.5.0) and it worked fine.

So I started to set up test data and repeated the exercise between my two system and discovered that the Uuids had not been changed. The receiving system did not change the Uuids when it added them to the database but retained their original values and so they retained to System A still matching the database and not a VC in sight!!??

Was this a new thing and on what release did it occur?

So I went back to PL4 and the same thing happened (or rather didn’t), no DOP change on System B when discovering the transferred directory so they came back to System A just fine.

So the potential for this working successfully has been there all along, DxPL must change the Uuid of an incoming DOP if that Uuid is already in use. During testing I am frequently clearing the database (deleting the db file) but I also reuse the same data many times and in my case there could easily be an entry in the database with the same Uuid if I haven’t cleared down the database before the test.

But why should this happen to users in “normal” circumstances?

So please conduct a test as follows:-

  1. Select some images, just the images with NO DOPs and put them in a test directory on System A as you would normally do for any files that are going to be moved between systems.
  2. Open in DxPL and make some obvious adjustments
  3. Select those images and put them in a DxPL ‘Project’.
  4. Close DxPL and copy the DOPs to a subdirectory for later comparison, e.g. “System A - DOPs”
  5. Use whatever method you are going to use to move the images to System B
  6. Open on System B and make some alternative obvious edits so that they have obviously changed
  7. Close on System B and copy the DOPs to a subdirectory for later comparison, e.g. “System B - DOPs”
  8. The DOPs can be compared at this point to see if the Uuid has changed. If it has then you can expect VC when those images and the new DOPs are “repatriated” to System A.
  9. Transfer the updated images back to System A via whatever is your chosen mechanism.
  10. Put them in place of the original images or simply attach the USB drive as it was originally etc.
  11. Open DxPL on System A and navigate to the directory and see what happens.
  12. Open the project created and see if the images are found as expected

Anyone who conducts this test please report back!?

If you regularly experience issues then please post exactly what you are doing that creates the problem!

1 Like

@DxO_Support-Team Can you please explain how Uuid’s are generated and how other users who are not using their images again and again (with their associated DOPs) can get the VC issue when moving between systems when my recent tests (also conducted on PL4) shows that DxPL will happily take a DOP into a system without changing its Uuid, as demonstrated by my tests!?

I have a simple solution for the unwanted VCs problem…

Always delete the database after changing file locations and before opening PL.

2 Likes

@joanna hello again. The problem is that some users want to move from a laptop to their main system and then back when they want to go on fields trips. If they want continuity then the two systems need to be able to track each other and stay in Sync…

The repatriation procedure I proposed and tested is fine and works providing there are no other references to the images, e.g. ‘Projects’. Given that you can’t even back-up projects and there are no project details in the DOP (although that could be a nightmare with really old DOPs that have lain untouched for a long time, referencing long “dead” projects) so projects exist only in the database!!

So throwing away the database rules out projects, using “my” repatriation procedure rules out projects so …

While I like the proposal I put forward, to be added to a (long) list of things that might never happen, if the critical DOP Uuid survives the journey to the other system then the DOPs will return just as if they never left!?

I don’t understand why all the problems we encountered occurred and wonder if the medication I am taking meant that my recent tests were not conducted correctly but a simple “fix” for the problem is to preserve the Uuids at all costs (but that is impossible if the Uuid is already in-use on the other system)!

My first attempt was System A (my Main system) on 5.5.0 to System B (my Test System) on 5.1.4! The DOPs were not backwards compatible, no edits appeared on System B and the Uuid was instantly changed!

Cloned the Test system SSD, upgraded to PL5.5.0 and reran the tests and the DOPs came back onto System A with the Uuids unchanged or so I believe, I will rerun the tests with PL6 later today or tomorrow.

PL4 is also installed on both machines and the DOPs went from System A to System B without Uuid change (or so I believe).

If the Uuids between systems are unlikely to “clash” then many transfers should go unscathed?

Which then leaves This image cannot be processed since the source file could not be found ( After I cloned the Drive ) where on WIN 10 using a ‘Mount Point’ seemed to be the only answer. The issue here was the drive was not getting the same UniqueId = Volumeid/number in the ‘Folders’ Table in the database!

So though it appeared that the drive was back in the same place it was actually getting a new UniqueId taken from its Volume Number/Id and a new entry in the ‘Folders’ structure!

So

  1. No unwanted Virtual copies but
  2. Duplicate data in the database (entries for both copies) were being maintained in the database)
  3. Project references were being “broken”

So I need to repeat my tests as follows

  1. Clear databases on both systems
  2. With new images carry out my procedure outlined above(steps 1-12 in the first post) and test projects.
  3. With more new images repeat step 2.
  4. With the same images from 2 but without DOPs repeat step 2
  5. Find a large database in my saved list and import into PL6 and repeat some or all of steps 2,3 and 4
  6. Report back in this topic
  7. Hope that any who are experiencing problems update this topic with their current experiences!

But all the metadata are not saved in the DOP ?
So, we lose this data ?

@Franky but all the metadata is stored in the PL5 DOP and you should also be passing that back to the image unless you are “protecting” the image.

With AS(OFF) from the PL5.3.0.release ALL metadata will be taken from the DOP and NONE (or so I believe) will be taken from the DOP on first discovery regardless of what release of PL5 created that DOP. For pre PL5 DOPs only the ‘Rating’ (‘Rank’) and ‘Rotation’ will be taken from a PL4 DOP and the rest from an xmp sidecar or image!

With AS(ON) the metadata for the [M]aster will be taken from the image on first discovery.

EDIT:-

The statement about ALL from the DOP is obviously wrong because the DOP doesn’t hold all the metadata from the image, in particular the EXIF data etc. but the keywords, IPTC data, ‘Rating’, ‘Rotation’ (now ‘Orientation’) will be taken from the DOP on first discovery with AS(OFF) and will “block” any potentially later metadata from the image. So null fields in the DOP for the IPTC for example will “block” data that might have been put into the image after processing with DxPL.

If in doubt after first (re-)discovery with AS(OFF) then use ‘Read from image’ to update from the image but please remember ‘Read from image’ and ‘Write to image’ are potentially destructive commands ‘Read from …’ will overwrite DxPLdb values (and then the DOP) and ‘Write from’ will overwrite the image metadata using DxPLdb values.

Franky is correct on that point; Metadata such as Ratings, Color tags, Keywords, etc - plus Project membership - are not held in the sidecar/.dop file … So, they’ll be lost if the database is deleted - - - Depending on how you work, that may not bother you - as it does not me.

John M

1 Like

DOP files written by DPL6.0.1.25 on Mac contain the following editable metadata

  • GPS
  • Colour Label
  • All 30 IPTC tags
  • Keywords including hierarchy
2 Likes

Thank you @platypus! I am seriously worried about the responses because it means that the communications for new releases is not getting through nor is a lot of what is written in forum posts.

I am not going to take the blame for “burying” crucial bits of data in long posts that are off-putting to the readers but we need to remove some major misconceptions about what DxO did in the PL5.3.0 release in particular!

Oh yes they are held and they will not be lost to the first discovery process, providing the first discovery system is running with AS(OFF)!!

PS:=

When the change came out in PL6 beta testing I complained, when it came out in PL5.3.0 release I complained and wrote a post to enable users with AS(OFF) to “restore” the old way of working (taking metadata from the image).

I complained, not about the facility’ but the way that it was attached to AS(OFF) rather than via a ‘Preferences’ option but that would have meant more work for DxO, so I understand (sort of), i.e. I would have preferred that the release stayed the same and a new option was provided for “first discovery” actions available to both AS(ON) and AS(OFF).

That’s for some people important and by suggesting “delete the whole database” without any disclaimers the harm is done and the adviser is not the person spending hours in gaining it back. I really hope (but don’t believe) that everyone in this forum is aware of two parties giving advice including a healthy database and the other doesn’t care for the database or destroys it systematically on a regular basis.

PhotoLab has evolved in writing stuff to the DOP sidecar files. What was true a few releases ago is not necessarily true any more. Therefore, we have to check, what our currently used version is doing - and we have to be precise about test results vs. DPL version and platform.

@JoJu I don’t advocate keeping or destroying the database and part of the reasons my posts are so long is that I attempt to impart as much knowledge about the situation as possible so users can make up their own minds but some have complained about the length of my posts in the past if my old memory serves me well!

Before I “destroy” the database for testing I back it up, once via DxPL and once in the name change I use to “remove” it from the system (i.e. two copies).

This whole post is about looking at preserving as much as possible while moving images backwards and forwards between system, if that is possible/practicable.

Not true. Metadata including Rating, Color Tags, Keywords, etc are all stored either directly in a non-RAW file or in an XMP sidecars for RAW files.

The only things you lose by deleting the database are projects and persistent history.

I don’t use projects or history and have never lost any metadata in years of deleting databases regularly.

2 Likes

@BHAYT I was not saying you were advocating to whatever. My reply meant this post

And I’m not sure if Joanna meant it sarcastically.

No. I am perfectly serious. And, because I do this, I have never had a problem of DOP conflicts.

Aha. And how does one know which DOP files are concerned by which kind of evolution? Files I edited 5 years ago then have different DOPs (if nothing has been changed) than today’s RAW-files. Meaning:

What metadata is stored today in the DOP was not stored in PL 3, 4 or 5 but in the database belonging to these versions, right? IMO a clear disadvantage of the DOP + database strategy. I don’t even know what default was set, when I started using PL, my takeaway is “relying on metadata handling of PL only can become an unpleasant surprise”. Of course, most of the posters have their bags full of workarounds and additional apps, but I am stupid and try to keep it simple. :crazy_face: :grin:

@JoJu The good news is that you don’t really need to know too much (sorry that sounds a little patronising and it actually wasn’t originally intended too).

DxPL is aware of the changes and providing the database upgrade goes O.K things work well, but there were hiccups going from PL4 to PL5 for some and ‘Projects’ were lost on at least one occasion and if the PL4 database was not upgraded successfully then any keywords that were only held in the PL4 database would have been “lost”, i.e. they were confined to the database on PL4 and never made it to the DOP.

The actual edit formats appear to change from one DOP write to the next!!??

But the overall format can be summarised as

Part of a Python program to hunt out the bits I am interested in but I need to add IPTC for the sake of completeness!

Not really, the main issue with PL5 was the change with respect to what could be relied on from the DOP and the only other thing was the keyword formatting which up until PL5.2.0 conformed to the same formatting as Lightroom and IMatch with one option selected and after PL5.2.0 only conformed to IMatch with another option selected (or de-selected?) and in all cases with all elements in a hierarchical key selected in DxPL it conforms to Capture One keyword formatting!

The export format is the same as above and has not changed since at least PL3 right up until the change above on PL5.2.0.

The problem has been the forum posts like yours that simply spread confusion from people who are intelligent enough not to be confused. It is possible that DxO considered their documentation to be sufficient but the number of half-truths and downright misunderstanding that still seem to abound is …

So this is from a recent post and I believe it to be accurate, all determined by empirical evaluation, I have never seen the code nor been given any more information than any other users!

So for what it is worth

Now, @BHAYT imagine all your inclusions, exclusions and exceptions you just described so knowingly put into a manual and you need to design a matrix telling the user “coming from this version, the function X will only work with limitations and the function Y will not work as you were used to and …”

I’m not very deep involved into DxO development strategies, but apparently there’s much more than one and the various variants are not co-exisiting as peacefully as I imagine they should - very soon my conclusion about “unpleasant surprises not excluded” is closer to the truth than your fully scope of already known “issues”. Plus the differences between Mac and Win versions… I tend to exaggerate but the whole project looks more like a gameboy-playground than a solid, reliable app.

Although I admit always, editing with it can be really satisfying. But things like projects, collections/albums, small a die-hard PL user doesn’t know of or is convinced to be devil’s work and should be avoided at all costs are important to me. And I see them, if at all, only on the rougher regions with some big stones of the said playground.

I suppose that you’re not stupid enough to not have backups…