@JoJu I don’t advocate keeping or destroying the database and part of the reasons my posts are so long is that I attempt to impart as much knowledge about the situation as possible so users can make up their own minds but some have complained about the length of my posts in the past if my old memory serves me well!
Before I “destroy” the database for testing I back it up, once via DxPL and once in the name change I use to “remove” it from the system (i.e. two copies).
This whole post is about looking at preserving as much as possible while moving images backwards and forwards between system, if that is possible/practicable.
Aha. And how does one know which DOP files are concerned by which kind of evolution? Files I edited 5 years ago then have different DOPs (if nothing has been changed) than today’s RAW-files. Meaning:
What metadata is stored today in the DOP was not stored in PL 3, 4 or 5 but in the database belonging to these versions, right? IMO a clear disadvantage of the DOP + database strategy. I don’t even know what default was set, when I started using PL, my takeaway is “relying on metadata handling of PL only can become an unpleasant surprise”. Of course, most of the posters have their bags full of workarounds and additional apps, but I am stupid and try to keep it simple.
@JoJu The good news is that you don’t really need to know too much (sorry that sounds a little patronising and it actually wasn’t originally intended too).
DxPL is aware of the changes and providing the database upgrade goes O.K things work well, but there were hiccups going from PL4 to PL5 for some and ‘Projects’ were lost on at least one occasion and if the PL4 database was not upgraded successfully then any keywords that were only held in the PL4 database would have been “lost”, i.e. they were confined to the database on PL4 and never made it to the DOP.
The actual edit formats appear to change from one DOP write to the next!!??
Part of a Python program to hunt out the bits I am interested in but I need to add IPTC for the sake of completeness!
Not really, the main issue with PL5 was the change with respect to what could be relied on from the DOP and the only other thing was the keyword formatting which up until PL5.2.0 conformed to the same formatting as Lightroom and IMatch with one option selected and after PL5.2.0 only conformed to IMatch with another option selected (or de-selected?) and in all cases with all elements in a hierarchical key selected in DxPL it conforms to Capture One keyword formatting!
The export format is the same as above and has not changed since at least PL3 right up until the change above on PL5.2.0.
The problem has been the forum posts like yours that simply spread confusion from people who are intelligent enough not to be confused. It is possible that DxO considered their documentation to be sufficient but the number of half-truths and downright misunderstanding that still seem to abound is …
So this is from a recent post and I believe it to be accurate, all determined by empirical evaluation, I have never seen the code nor been given any more information than any other users!
Now, @BHAYT imagine all your inclusions, exclusions and exceptions you just described so knowingly put into a manual and you need to design a matrix telling the user “coming from this version, the function X will only work with limitations and the function Y will not work as you were used to and …”
I’m not very deep involved into DxO development strategies, but apparently there’s much more than one and the various variants are not co-exisiting as peacefully as I imagine they should - very soon my conclusion about “unpleasant surprises not excluded” is closer to the truth than your fully scope of already known “issues”. Plus the differences between Mac and Win versions… I tend to exaggerate but the whole project looks more like a gameboy-playground than a solid, reliable app.
Although I admit always, editing with it can be really satisfying. But things like projects, collections/albums, small a die-hard PL user doesn’t know of or is convinced to be devil’s work and should be avoided at all costs are important to me. And I see them, if at all, only on the rougher regions with some big stones of the said playground.
Really ?? … OK, I defer to Mr. P’s advice on that - - and apols to @bhayt too.
The reason for my confusion is that I don’t maintain any of those attributes within PL … But, hang-on; hasn’t it been a long-time qualification/caution, if the database is deleted, then all that metadata is lost ??
Where did I get that impression from ?!
John M
Edit:
Ahhh - I had not kept up with those advances … My understandings are obviously out-of-date !
That’s most interesting … That leaves far fewer reasons to bother with the database (I don’t at all).
The only reason I can think of (other than for Projects … as persistent History is not provided with Win-PL anyway) is if one has 1,000s of {RAW+Sidecar/.dops} in the one folder … which is ill-advised in any case.
I don’t have conflicts either, or unwanted VCs, but I don’t delete the database; I use a tool called Resilio Sync to keep my folders in step between my laptop and my desktop. I only run PL on one system or the other at any time, and Resilio syncs the raw, the XMP and the DOP files all together. It syncs locally, and doesn’t require an internet connection. The result is that no matter which computer sees a new image first, PL on the other computer never sees a raw without a DOP, so they both keep the same ID. I also sync the presets with the same tool, so they are the same on both computers (which are Macs - I have no idea whether this would work on Windows).
Most interesting … I checked it out … Yep, it’s available on “real computers” too !!
… … … … … … … … … … … … … …(Where that was weak attempt at a joke for @Joanna )
@Aerenda With Beyond Compare which is available on both Win10 and MAC and is how I transfer between my machines and have done for many, many, many … years!
However, that doesn’t fix or avoid the problem if DxPL then discovers the images, complete with identical DOPs and as part of the “first discovery” process allocates a new Uuid to the incoming data!
The data remains the same but the id changes and that is the process by which I have always transferred data and during testing to reproduce problems reported by other users “discovered” the change of Uuid for myself.
But while I encountered the problems reported by many users my recent tests did not encounter the problem, and I presume that you are not encountering it either?
Hence, this topic was in response to a user asking about the “Unwanted Virtual Copies” and did the problem still exist and so I responded and then created this Topic to “air” my own concerns about
Why did we encounter problems in previous tests
Why did I have some successful recent tests, even when I went back to PL4 and tested that version of product!
The reason for not having the problem was because DxPL “discovered” the “new (to that copy)” images and used the Uuid from the DOP rather than allocate a new one, the problem is as simple and as complicated as that, i.e. it appears to be doing what is sensible and “safe” but why have users experienced problems?
In my case I have used and re-used test images, complete with DOPs and sidecars, numerous times in testing and not always destroying the database before each test. Once a Uuid is used a ‘Duplicates’ error would be signalled by SQLite on any attempt to use it again, but the programmers will have checked for existence before attempting a store (write) anyway, and generated a new Uuid to avoid any issue!
So did I, an ICT 1301 in 1964, then a smaller IBM 1401, then an ICL 1904 and a “baby” IBM 1130 before going into the “real world” and supporting Burroughs/Unisys Large Systems B6700, B7800, then onto the A series while flirting with the “Medium System” and “Small systems”, specialising in DMS II support for 36 years.
Set the ‘Tone curve’ and added “System A” keyword and closed PL6(A).
Secured image and original DOPs to “Baseline” and copied DOPs to “System A - DOP”
Copied images across the LAN to System B using Beyond Compare
Opened System B PL6 and navigated to the directory on the local drive, turned off ‘Tone Curve’ and added ‘DeepPRIME’ and “System B” keyword and closed PL6(B).
Used Beyond Compare to compare DOPs between the systems and the critical Uuid is identical and the keywords are as they should be (Drive F: is System A and Drive V: is Drive F: on System B mapped on System A !)
DOPs copied from System B to System A, with AS(OFF) on both systems the only way that metadata could be transferred was via the DOP (@John-M and @Franky) so we have no Virtual Copies, the correct edits and the correct keywords!
So I went for “broke”!
Following on from the previous tests so the numbers should be 8. …
Edited the images in PL6(A) and ‘Exported’ the DOPs.
Navigated away from the directory on PL6(A) but PL6(A)still running.
Used Beyond Compare to copy the DOPs to System B (the V: drive)
Opened PL6(B) and it was already on the required directory.
All images had the new edits entered on System A
Actually not a lot of point on my system because both are identical processors and memory but the Test machine (B) has a slightly faster 1050Ti with 4GB rather than my Main machine which has a 1050 with 2TB, both fairly rubbishy but slightly f a s t e r on B!
a side note
Keeping track with posts, I “scan” them and WHEN there is some interesting, I start reading/noticing.
As that’s difficult with long and even longer posts, better to break them down in paragraphs and if possible, put a ‘summary’ first. Otherwise they get ditched too quickly.
Might sound superficial – but like doing a research, one wants to find the relevant information …
Correct. The problem has only ever occurred for me if the second PL instance sees the new image without the DOP, and then the DOP appears afterwards - to be converted into a VC. In practice, the two files cannot appear at the same instant, so PL must not be allowed to look in the folder while the copying is happening - and the easiest way to achieve that is to close it when it is open elsewhere.
Agreed and sometimes I remember to include a summary. All my posts are broken down into small(ish) paragraphs because otherwise I can’t see the “wood for the trees”!
This topic is fairly self explanatory by virtue of its title, except when we deviate, albeit I hope that those deviations are useful for those that missed the implications of the PL5.3.0 change!
In the meantime I loaded an old database which was not really big enough so “loaded” 11,000 images from a bulk test I did some time ago!
Then repeated the import on System B of the images from Step 3 (the second Step 3) above and they came into the now somewhat larger database without any change to the Uuid. The DxO FAQ states the following
Yes any import (discovery) will take the image into the database and a Uuid will be allocated at that point and any other DOP from any other source will not match that database entry Uuid and a VC will result.
But I new that when I did my tests but I ran into problems the same or similar to what others had reported! It is possible that the tests were conducted between different version of PL5 and I believe that is a no-no!
All the tests that I have been conducting in this topic have been two databases, two copies of PL6 and two identically named directories with data being synchronized between them manually using Beyond Compare, effectively my editing is flip-flopping between the two systems!
So are these the RULES for avoiding VCs and sharing images between systems?:-
The same version of DxPL must be installed on both (all) systems.
The first discovery on the second system (System B for want of a better name) must ONLY occur for an image from System A with a DOP(A) already in place, the DOP(A) that matches with the DxPLdb entry on System A.
The result should be clean import of the image and the details (and Uuid) from the DOP(A) into System B DxPL and a retention of the Uuid in the database, which will then be in DOP(B). Any difference in the critical Uuid (the one at the location in the DOP that I have been going on about since my posts in the original ‘Unwanted Virtual Copies’ topic) will ultimately result in a VC.
Thereafter disciplined use of the two systems should result in it being possible to transfer edits between the two systems via the DOP BUT
Once the DOPs have been found after that initial discovery then no metadata will be transferred from via the DOP to the database regardless of AS(OFF) or AS(ON)!
1 The use of the certain metadata from the DOP is confined to the first discovery event thereafter all metadata transfers must come from the image (embedded or sidecar xmp etc.).
2 BUT what about my test results that showed the “System B” keyword appearing on System A (a test I must repeat)
3 So I added another Keyword and some IPTC data on System A and copied DOP(A) to DOP(B) with the directory open on System B! No damage but no new keyword either.
4 Navigated away from the directory on System B and added another keyword on System A and copied DOPs(A) to overwrite DOPs(B) and navigated back to the directory on System B
5 Navigated back to the directory on B and no keywords detected but executed a ‘Sidecar’/‘Import’ and up popped the new keywords and the IPTC data!
6 With the directory open on B I added another keyword on A and copied the DOPs over system B DOPs! No keyword change detected, tried an ‘F5’ but no change until I executed a ‘Sidecar’/‘Import’ when the new keyword appeared
7 repeated the exercise with B not on the same directory and made an image edit and copied the DOPs A to B and navigated back to the directory on B and the edits were immediately visible (I had turned off DeepPRIME)
8 repeated the exercise with B on the same directory and turned on DeepPRIME on A and copied the DOPs to B and the DeepPrime icon came on immediately
9 repeated 8 but with ‘Rating’ and B did not react, ‘F5’ caused no change but ‘Import’ worked just fine!
Throughout the above tests both databases were active and nothing particularly untoward happened. I now need to repeat the tests with real metadata storage i.e. xmp sidecars and see what happens and what blocks what or not!!?
Please make sure to be precise (not more detailed)
DPL can be set/used to act on .dop sidecars in several ways
auto import
auto export
manual import
manual export
combinations thereof
DPL can be set/used to act on XMP (data in files or .xmp sidecars)
auto sync (not separate for I/O)
manual import
manual export
combinations thereof
I suppose (have not tested it) that XMP is not involved in the creation of virtual copies, which means that only the image files and/or the respective .dop sidecars cause VCs to appear…which nevertheless creates quite a bunch of scenarios that require testing.
Creation of VCs = f (DB, image file, .dop sidecar, app settings, manual actions…)
@platypus I wrote the post as I conducted the tests with no preconceived notion about what I might “discover” hence, the detail.
I could (and ultimately will) rewrite the outcome of any tests that I conduct.
Because I use the defaults I look forward to your tests with the other combinations which will give us an insight into both what those combinations might help/hinder and what happens on the MAC platform!
and the DOP sidecar settings will remain as they are for the remainder of my tests. They will have a material impact of the way things work because some of my tests produced metadata from the DOP but only after an ‘Import’.
Hence. if you run with the ‘load settings’ OFF (unchecked) you won’t know/see what happens automatically but will ultimately get the same results as I did with my automatic DOP load not refreshing the metadata but the ‘Import’ doing that job, because the only way of getting new DOP data is via the ‘Import’ with the option unchecked.
You are welcome to check all the combinations but it should be easy to extrapolate them from my results with the options at their default settings.
The xmp sidecar and embedded (RAW and JPG) images is a different situation and I will test a number of scenarios but not every combination, what I don’t test you are welcome to fill in the additional detail.
THE ONLY THING THAT GOVERNS VCs IS THE Uuid BUT the state of the image might, i.e. not just the DOP, might cause a different reaction.
EXCEPT that what seals the fate of the Uuid is that first import into the second system (System B) and since all the data is new to System B what state of that data could affect the import!?
The answer to that is where that data resides and where the database resides and those are yet more tests
Data moved between system on a USB drive (Flash or SSD or HDD or …)
Two systems attempting to access one set of data.
Two systems attempting to access one set of data with just one database
Please add other scenarios to the above list and I will then be looking for volunteers!
Good of you to enumerate the various combinations now how about helping by testing some!
Update:-
If you ever want to synchronise your systems DO NOT TURN OFF the automatic DOP load. If you do you fall into the trap discussed earlier. The image is discovered on B but no edit nor metadata is imported but a Uuid is allocated by System B.
Now the fate of the image is sealed and when you ‘import’ from the DOP you are going to get a guaranteed VC without really trying too hard.
So another Rule for Synchronisation to work the ‘Load from DOP’ option must be active but that issue (discovery of an image must be with a DOP) was discussed at the beginning of my post courtesy of a reminder from @Aearenda !
