Really ?? … OK, I defer to Mr. P’s advice on that - - and apols to @bhayt too.
The reason for my confusion is that I don’t maintain any of those attributes within PL … But, hang-on; hasn’t it been a long-time qualification/caution, if the database is deleted, then all that metadata is lost ??
Where did I get that impression from ?!
John M
Edit:
Ahhh - I had not kept up with those advances … My understandings are obviously out-of-date !
That’s most interesting … That leaves far fewer reasons to bother with the database (I don’t at all).
The only reason I can think of (other than for Projects … as persistent History is not provided with Win-PL anyway) is if one has 1,000s of {RAW+Sidecar/.dops} in the one folder … which is ill-advised in any case.
I don’t have conflicts either, or unwanted VCs, but I don’t delete the database; I use a tool called Resilio Sync to keep my folders in step between my laptop and my desktop. I only run PL on one system or the other at any time, and Resilio syncs the raw, the XMP and the DOP files all together. It syncs locally, and doesn’t require an internet connection. The result is that no matter which computer sees a new image first, PL on the other computer never sees a raw without a DOP, so they both keep the same ID. I also sync the presets with the same tool, so they are the same on both computers (which are Macs - I have no idea whether this would work on Windows).
Most interesting … I checked it out … Yep, it’s available on “real computers” too !!
… … … … … … … … … … … … … …(Where that was weak attempt at a joke for @Joanna )
@Aerenda With Beyond Compare which is available on both Win10 and MAC and is how I transfer between my machines and have done for many, many, many … years!
However, that doesn’t fix or avoid the problem if DxPL then discovers the images, complete with identical DOPs and as part of the “first discovery” process allocates a new Uuid to the incoming data!
The data remains the same but the id changes and that is the process by which I have always transferred data and during testing to reproduce problems reported by other users “discovered” the change of Uuid for myself.
But while I encountered the problems reported by many users my recent tests did not encounter the problem, and I presume that you are not encountering it either?
Hence, this topic was in response to a user asking about the “Unwanted Virtual Copies” and did the problem still exist and so I responded and then created this Topic to “air” my own concerns about
Why did we encounter problems in previous tests
Why did I have some successful recent tests, even when I went back to PL4 and tested that version of product!
The reason for not having the problem was because DxPL “discovered” the “new (to that copy)” images and used the Uuid from the DOP rather than allocate a new one, the problem is as simple and as complicated as that, i.e. it appears to be doing what is sensible and “safe” but why have users experienced problems?
In my case I have used and re-used test images, complete with DOPs and sidecars, numerous times in testing and not always destroying the database before each test. Once a Uuid is used a ‘Duplicates’ error would be signalled by SQLite on any attempt to use it again, but the programmers will have checked for existence before attempting a store (write) anyway, and generated a new Uuid to avoid any issue!
So did I, an ICT 1301 in 1964, then a smaller IBM 1401, then an ICL 1904 and a “baby” IBM 1130 before going into the “real world” and supporting Burroughs/Unisys Large Systems B6700, B7800, then onto the A series while flirting with the “Medium System” and “Small systems”, specialising in DMS II support for 36 years.
Set the ‘Tone curve’ and added “System A” keyword and closed PL6(A).
Secured image and original DOPs to “Baseline” and copied DOPs to “System A - DOP”
Copied images across the LAN to System B using Beyond Compare
Opened System B PL6 and navigated to the directory on the local drive, turned off ‘Tone Curve’ and added ‘DeepPRIME’ and “System B” keyword and closed PL6(B).
Used Beyond Compare to compare DOPs between the systems and the critical Uuid is identical and the keywords are as they should be (Drive F: is System A and Drive V: is Drive F: on System B mapped on System A !)
DOPs copied from System B to System A, with AS(OFF) on both systems the only way that metadata could be transferred was via the DOP (@John-M and @Franky) so we have no Virtual Copies, the correct edits and the correct keywords!
So I went for “broke”!
Following on from the previous tests so the numbers should be 8. …
Edited the images in PL6(A) and ‘Exported’ the DOPs.
Navigated away from the directory on PL6(A) but PL6(A)still running.
Used Beyond Compare to copy the DOPs to System B (the V: drive)
Opened PL6(B) and it was already on the required directory.
All images had the new edits entered on System A
Actually not a lot of point on my system because both are identical processors and memory but the Test machine (B) has a slightly faster 1050Ti with 4GB rather than my Main machine which has a 1050 with 2TB, both fairly rubbishy but slightly f a s t e r on B!
a side note
Keeping track with posts, I “scan” them and WHEN there is some interesting, I start reading/noticing.
As that’s difficult with long and even longer posts, better to break them down in paragraphs and if possible, put a ‘summary’ first. Otherwise they get ditched too quickly.
Might sound superficial – but like doing a research, one wants to find the relevant information …
Correct. The problem has only ever occurred for me if the second PL instance sees the new image without the DOP, and then the DOP appears afterwards - to be converted into a VC. In practice, the two files cannot appear at the same instant, so PL must not be allowed to look in the folder while the copying is happening - and the easiest way to achieve that is to close it when it is open elsewhere.
Agreed and sometimes I remember to include a summary. All my posts are broken down into small(ish) paragraphs because otherwise I can’t see the “wood for the trees”!
This topic is fairly self explanatory by virtue of its title, except when we deviate, albeit I hope that those deviations are useful for those that missed the implications of the PL5.3.0 change!
In the meantime I loaded an old database which was not really big enough so “loaded” 11,000 images from a bulk test I did some time ago!
Then repeated the import on System B of the images from Step 3 (the second Step 3) above and they came into the now somewhat larger database without any change to the Uuid. The DxO FAQ states the following
Yes any import (discovery) will take the image into the database and a Uuid will be allocated at that point and any other DOP from any other source will not match that database entry Uuid and a VC will result.
But I new that when I did my tests but I ran into problems the same or similar to what others had reported! It is possible that the tests were conducted between different version of PL5 and I believe that is a no-no!
All the tests that I have been conducting in this topic have been two databases, two copies of PL6 and two identically named directories with data being synchronized between them manually using Beyond Compare, effectively my editing is flip-flopping between the two systems!
So are these the RULES for avoiding VCs and sharing images between systems?:-
The same version of DxPL must be installed on both (all) systems.
The first discovery on the second system (System B for want of a better name) must ONLY occur for an image from System A with a DOP(A) already in place, the DOP(A) that matches with the DxPLdb entry on System A.
The result should be clean import of the image and the details (and Uuid) from the DOP(A) into System B DxPL and a retention of the Uuid in the database, which will then be in DOP(B). Any difference in the critical Uuid (the one at the location in the DOP that I have been going on about since my posts in the original ‘Unwanted Virtual Copies’ topic) will ultimately result in a VC.
Thereafter disciplined use of the two systems should result in it being possible to transfer edits between the two systems via the DOP BUT
Once the DOPs have been found after that initial discovery then no metadata will be transferred from via the DOP to the database regardless of AS(OFF) or AS(ON)!
1 The use of the certain metadata from the DOP is confined to the first discovery event thereafter all metadata transfers must come from the image (embedded or sidecar xmp etc.).
2 BUT what about my test results that showed the “System B” keyword appearing on System A (a test I must repeat)
3 So I added another Keyword and some IPTC data on System A and copied DOP(A) to DOP(B) with the directory open on System B! No damage but no new keyword either.
4 Navigated away from the directory on System B and added another keyword on System A and copied DOPs(A) to overwrite DOPs(B) and navigated back to the directory on System B
5 Navigated back to the directory on B and no keywords detected but executed a ‘Sidecar’/‘Import’ and up popped the new keywords and the IPTC data!
6 With the directory open on B I added another keyword on A and copied the DOPs over system B DOPs! No keyword change detected, tried an ‘F5’ but no change until I executed a ‘Sidecar’/‘Import’ when the new keyword appeared
7 repeated the exercise with B not on the same directory and made an image edit and copied the DOPs A to B and navigated back to the directory on B and the edits were immediately visible (I had turned off DeepPRIME)
8 repeated the exercise with B on the same directory and turned on DeepPRIME on A and copied the DOPs to B and the DeepPrime icon came on immediately
9 repeated 8 but with ‘Rating’ and B did not react, ‘F5’ caused no change but ‘Import’ worked just fine!
Throughout the above tests both databases were active and nothing particularly untoward happened. I now need to repeat the tests with real metadata storage i.e. xmp sidecars and see what happens and what blocks what or not!!?
Please make sure to be precise (not more detailed)
DPL can be set/used to act on .dop sidecars in several ways
auto import
auto export
manual import
manual export
combinations thereof
DPL can be set/used to act on XMP (data in files or .xmp sidecars)
auto sync (not separate for I/O)
manual import
manual export
combinations thereof
I suppose (have not tested it) that XMP is not involved in the creation of virtual copies, which means that only the image files and/or the respective .dop sidecars cause VCs to appear…which nevertheless creates quite a bunch of scenarios that require testing.
Creation of VCs = f (DB, image file, .dop sidecar, app settings, manual actions…)
@platypus I wrote the post as I conducted the tests with no preconceived notion about what I might “discover” hence, the detail.
I could (and ultimately will) rewrite the outcome of any tests that I conduct.
Because I use the defaults I look forward to your tests with the other combinations which will give us an insight into both what those combinations might help/hinder and what happens on the MAC platform!
and the DOP sidecar settings will remain as they are for the remainder of my tests. They will have a material impact of the way things work because some of my tests produced metadata from the DOP but only after an ‘Import’.
Hence. if you run with the ‘load settings’ OFF (unchecked) you won’t know/see what happens automatically but will ultimately get the same results as I did with my automatic DOP load not refreshing the metadata but the ‘Import’ doing that job, because the only way of getting new DOP data is via the ‘Import’ with the option unchecked.
You are welcome to check all the combinations but it should be easy to extrapolate them from my results with the options at their default settings.
The xmp sidecar and embedded (RAW and JPG) images is a different situation and I will test a number of scenarios but not every combination, what I don’t test you are welcome to fill in the additional detail.
THE ONLY THING THAT GOVERNS VCs IS THE Uuid BUT the state of the image might, i.e. not just the DOP, might cause a different reaction.
EXCEPT that what seals the fate of the Uuid is that first import into the second system (System B) and since all the data is new to System B what state of that data could affect the import!?
The answer to that is where that data resides and where the database resides and those are yet more tests
Data moved between system on a USB drive (Flash or SSD or HDD or …)
Two systems attempting to access one set of data.
Two systems attempting to access one set of data with just one database
Please add other scenarios to the above list and I will then be looking for volunteers!
Good of you to enumerate the various combinations now how about helping by testing some!
Update:-
If you ever want to synchronise your systems DO NOT TURN OFF the automatic DOP load. If you do you fall into the trap discussed earlier. The image is discovered on B but no edit nor metadata is imported but a Uuid is allocated by System B.
Now the fate of the image is sealed and when you ‘import’ from the DOP you are going to get a guaranteed VC without really trying too hard.
So another Rule for Synchronisation to work the ‘Load from DOP’ option must be active but that issue (discovery of an image must be with a DOP) was discussed at the beginning of my post courtesy of a reminder from @Aearenda !
Well, I’m not going to test all the variations because I don’t work on more than one computer, something that is not meant to happen anyway. Maybe DxO will add some functionality that allows the sync of databases and/or files across computers for a seamless user experience in a future release…
Congrats to all of you having the will and stamina to try nonetheless.
@platypus What the tests show is that arguably DxO could claim the functionality is there already!
@JoJu won’t like it because I have made some tentative additional tests and they show certain issues once Metadata is taken into account, i.e. currently in the sidecar in my case, so there will be caveats but if you have a laptop and a main machine then
With two copies of exactly the same version of DxO.
With AS(OFF) but the DOP options left to default.
With two separate databases and two separate file systems (currently mine are identical for the directory under test but I don’t believe that is even vaguely important)
Synchronisation is not only possible, it is also possible to pass edits backwards and forwards with only a little care in real time!
But that might start to come unglued if the metadata is carried around using the correct metadata handling instead of DOPs, but I am not yet sure of the exact boundary conditions where the well behaved system that I was happily using earlier becomes more complicated (or not).
I have no problems updating XMPs via my dam if I make changes through that, new keywords as done today. The data was copied to the laptop from the desk pc and later found out some fungus identities. Added to PL ok and it updated the information it held correctly and displayed the updated keywords.
After an appalling set of tests, all the problems were started by my own silly mistakes and I will write up the issues tomorrow. In the meantime the following bug is present in PL5.5.0 and PL6.0. and on PL6.0.1 which has just finished downloading and installing!
While conducting tests I had to keep clearing data because of my errors and the occassional DxPL “glitch” so I stopped deleting images and started renaming the directory so here is the error I encountered.
My test directory contains 4 images, 2JPG and 2 RAW, 4 DOPs and 2 xmp sidecar files.
I copy them to a test location and navigate to that in DxPL (AS(OFF)) but DOP load etc. set as default (ON)!.
Change the name of the directory to “{name}-Renamed”
Now copy the original set of images, DOPs and sidecar files to (as) the original directory so that there are two directories containing the same sets of files, “{name}-Renamed” and the newly reconstructed “{name}”
Select the 4 images in the “{name}” directory and ‘Remove’ in DxPL
DxPL deletes the images from “{name}-Renamed” but shows “{name}” as being empty but “{name}-Renamed” also shows as empty and is indeed empty!
Only a Restart will show the true state of “{name}”, i.e. images intact but the “{name}-Renamed” images have all been deleted from the disk!!
DxPL is getting its wires (database references) crossed or something like that!!!
@platypus Sorry if I had used ‘Rename folder’ it would have made it clearer than just “Change the name…”.
It was done in DxPL to minimise the “risks” of any time lags that might have occurred when DxPL discovered a directory had vanished and needed to clean up the database as a consequence!
Instead it ties itself in knots instead!
It just rounded out a set of tests that I should have left until the following day because DxPL does not handle removing directories (as a consequence of my own mistakes) particularly well as you will see in posts when I have composed the post for two other issues that occurred along the way!?
Bryan, I must say that I have enormous difficulty following any of your “tests”. They are just way too complex since they seem to deal with more than one problem at a time - far too much to hold in the brain.
You ask about rules for avoiding problems. There is one simple rule - don’t pass and repass images and DOPs between machines unless you…
