Unwanted Virtual Copies and Moving DxPL edited images (DOPs) between System - Revisited

@Joanna I am sorry if you find my tests and explanations too long and tortuous to follow I must work on that. The truth is that I find the same with some of yours and others where the writer knows and understands the problem but the reader can’t necessarily get up to speed quickly enough! But the cryptic posts don’t work either so …

But your solution is, I believe,

  1. Largely unnecessary

  2. Way too destructive, particularly to achieve harmony between a field laptop and a desktop system.

  3. My alternative procedure that I espoused some time ago should work, although the deletion bug I have just found might just “interfere” with that.

  4. The tests above worked perfectly and so I moved into testing with metadata in tow (in the containers that should be used) and attempted the tests after an afternoon chopping down vegetation and clearing up the debris with only a head torch for light! I made too many mistakes setting up the tests because I was tired and found some old bugs I have encountered before and the new “slight of hand” deletion bug.

  5. None of this is really hard, the hardest part is actually understanding what is possible when we are working in a vacuum of non-communication from DxO.

I believed the fallacy about how hard hierarchical keys are and how they should be avoided at all costs! Then I realised it was simply a formatting issue and every package has adopted its own variation in rules! If you implemented my Keyword Formatting Template design your program could be everything to everyone on the MAC, except a purist like you!?

But that is actually just another entry in the table so your package can interwork between all the other packages!?

The problem with your package and the Python KFT script I am (slowly) developing when not gardening, DIYing, testing and writing huge posts, is that they are yet another component when the technology could so easily be built into DxPL @Musashi!?

The hardest part about the tests that I am currently doing is the time it takes to set them up and capture results when the tests are over in a couple of minutes!

So cryptic version of the long post

  1. According to my tests it is possible to synchronise two copies of the database, and images and DOPs between two system running identical versions of DxPL when the metadata is passed via the DOP. So well that DxPL can be active on both systems at the same time and the edit data will happily flip flop between the systems automatically!

What I was starting to test when I made errors and then found errors was to repeat that success with “conventionally held” metadata in tow!

My rule is: Never let PL ‘see’ an image without a DOP sidecar, apart from when first importing.

This process works for me, and involves no pain and no database deletion (but I don’t manage ratings, keywords or colour labels in PL):

  1. In preferences, set PL to automatically import and export sidecars.
  2. Introduce new images to PL on one computer, and make sure it generates DOPs for all the images (a default preset other than ‘No corrections’ will do that).
  3. With PL closed on both computers, copy the images and associated DOPs (and any XMPs) from the first to the second.
  4. Open PL on the second computer and move to the folder with the new images; it will ingest them and use the same ID as it finds in the DOP files.
  5. Thereafter, the DOPs can be copied back and forth after each editing session, but never let PL be running on both computers at once.

In practice, as I mentioned earlier, I use Resilio Sync to keep the folders in step between the computers, and this works even when PL is open on one of the computers. This all works reliably for me on Macs; other sync tools on other computers may work as well - YMMV.

3 Likes

I agree, using Windows and Syncovery, with 2 machines I have never had a problem working as Aearenda describes. Indeed its 3 way using a DAM and XPMs can be updated and the resulting changes imported into PL on both PC and laptop. It can get complicated as weeding in PL needs my DAM to be updated on both machines and PL updated its self after copying the changes between them. But as PL is hopeless for lack of database management I regularly delete the data base to clear out any problems that might be create on them. The DAM is no problem it is run to verify changes and this keeps its database and thumbnails updated. But I have never hadf any problems doin this.

My testing indicates the same!

A must! If you turn off the DOP import then transfer without VCs will be impossible

I believe that a forced DOP Export on all images selected within a directory will ensure that a DOP is created regardless of whether the image has been accessed or not. Applying a ‘No Correction’ preset also seems to have created a DOP which was written in the 20 second DOP cycle!

Confirmed by my testing, including for images with embedded and sidecar xmp data! With AS(OFF) the metadata will be taken from the DOP of first discovery and can then be updated from the image with a ‘Read from image’ if deemed necessary or expediate.

But so far my tests have indicated that even leaving the same directory (not the same but the clones) open and selected is perfectly O.K. and having the directory active at the time that the copy is made from one to the other is fine!

The edit data, including the Tag is discovered and displayed immediately but the metadata from the DOP needs to be updated with an ‘Import’ and from the image needs to be updated with a ‘Read from image’ !

Beyond Compare is my software of choice and is available on Windows and Mac and perhaps a little cheaper. The main reason that I use it is that I don’t want automatic backup but I do want to know what differences there may be and to compare the contents of the files in detail!

But each to their own favourites!

You may be right but be specific please, if the features that you are referring to are the ones that @platypus and I have requested that is understandable particularly as I understand from @platypus’s comments

then the rename capability is not available on the MAC.

The DxPL database is not particularly weak and is SQLite like most of the other products, ACDSee uses a Dbase derivative, Photo Mechanic uses none (except with the Plus) version, FRV uses none.

Please be specific, what problems have you personally experienced.

I call DxO out time and time again for issues I have found and consider should be fixed and could be fixed with very little development but there is way to much generalised criticism in the forums.

The reason for this topic was to see if I could get to the bottom of what was true and untrue about using DxPL to maintain “paired” system processing" The original topic which went by the title of “Unwanted Virtual Copies” ran for a year plus others that discussed nightmare issues when trying to synchronise between machines why do these problems appear to exist what are the facts and what is the fiction!

Thanks to what @Aearenda has written and I have discovered in my testing (thus far) is that databases don’t need to be destroyed (but can be if that suits the user) and synchronicity can be successfully maintained between two systems!

However, when I went “off track” with my testing I (re-)discovered issues that I need to re-rest and report which may indicate that the database clean-up routines in DxPL can be “fooled” but I was reloading the same images again and again with the same DOPs and …

and
You may be right but be specific please, if the features that you are referring to are the ones that @platypus and I have requested that is understandable particularly as I understand from @platypus’s comments

I remember the time early PL and Optics Pro when the standard support response to almost every problem was delete the database. Indeed many times that was indeed the problem, though not always!

I haven’t had problems as I delete it, I don’t need it. BUT copying between two PC’s and at usually this includes when copying back to the laptop a largish proportion of the original image’s will have been deleted after dealing with them on good screens. On the laptop as well I only keep the last two years image’s, to keep Photo Supreme updated they are deleted via it (don’t think any way of doing so via PL). Thus the combined regular deletion of image’s and annual year deletion means the PL database would be filled with no longer existing information, I don’t need any way as I use Photo Supreme for keywords and projects. Photo Supreme has a range of tools to maintain the database and its contents. I can verify folders, years or even the whole collection as I have done a number of times when retrospectively adding keywords. I can scan my drive for image’s that are missing files or folders. Unlike PL I compact the database which also verifies it and can export with verification.

Thus on the laptop there is a constant flux of image’s that PL isn’t designed to deal with. On the desk top I don’t use anything in the data base and as said right at the beginning go back to the days when the data base was a first port of call in problems with PL and earlier.

Indeed, that often solved the issues. Recent versions of DPL have been more stable though, but the DB and photo archive tend to diverge, unless we do not move, rename or delete files outside of DPL.

I have tried to keep a healthy database for many years, but it is hard to do while testing and in view of the fact that I cannot tell DPL to create and use a different database. I’ve therefore switched to delete the DB regularly, stick to Lightroom for its robust management functionalities and DPL for is optical and noise corrections.

2 Likes

Photo Supreme both allows multiple data bases and you can have material in it that’s not directly accessible. Its not something I have used, I clear out the former years images to keep the storage down on my laptop that getting on a bit now and doesn’t have the size of storage more widely used now. But switching databases is an option easily used as I expect most proper DAM’s have…

@platypus I know you use a Mac so what works on a Win10 PC may not work on a Mac but although the commands are not directly accessible the following is possible (on a Win10 P.C. at least) and useful for testing!

  1. Navigate to an agnostic directory, possibly one added for the purpose, i.e. one without actual images in that directory
  2. ‘Create a backup’ of the current database in DxPL and close DxPL
  3. Delete or change the name of the active database!
  4. Re-open DxPL and immediately ‘Create a backup’ e.g. “PL6.0.1 - Blank DB for Testing”
  5. Restore the previous database which requires a restart.
  6. Whenever you want to test then ‘Create a Backup’ of the current DB, be that a production copy or another test copy and Restore the blank ready for a new test or a previous backup of an in progress test or …
  7. BUT nowhere in DxPL (I believe) is there any clue to the origins of the database and any browsing of directories will quickly just add those items to the database! I actually hate the import process of other products but it means that it is harder to add more and more to the database simply by the act of browsing! !?

@John7 Photo Supreme that you use and IMatch that I own are full blown Image DAMs, that is their principal or sole reason for being!

DxPL is a RAW processor and editor with a workable DAM infrastructure, which is essentially an extension of what was put in place to store editing data (with the DOPs as a portable backup which can “live” with the image wherever it goes!?)!

There is a danger of condemning one for not being on a par with the other, but DxPL does require some additional features as various forum members have posted from time to time. The most infuriating aspect is that many of these features are already available in free software and would take very little effort to add to DxPL.

They are not new headline grabbing features that will drive sales but they would inch by inch, sorry centimetre by centimetre (I typically use both when measuring for cutting wood etc.), improve the overall usability of the product for existing and future users

@BHAYT, my point is, that I want to use DPL as is, without workarounds. Today, PhotoLab on Mac provides no means to use several databases. Paths are buried in preferences files and no provisions exist to access the respective entries. That is okay for me as long as I’ll use DPL as a sidestory to Lightroom.

DxO could easily produce a “DxO PhotoLab Pro” package with decent DAM services and DB maintenance plus all the editing goodies of today’s DPL. Some paradigms would need to change though, one of them being automatic import, which is the first weak spot of the current approach imo.

2 Likes

Do you mean import of sidecar/.dop files ? … If so then I say “au contraire” … because , for me, this is a key feature of PL … in fact, the very one that led me to choose OpticsPro (as it was named pre-PhotoLab) over alternatives that depended on a “black-box” database.

John M

Why, what in particular requires the automatic import to go, please be precise. I indicated why it can be a nuisance but what is your reasoning?

@John-M you have explained why you don’t want it to go but your response is a “typical” DxO forum response that sometimes starts a lot of batting between forum members!

The topic to which this topic owes part of it title ran for almost a year with 91 replies and currently has 5,300 views and 20 users taking part. So many things about that topic were wrong, wrong, wrong and it could have all been explained at the very beginning had DxO stepped in and explained what was actually happening @Musashi, @CaptainPO, @DxO_Support-Team but they didn’t, it was just left to run.

I consider that lamentable!

We are unlikely to get such traction in this topic with ‘Import’ versus automatic import (the current situation) but the statement and the response is how it starts and DxO watch and take whatever they want or don’t want from it and we get PL5.2.0 keyword format change for the worse and we get PL5.3.0 reversion to old DOP handling but attached to a parameter that didn’t even exist prior to PL5!

The evidence of how effective the change can be in testing is shown in this topic but the better way would have been via a parameter giving the choice of automatically using the DOP (or not) for first discovery metadata with AS(OFF) and AS(ON).

So, given I made a comment above about “careless” browsing populating the DxPLdb with “detritus” I would like an option to “not” import but given I use IMatch, I hate one aspect of that product, the “Media” screen attached to its import function or rather shown after the actual import.

it is claustrophobic etc. etc whereas the DxPL input screen is

and enables me to see a directory amongst other carefully named (except the one that I am showing) directories but if I browse to one of those in DxPL all the items will be immediately “ingested” into the database!

So I believe that both the freedom that DxPL offers but with the ability to limit the automatic importation, ingesting, absorbing etc. gives the best of both worlds. I believe that one or both of you has complained about not being able to browse without importing in the past.

I avoid (workaround) the problem by browsing with FastStone Image viewer and then passing a single image to DxPL, loading the directory in DxPL, if, as and when I want to! But it would be useful to be able to browse without any importation and then select to import (for simplicity, probably at the directory level rather than individual images).

Already imported directories can then be marked in the tree (by colour) etc. and I don’t have to use any other viewer etc., if I don’t want to!

@platypus The issue with database placement is something I would find way too restrictive (but it would be a lot harder for me to lose the database at any given time), although with my empty database suggestion I don’t need to destroy the database in the same way that I normally do!!

I understand and share your frustration with respect to the general lack of features for housekeeping but sadly can do nothing about it until someone at DxO starts taking such issues seriously and “hopefully” starts there own “Database & Image Management” topic and invites comments to their proposals @Musashi.

With respect to my suggestion about creating an empty database @John-M that would make it ever so easy to replace your active database with a clean one, in a single selection and restart operation!

I understand that, but as long as DPL adds every image it sees to its database and provides no means to remove the image from the DB without moving it to the trash, DPL sucks up all your kid’s Legos and you don’t want to remove them from that yucky dustbin inside your vacuum cleaner (even though you could)…

Trashing the DB as a regular step works around that issue, but I don’t consider it to be a solution…even though I also trash the DB regularly during tests.

OK. I don’t disagree about the need for improved db management … Seems I wrongly interpreted your comment as suggesting PL should not do auto-import of sidecar/.dop files.

I should have guessed you did not mean that, as it’s already discretional (via Prefs).

John M