PL6 cannot find edits after converting Win 11 disk from MBR to GPT

BHAYT · May 22, 2023, 11:17pm

@MikeR Sorry but I do not agree! Backing up the database is the best strategy for normal issues. It means the least interruption for normal processing, protects ‘Projects’ and means that searches work as well as they normally do (i.e. more work is required for that feature but that is another issue).

Having the DOPs means that in the event of a catastrophe the database can be rebuilt, i.e. it provides a second level of backup, but that rebuild may not be a quick or particularly painless task.

@jee Because DxO are living in the dark ages and don’t provide the necessary commands and design to allow for overcoming the Uuid issue.

My personal strategy involves both multiple copies of the files and the images and the DOPs and the xmp sidecar files and database dumps!?

But recovering the database quickly if I need to move to another system or use one of my USB backup disks will be an issue if I have not established mount points from the beginning.

The mount-point replaces the Volume, i.e. mount-points introduce another element of indirection and all DxO database references are to the mount-point and to that mount-point different disks can be attached at different times!

MikeR · May 22, 2023, 11:24pm

Good luck to you!!

BHAYT · May 23, 2023, 12:15am

@MikeR I backup both to multiple machines and USB3 drives giving the option to use database backups in the first instance, i.e. a quicker recovery to within a given state of the database and the DOPs will take over from there or, ignore the database dumps and just let the DOPs recover the data as and when I choose to (re-)discover or (re-)index directories.

Luck should play no part in it!

MikeR · May 23, 2023, 12:38am

In other words:

Seems you agree with me. As I said the only backup strategy involves backing up the sidecars which I see is a part of your backup strategy.

It seems we are in violent agreement…

jee · May 23, 2023, 8:02am

Interesting … so are you saying that if the drive is a USB drive, then it doesn’t worry about UniqueId? And if you then have two different USB drives with the same RAW files on them, the database would still work regardless of which of those drives you connected? So a loss of an internal (local) drive where UniqueId is used as reference is a big recovery problem without Sidecars, but external drives will actually be recoverable without Sidecars? In any case always use Sidecars …

jee · May 23, 2023, 8:07am

A bit painful … but not as painful as re-editing >1,000 RAW files :-o

BHAYT · May 23, 2023, 8:26am

@jee I have “spare” USB3 drives both MBR (old SATA SSDs with and adapter) and a 6TB drive so I can test but I believe that just using an external drive actually doesn’t work, i.e. the UUID is there to enable DxPL to recognise an old drive using the number rather than the drive letter!

The problem I investigated was about a user who alternated using a pair of drives to ensure that any disk failure did not go undetected but they were using ‘Projects’ and ran into the ‘image missing’ problem, because DxPL was re-discovering the images when a drive was connected, i.e. it looks as if things are working but DxPL is re-adding images to the database (in addition to the copy that was already there.

Hence when testing various scenarios I frequently create ‘Projects’ to provide a way of checking what I am actually looking at, the original data or another copy in the database!?

During the investigation I learned about the UUID in the ‘Folders’ Table and the only way of “cheating” DxPL is to create a single Directory to which different drives (Volumes) can be attached, i.e. a “Mount Point”.

The ‘Folders’ Table refers to the “Mount Point” which remains “intact” but can have different volumes attached!

The Plus is that my research seemed to show that it worked but the disadvantage is that it needs to be established from the start. There is now a real issue that any ‘Projects’ may refer to “missing” images on the unmounted drive but providing DxPL does not rush to clean up such entries they will re-appear when the appropriate drive is re-mounted.

Arguably DxO should provide a mechanism to “override” the UUID but …

Plus although the mount point helps with USB drives I am not sure how to use it with a combination of Hard drive (normal use) and USB3 drives (for backup), i.e. my USB3 drives are copies of the larger part of my F:\ and G:\ drives and the file structure for my photos are identical!

I could create a mount point on E:\ and see if I can attach F:\ and then replace it with an external USB3 drive! Given I have two backup machines and they have SATA switches to change OS drives I should be able to do this without risking my main machine!

But all this testing takes time and I have decided it is essentially a waste of a valuable resource, namely my time. Given the reluctance of DxO to change things that would be easy to “fix” I have decided to limit (to 0 if possible) my testing, except in this case I am laying the ground work for my own recovery plan!?

Regards

Bryan

PS:- I can confirm that the UUID has the casting vote when deciding whether DxPL has seen a directory before or not. Using two SSDs labelled TEST with an identical directory I

Plugged one into USB and was allocated a drive of Z:, discovered in DxPL, created a project and dismounted the drive
Plugged the other SSD to the same adapter and was allocated a drive letter J:, discovered etc.etc.
Eventually got the original Z: with the drive letter J: and the database shows

and DxPL shows

but

with

JoPoV · May 23, 2023, 11:04am

If I understand well, this prevents the use of a backup when database is used ?
How can this be conceivable ? No backup possible (and worst : backup created but not usable !) ?

Tell me I’ve missed something … This can’t be possible ? This would be totally amateurish and deconnected from reality …

BHAYT · May 23, 2023, 12:41pm

@JoPoV I believe it is correct from the work I have done in the past and just repeated above!

Please note that I might be wrong!

I have started testing Mount Points but made the fatal mistake of closing a mounted drive without first closing (removing) the mount point!! However when I got that right it seems to work as I expected (and previously tested).

But using Mount points needs to be done before discovering the images via the mount point, i.e. with Mount Points the mounted drive can be accessed directly or via the Mount Point but only if accessed via the Mount Point, which can then be used by another drive, can drives “share” a single UUID in the database!

The alternative is that the database has to be “rebuilt” using DOPs but any ‘Projects’ will be lost as will any data that was in the database but not “flushed” back to the DOP (an unlikely occurrence if DOPs are being maintained) or back to the image in the case of the metadata (determined by the metadata preferences etc,).

JoPoV · May 23, 2023, 3:26pm

I thought that the backup was one of the essential pillars for a professional work …
But a backup that does not work is the worst trap imaginable …

That can ruin a production, and more …

BHAYT · May 23, 2023, 3:57pm

@JoPoV Presuming that I am correct about the way that it works, which makes sense from the normal running perspective, namely

Each image, even if added multiple times will be treated as independent, i.e. some software seems to treat such “Duplicates” as “versions” but DxPL does not do that, they are all independent!
DxPL can handle different drives being used that may have the same title and Drive letter as a previously used drive without confusion.

but

this requires a unique identifier that can be trusted and DxO have chosen the Volume ID which it treats as a UUID.

However, this UUID is essentially the topmost node of a “tree” that then has images at the leaves and the only way to those leaves is via the UUID. Not completely true because searching goes through the images (‘Items’) looking for matches of one sort or another but the principal method of matching a found image on disk with an image already in the database (or not) is via the “tree”.

But we now have the issue of what happens if the data on disk becomes unavailable for one reason or another?

In my case the directory structure on my backup disks, accessible via the LAN or via a USB3 port are identical to the structure of the original images but on a different disk with a different UUID! The DOPs are exact copies, the xmp sidecar are exact copies and the images themselves are also exact copies and, hopefully, reasonably or completely up-to-date but I cannot simply get DxPL to accept those images as replacements because of this one field in the database!

For other users things might be even more complicated and the backups may exist but not with exactly the same directory structure and how that could be handled I am not really sure!

Realignment for my own particular case is simple “just” change the UUID but for others …

I could use my favourite comment when it comes to many things about DxPL infrastructure handling, i.e. it is a simple job to fix the problem and create a much more user friendly product but my comments have fallen on deaf ears for some years now and I am bored with the lack of response from DxO. I am aware and thankful for the new releases that are coming out for both DxPL5 and DxPL6 but …

JoPoV · May 23, 2023, 4:26pm

In other programs I use, this is called content directory (or other terms). You can move all your project to an other place and only have enter the new content directory for all related files can work together again (and there are a lot (!!!) of files, called by a lot (!!!) of different parts of the project … The begining of the path (content directory - easily editable by the user) is concatenated the other part of the path (the relative path - contained any sort of database). And that’s it. No backup problem, no project relocation problem.

THIS IS BASICS !!! and works like this since more than 30 years with softwares I use ! … Not a really new way of doing things, isn’t it ? no need to be houdini to do that !

Lot of details like this, solved since a while in other softwares, aren’t in photolab. It looks like they leave on an other planet.

I think the same : content directory ! AND database saved with project and not in a user space defined by windows.

I could elaborate, but have not time for this.

MikeR · May 23, 2023, 5:26pm

In the end, everyone should be generating sidecars as this is the only solid container for your edits that can be backed up and restored without much thought.

In my mind the database only serves as a cache for the information in the sidecar. Of course the fault is certain things like project membership is only stored in the database. So if someone experiences a drive failure, aside from spoofing the old HD’s UUID onto the new disk which I assert is beyond the capabilities of the average photographer, that project information is gone regardless.

Again. In my mind the database is a cache only to speed performance of the tool. The sidecar is the reference…

Perhaps a strong opinion, but in light of this discussion I believe it’s a fair assessment.

jee · May 23, 2023, 5:51pm

You are right it is not straightforward in many cases to change UUID. If your disk is GPT then it is a complex long hex string with no ready tool to change it. Also, if that disk is used for other important applications or as a boot disk - which uses the UUID - then you have further issues. This is not something you should attempt. So there is no database backup solution for a blown up internal hard disk, unless you happen to be MBR (old type PC disk cfg limited to 2Tb) and you’re prepared to get a bit techy. I totally agree that DXO should consider how to improve this. Photo editing takes hours and hours and losing the edits is potentially a reason to change editing software … you’re back to square one anyway …

maderafunk · May 23, 2023, 6:28pm

Do I understand well that there is no way to transfer the project structure to another computer if I want to switch?

JoPoV · May 23, 2023, 6:28pm

But it does not contains everything. Not projects …
Anyway, work half done is not done !!!

JoPoV · May 23, 2023, 6:36pm

This is what I understand too. Informations in the database, can only act on only one drive. If this drive has a problem, even a backup does not work. You have to tweak database file the hard way - need to like going into and modify ascii (I hope) lines of codes.
There is no problem with informations in sidecars (.xml, .dop).
I think you can manage to have only projects in the database, and other relevant information in side-cars.

maderafunk · May 23, 2023, 6:40pm

Their project structure is still next to useless for me until they allow manual sorting, but before they add any new features, DxO should really make it a priority to implement a proper backup strategy. At the moment, noone can seriously recommend using ther projects. This is baffling, as they have even added some new features to the projects with PL6. In retrospective one must be happy that the projects are next to useless right now, otherwise many would have done a lot work organizing their files for nothing.

MikeR · May 23, 2023, 6:54pm

That’s a real problem but currently unless you want to hack a replacement drive’s GUID there is no way to restore project information to another disk…

Anyway there is another topic which is about being able to use PL across multiple systems and in that I suggest attaching the project information to the sidecars is my preferred option…. Feel free to vote for it.

platypus · September 11, 2023, 8:03am

Came across this (elderly thread) by chance and had a look into DPL’s database on my Mac.

All volumes encountered by PhotoLab seem to be added to the database, even temporary items like the “Install Nik Collection” drive and other drives that I’ve removed quite a while ago.

The good news here is, that the listed unique IDs (rightmost column show) are the IDs that macOS has assigned to the respective drives. Replacing these IDs in the database should reroute DPL to the changed location, unless other hooks are used in other tables of the DB (have not checked for this).

If I were in the situation of the OP, I’d backup the DB and edit the ID and drive name fields.
I’d probably also remove the deleted volume’s records, hoping that they are only stored in this table.

But again: DxO should make PhotoLab sustainable by adding the necessary capabilities to (manually) fix lost relations, automatically create (compressed) DB backups etc. Corresponding feature requests are available - and have been for a while.

Caveat: If other locations (folders) have changed names, these locations need to be edited too.