PL6 cannot find edits after converting Win 11 disk from MBR to GPT

The reason I gave the comments that I did was that if DxO had done their DOP file and XMP file correctly, the OP would possibly not have this problem in reformatting his disc to a different system.

My comment was not directed at you and was about about the diversion regarding the usefulness of history retention between sessions. That point is completely irrelevant here.

My point is any data that is available in the database should be available in the sidecar as well since the database is “fragile” as shown by the OP who is having to figure out how to hack Photolab or his database to recover his edits.

Point taken.

@jee The use of the Volume Id is a useful part of the DxO mechanism and means that the same images in the same directories can exist on multiple disks and be kept separate. Useful until someone wants to use a backup disk in place of the original disk!?

I have investigated and succeeded in changing the entry in the ‘Folders’ Table I believe, i.e. it was some time and and I did not post that information. It is arguably also possible to change the disk id. but changing the database is easier. I need to test the procedure thoroughly but am very busy today!!

  1. You must backup the database and never work on the sole copy.
  2. You need to identify the correct field being used for the disk now, e.g. locate a simple directory, with a JPG or RAW image, in PL6 on the “new” (was “old”) drive and look at the entry in the ‘Folders’ structure.

This is using

Effectively you need to copy the value of the ‘UniqueId’ from the “new” disk and paste onto the “old” ‘Folders’ ‘UniqueId’ and then save the update with PL6 shut down!!

Then shut down ‘DB Browser for SQLite’, if it asks for updates to be saved the answer yes. Re-open and check that all now appears to be O.K. and then shut down again.

Now re-open PL6 and see if things are as you hope they are and back-up the database but ensure that all old backups are retained just in case.

I will have an opportunity to test and snapshot the steps later today.

PS. I believe that the use of the VolumeId is to prevent DxPL becoming confused by Folder names that can change!

Better procedure

  1. Navigate to am image on the dirve to get an entry in the ‘Folders’ structure
  2. Backup the database ‘ to be Fixed’ (and to be fixed 2 and 3 and…)
  3. Locate the ' to-be-Fixed database in DB-Browser
  4. Select ‘Browse data’ and then select ‘Folders’ where there should be two entries one with the old VolumeId (‘UniqueId’) and the other with the new VolumeId.
  5. Highlight the new VolumeId and Copy
  6. Highlight the old VolumeId and Paste!
  7. Execute ‘Write Changes’
  8. Quit DB Browser.
  9. In PL6 backup the database using the default name or a name of you choice.
  10. in PL6 ‘Restore’ the database and choose the “fixed” database.
  11. PL6 will need to restart and copies the ‘Restored’ to the standard name in the designated location.
  12. On restart PL6 may or may not be what you want!!

ERROR:- Attempting to change the UniqueId causes an error because that field must be unique! That was easy to overcome but I didn’t get the result I was after, i.e. I have got something round the wrong way so will test again later!! The database opened just fine but …

Thanks Bryan,

I spent most of yesterday trying the above, i.e. changing the UniqueId in the PL6 database. It didn’t work for me.

I located the Folders table and the disk in question. It had two entries. The first as an MBR disk, the second as a GPT disk. The second entry’s timestamp was just after I converted the disk to GPT. So that makes sense (first time I used PL6 post my disk conversion). As the previous entry was there, with the UniqueId for the MBR state of the disk, I deleted the new row, leaving the only row for that disk as the UniqueId for the pre-GPT state. All else was equal for both entries (volume letter, label etc).

Using your approach to take a backup - make the change on the backup - restoring this backup (and checking that the new ‘live’ database only had the old MBR UniqueId in it), I then launched PL6 and tried to access the edits for the files. But again none showed up.

Inspecting the database after closing down PL6 again. PL6 had inserted a new row for that disk with the GPT GUID / UniqueId. So, back to square one with two entries for the drive, and the newest entry with the current GPT GUID in it. So concluded that PL6 somehow in its logic tried to match volume/path to the disk and only found one matching and updated the UniqueId to the new GUID …

So not sure fixing this at the PL6 database end will work. Changing GUID/UUID on the PC is also not straight forward, as though it is easy on MBR disks (using cmd line diskpart app) it is far from easy for GPT files (no system tool exists for this and there are checksums etc involved … even if my disk was not a boot disk it is not easy and I could find no way to safely do it).

So I have another avenue I’m trying now … namely to insert a new disk - formatted as MBR then copy the files structure across … then change the volumes / GUID on this MBR info to match the old disk in its MBR state … RAW file data copying in progress … let’s see

@jee My test was an attempt to “fool” the system to use the database entries for 4 subdirectories on my NVME (N:) with “obvious” edits that had been copied to my Z:\ drive (and SATA SSD connected as USB3) as images with NO Dops.

But It didn’t work, I need to make the most of the weather for some DIY so will investigate further later.

My test case is somewhat more complicated because the original drive is still present!

I will try it with two removable drives or between two systems both of which are possible, each machine has an identical disk/directory configuration for the F:\ drive, excluding the VolumeId!

Ok I have no experience here but may I suggest move the existing database away so PL will create a brand new database with the new disk. Then don’t o anything after new database creation but close PL and inspect the database structure. It may be easier to understand and investigate this minimum sized database to find the key points that need to be changed.

Then restore a copy of the old database you’re trying to recover and see if you can make the new GUID work…

@MikeR having a smaller database might help but the problem is actually in a very small part of the database, i.e. in the ‘Folders’ structure!

@jee what controls the old entries is actually the ‘Id’ in ‘Folders’

That is accessed from ‘Items’, the entries for the images that contains the settings, via ‘Sources’

i.e. from

If ‘Sources’ point to an entry in ‘Folders’ that corresponds to the new GPT drive entry rather than the old MFT entry could that work!

I will try to test after “afternoon” tea, Coffee with a scone, Plum jam but no cream!

PS:-

Ignore the above it is definitely not a cunning plan!

I fixed it :slight_smile: Even without Scone and Jam! Many thanks for inspiration, sympathy, insights and support from this thread and older ones on the forum!

The moral of the story is that you MUST ALWAYS enable and use Sidecars! Being dependent on an ever-growing database with thousands of edits over many years’ PhotoLab use is ‘putting all eggs in one basket’ … I learnt that the hard way …

… and back up those Sidecars with your RAW files too.

For those interested here is how I fixed it:

  1. I installed and used DB Browser (SQLite) which is a great tool to access SQLite databases which is what PL6 uses. Using this tool I could view the Folders table and obtain the old MBR GUID for the disk in question (see UniqueId column in the table). (note there was two entries for this disk - searching on drive letter - the old one when the disk was MBR and the new one after I converted it to GTP. I noted down the old MBR UniqueId).

  2. I bought a new 2Tb disk and temporarily installed it in my Desktop PC (I stole the SATA leads for the DVD ROM drive). I configured this disk as MBR and used the same naming label as the GTP disk with the issue.

  3. I then changed the mount letter on the GTP disk with the issue to a new unused one, and changed the mount letter of the new MBR disk to the one previously used by the GTP disk. The last part in ‘spoofing’ the disk was to use the Windows tool Diskpart (using the CMD line interface) and changed the GUID on the new MBR disk to the UniqueId found in the PL6 database. See google on how to change GUID on an MBR disk) [note - do not do this if it is a boot disk! Nor if it is a GPT disk].

  4. Now my new MBR disk looked exactly like my GTP disk before I converted it to GTP. Same GUID, Mount Letter, Label etc … exactly matching the entry in the Folders table in the PL6 database.

  5. I copied the RAW files across in exactly the same directory structure. This took a while … 15,000 files. I then double checked the structure, naming, letter mount and rebooted and checked again. All good.

  6. I launched PL6. Took a backup. Then I entered the directory structure and vola! All my edits were there - as before :slight_smile: For each directory with RAW files I filtered on the edited files and exported the Sidecars. I then copied these Sidecars into the GPT disk in the relevant directories.

  7. I removed the MBR disk and rebooted. Launched PL6 and every edit is there available on the bigger GPT disk. Back to normal.

The lesson is that you cannot fiddle with the underlying disks holding your PL6 RAW files unless you have Sidecars. Indeed the lesson is that you should ALWAYS use Sidecars.

2 Likes

YES!

Glad you got it sorted and sorry it was so painful.

1 Like

@jee glad you finally sorted it out and I agree that without DOPs all the eggs are in one basket which is normally acceptable when the status quo is maintained and regular backups are taken but with DOPs you should be either bang up to date or 20 seconds or thereabouts behind in “auditing” and they are available for major recovery (but no ‘Projects’).

1 Like

The warning really is the only backup strategy for a Photolab involves backing up the DOP sidecars. There is no other effective backup strategy.

Yea exactly … unless you always backup to same disk … but what if the disk blows up? The way PL links database edit references to a UUID, I.e. one specific physical disk, you are in a world of pain without Sidecars. Indeed, it would be almost impossible to recover from a GTP disk failure in this scenario … food for thought DXO!

@MikeR Sorry but I do not agree! Backing up the database is the best strategy for normal issues. It means the least interruption for normal processing, protects ‘Projects’ and means that searches work as well as they normally do (i.e. more work is required for that feature but that is another issue).

Having the DOPs means that in the event of a catastrophe the database can be rebuilt, i.e. it provides a second level of backup, but that rebuild may not be a quick or particularly painless task.

@jee Because DxO are living in the dark ages and don’t provide the necessary commands and design to allow for overcoming the Uuid issue.

My personal strategy involves both multiple copies of the files and the images and the DOPs and the xmp sidecar files and database dumps!?

But recovering the database quickly if I need to move to another system or use one of my USB backup disks will be an issue if I have not established mount points from the beginning.

The mount-point replaces the Volume, i.e. mount-points introduce another element of indirection and all DxO database references are to the mount-point and to that mount-point different disks can be attached at different times!

Good luck to you!!

@MikeR I backup both to multiple machines and USB3 drives giving the option to use database backups in the first instance, i.e. a quicker recovery to within a given state of the database and the DOPs will take over from there or, ignore the database dumps and just let the DOPs recover the data as and when I choose to (re-)discover or (re-)index directories.

Luck should play no part in it!

In other words:

Seems you agree with me. As I said the only backup strategy involves backing up the sidecars which I see is a part of your backup strategy.

It seems we are in violent agreement…

Interesting … so are you saying that if the drive is a USB drive, then it doesn’t worry about UniqueId? And if you then have two different USB drives with the same RAW files on them, the database would still work regardless of which of those drives you connected? So a loss of an internal (local) drive where UniqueId is used as reference is a big recovery problem without Sidecars, but external drives will actually be recoverable without Sidecars? In any case always use Sidecars …

A bit painful … but not as painful as re-editing >1,000 RAW files :-o

@jee I have “spare” USB3 drives both MBR (old SATA SSDs with and adapter) and a 6TB drive so I can test but I believe that just using an external drive actually doesn’t work, i.e. the UUID is there to enable DxPL to recognise an old drive using the number rather than the drive letter!

The problem I investigated was about a user who alternated using a pair of drives to ensure that any disk failure did not go undetected but they were using ‘Projects’ and ran into the ‘image missing’ problem, because DxPL was re-discovering the images when a drive was connected, i.e. it looks as if things are working but DxPL is re-adding images to the database (in addition to the copy that was already there.

Hence when testing various scenarios I frequently create ‘Projects’ to provide a way of checking what I am actually looking at, the original data or another copy in the database!?

During the investigation I learned about the UUID in the ‘Folders’ Table and the only way of “cheating” DxPL is to create a single Directory to which different drives (Volumes) can be attached, i.e. a “Mount Point”.

The ‘Folders’ Table refers to the “Mount Point” which remains “intact” but can have different volumes attached!

The Plus is that my research seemed to show that it worked but the disadvantage is that it needs to be established from the start. There is now a real issue that any ‘Projects’ may refer to “missing” images on the unmounted drive but providing DxPL does not rush to clean up such entries they will re-appear when the appropriate drive is re-mounted.

Arguably DxO should provide a mechanism to “override” the UUID but …

Plus although the mount point helps with USB drives I am not sure how to use it with a combination of Hard drive (normal use) and USB3 drives (for backup), i.e. my USB3 drives are copies of the larger part of my F:\ and G:\ drives and the file structure for my photos are identical!

I could create a mount point on E:\ and see if I can attach F:\ and then replace it with an external USB3 drive! Given I have two backup machines and they have SATA switches to change OS drives I should be able to do this without risking my main machine!

But all this testing takes time and I have decided it is essentially a waste of a valuable resource, namely my time. Given the reluctance of DxO to change things that would be easy to “fix” I have decided to limit (to 0 if possible) my testing, except in this case I am laying the ground work for my own recovery plan!?

Regards

Bryan

PS:- I can confirm that the UUID has the casting vote when deciding whether DxPL has seen a directory before or not. Using two SSDs labelled TEST with an identical directory I

  1. Plugged one into USB and was allocated a drive of Z:, discovered in DxPL, created a project and dismounted the drive
  2. Plugged the other SSD to the same adapter and was allocated a drive letter J:, discovered etc.etc.
  3. Eventually got the original Z: with the drive letter J: and the database shows

and DxPL shows

but

with