Database hell between PL4 and PL5

I’ve been using PhotoLab since OpticsPro6.
Yesterday I installed the trial of PL5, to test alongside my existing PL4 install.
Everything seemed fine yesterday, but I’m getting database errors when I switch between the two products; and none of my previous edits are visible in PL4 now.
This is the message I get in PL4:


This is what I get in PL5:

Having looked at the preferences it appears that PL5 uses the PL4 directory by default :frowning:
Why would it do that?! Is there any way I can get my PL4 edits back, without having to restore a backup?

Starter:
If you use DPL 4 and 5 in parallel, you should switch off automatic export of settings files in DPL5! Why? .dop sidecars written by DPL5 cannot be understood by DPL4!

Fix:
If DPL5 uses the same database directory as DPL4, something has gone wrong with DPL5’s installation. Be sure to keep the database that has been backed up by DPL5. Restore that backup in DPL4. This should get you up and running again in DPL4.

Check the location of DPL5’s database and change it to something different than what DPL4 used. Afaik, you can set the location in DPL(win) settings. Restart DPL5 and restore the database that it had previously backed up. This works on Mac and I suppose that it should work on Win too…

Backups are essential to fixing such issues. Luckily, DPL can back up and restore its databases, no need to restore a PC-backup.

att: @StevenL

Hello @AdrianW and welcome to the forum!

This is really strange. Could you, please, provide us with this folder - %LocalAppData%\DxO ? Please, upload it here - upload.dxo.com with your forum name and let me know when ready.

Thank you
Regards,
Svetlana G.

Database what database (or should that be which database)!?

I had the same problem when I installed the PL5 Trial on my main (Win 10) machine! I keep the “catalogs” created by the various bits of photo editing either on the F: drive (HDD) or the E: drive (Sata SSD).

PL databases reside on the E drive in an appropriately labelled directory. The PL5 installation caused the same problems identified by AdrianW ; PL5 took over the PL4 database and upgraded it (allegedly) and when I ran PL4 (still my “production” version) I ran into the same database incompatibility problem.

I just copied one of the databases (PhotoLab_1520-2.db) in the PL4 folder back as the main (PL4) database and it seems to have restored things as they were for PL4 (albeit I was not concerned about the loss at this time).

However, looking at the PL5 directory it was clear that the PL5 database was too small so I repeated the exercise by copying the newly re-purposed PL4 database as the PhotoLab.db database in the PL5 directory and hung PL5 on startup (or so I thought)!

I then used the PhotoLab_1520-1.db (copied as PhotoLab.db) in the PL5 directory and PL5 started correctly but simply made the warning and threw away that database and created a new one!!

This does not auger well for those that use PhotoLabs as the custodian of their valuable keyword data!

I then re-read platypus post and restored PhotoLab_1520-1.db in PL5 and selected restart and it “hung”; sorry in the time that I was “fiddling” with this post the “hung” PL5 continued to work and I finally had the PL4 contents in a PL5 database!

Notwithstanding the fact that the situation can be recovered it should not be necessary.

PL5 installation should have an option to select a new location for the database and an option to import an existing database into that location, performing the upgrade as required.

So I think that the situation goes like this;

  1. PL5 locates its database in the same location as PL4 on installation by default (I know of no option to change this default action).

  2. PL5 warns, creates a backup copy in that location and creates a new PL5 database.

  3. I then switched PL5 to use its own directory but I am not sure what I copied to the new directory at that time but there is a recovery database in that folder!?

  4. Running PL4 will object to the PL5 database that it finds and will revert to a new PL4 database.

  5. At this point it should be possible to restore the PL4 database using the restore option (selecting the PL5 created backup, which should be in the same directory)

  6. In PL5 restore the original PL4 database from the backup created by PL5 when first installed and run, at which point PL5 will transcribe the data from one database to another! Don’t panic when it appears to be hung (like I did), the bigger the database the more work it has too do!

The issue of incompatible DOPs is just that and photo processing needs to be carefully managed while there is more than one version of PL or DxO installed!

I have my database on a different drive D: one that’s backed up overnight. V5 created the data base back at the default C: and I had to move it back to D: This is worrying, if the install ignores where the database is actually installed then the one it uses for the new one will be a leftover one (whey when you move the database that’s not what happens is beyond me) but it looks like it copies the database over and leaves the original in the default location. I am pleased I do not use PL for DAM it’s a mess and the database handling is not very good and still lacks any database tools to check/repair it.

1 Like

John7 as I stated in my post above this did not happen to me in quite the same way. The installation defaulted to the PL4 location and appropriated the database as I stated in the post.

So between us, and I suspect a lot of other users, we have “found” most(!?) of the issues with the installation process, although I am still puzzled why your installation did not default to your PL4 location.

One potential difference is that I have never used the backup and restore options (until I used the restore option today!) but I cannot see why that should change things!!??

The database housekeeping certainly needs attention, particularly if it is actually used as the sole repository if it is used for some or all keywords.

I tried a test as follows;

  1. Add a keyword ‘B1’, ensure that is in PL5.
  2. Close PL5
  3. Copy database (I would now use the ‘Create backup’ option)
  4. Restart PL5 and add another keyword ‘B2’. At this point the PL5 DB and the DOP have both keywords.
  5. Close PL5 and capture the database.
  6. Revert the database to the B1 version.
  7. Restart PL5
  8. Look to see if there is any PL5 reaction to the DB to DOP keyword Mismatch - None
  9. Explicitly import the DOP and check.
  10. Somewhen the DOP was changed to correspond to the database.

The DOP offers a potential source of recovery data but the process needs to be managed and an automatic process could do way more harm than good.

I don’t use the backup and restore either, the only time I have had to restore a backup years ago I used my backup program. As I don’t use PL as a DAM or projects its no loss to me to lose the database as the dop’s will, as used, rebuild it. The install for 5 was odd as it added all the keywords (except a few) from Photo Supreme but doesn’t display any terms added with Photo Supreme from the txp files. Photo Supreme is keen on regular manatance and nag’s you if it’s overdue. For me the advantage of PS is the searching, I am dyslexic so just being able to drag terms, locations and dates into a search box is great

John7 I was sorry to hear that you are dyslexic, my oldest son is also dyslexic and red/green colour blind but I am glad to say that hasn’t held him back in his chosen career as an architect (with a company designing some of the “smaller” high rise office buildings in London)!

I haven’t used keywording extensively nor do I use a DAM but worse than that the “ancient” program that I did use for allocating some keywords (ExifPro) really does not get on at all well with PL5! Hopefully it is the only software that has any problems.

With regard to your post about the PL5 install being “odd” I would be interested to know why you considered its behaviour as “odd”, i.e.

  1. What was already in the PL4 database with respect to keywords?

  2. What type of images were involved, jpg, DNG, RAW (+sidecar)?

  3. What actually happened, your statement seems to imply that during the installation PL5 actually added keywords to the database that could only have come from the photo files (and/or XMP sidecar) files?

    In fact regardless of the ‘Auto’Sync’ option in the ‘Preferences’ menu PL5 will automatically detect keywords in any files in folders that you browse while you are browsing. The PL5 database entry for the photo will be updated with the keywords, as will the ‘Keywords List’.

    If you use the ‘Index a Folder’ option then all directories contained within the chosen directory and all associated subdirectories will be indexed. I just set that running on my main machine and it seems to work OK.

    However, while searching for keywords in the group of 5,122 photos was quick, PL5 seemed to spend a long time checking the ‘Camera model’, ‘Lens model’, ‘Manufacture Name’, ‘File Name’ and ‘Folder Name’ for the search item and I was not interested in these other data fields. I do not know how I can exclude these additional items from the search!?

    Moving up an down the list of items found and presented by PL5 seems to send PL5 off searching for the word in the ‘Camera model’ etc all over again which pushes the I/O “through the roof”.

  4. The “except a few” is also intriguing; is there anything that ties these items together in some way.

  5. I am not a Photo Supreme user and certainly did not understand the statement “but doesn’t display any terms added with Photo Supreme from the txp files”. What is a ‘txp’ file and how would such data be visible to PL5, i.e. how would that data have been added to the photo file or associated sidecar file?

Sorry this isn’t meant to be the ‘Spanish (or any other) Inquisition’ I am just interested (nosey)!! I actually got to play with the indexing facility and the searching as a result!!

Hi
As far as I know there wasn’t any keywords in PL4, I only keep RAW with there sidebars in the folders. As far as I can see PL5 install added the majority of the keywords in use in Photo Supreme 5. The keywords list has 0 for all bar one keyword where I did use in in PL5. The only keywords shown for each RAW image is the folder name its in no others that are in the Photo Supreme 5 xmp (not txp my typo) file.
I have never indexed a folder in PL as I am using another DAM and don’t want to confuse things. Though since PL5 others have said and I can see why having two programs acting as DAM’s will lead, something some of the DXO staff have agreed with.
I had expected PL5 to read the keywords (as it must have done from somewhere to populate the keyword list)from xmp files as Photo Supreme does.

Glad your son found something that worked out with dyslexia, my oldest son with a degree of dyslexia went into programing, in satellite communication. Again it has worked out well for him.

I have now noticed that looking at the keyword l;ist in PL5 I had two entries for plants with sub entries for both. In Photo Supreme I have sorted it but this hasn’t been reflected in PL5. It must have been the install program that dug out the keywords fro?

PL5 caught in a “Corrupted Database” loop!:-

The reason that the problem occurred in the first place was because of my “adventurous” spirit. While responding to a post yesterday I indexed some 5992 photos and then started searching them and I was very disappointed with the search performance of the PL5.0 Trial Release!

So I tried the same on the last PL5.0 beta on my other machine and it was near instantaneous! I moved to PL4 and repeated the indexing etc. and the searching was also excellent!

I backed up my databases using the backup command and then did something very “stupid” I attempted to import the beta test database across the LAN into the Trial copy on my other machine!!

Needless to say it ran into problems and hung!! PL5 had to be terminated with prejudice and when I ran it again it complained about a corrupt database and stated it would create a new one but this then repeated on the restart as it marked every database that it had just created as corrupt and then made another and another and…

So PL5.0 was uninstalled from my main machine, a new copy downloaded (not really necessary but …

The new copy was installed and started and … off we go round the corrupted database loop again and again and …

I was originally going to ask for assistance!

However, before transmitting this post I renamed the ‘Database’ directory and created a new one, with no database of any description in it and PL5 started successfully, creating a new empty database in the process. As far as I can tell the new database created was the same as the ones it kept rejecting as corrupted (I need to check that) or simply having an empty directory took PL5 down a different path!?

Using this new empty database I re-indexed the Beta test photos directory and repeated the search “test” and it now works as it had with the Beta PL5 and PL4 but the original database was an import from PL4 so…

I restored the last backup from PL4.0 and “panicked” again because I thought it had “hung”. I would like to suggest that showing a “Converting” and/or an 'Importing" and/or a “Busy” message/icon would help the user to be aware that all is as expected!!

I re-indexed the Beta test photos, I feel that replacing the left to right icon with a progress bar icon would be useful, adding the left to right busy icon would be useful for “Converting”/“Importing” but that would also be better served with a progress bar icon as well.

The tests resulted in instantaneous search results for all categories whereas the “incident” I had experienced showed progress icons while PL5 seemed to spend a long time checking the ‘Camera model’, ‘Lens model’, ‘Manufacture Name’, ‘File Name’ and ‘Folder Name’ for the search item and I was not actually interested in these other data fields. I do not know how I can exclude these additional items from the search!?

Moving up an down the list of items found and presented by PL5 seems to send PL5 off searching for the word in the ‘Camera model’ etc all over again which pushes the I/O “through the roof”.

What caused the original problem I do not know but it appears to have disappeared!?

What is that database doing there (another “wrinkle” on the “Database hell” issue)?:-

During the corrupted database loop that I reported in the earlier post I went looking for databases that might be being picked up by PL5.

AdrianW had stated that PL5 started using the PL4 database directory after installation and while that was the same for me, John7 stated that PL5 had reverted to the default directory!?

What I found when I did my checking for other copies of a PL5 database was one in the default directory that was the same size and the same date and time as one of the of one of the databases (PhotoLab_1520-2.db) in the PL4 directory on my E: drive and a ‘Beyond Compare’ comparison shows the files to be identical!

It looks a bit like PL5 was starting to upgrade the PL4 database and copied the PL4 database to the default directory (from my E: drive to my C: drive), then “lost interest” or discovered that was not where it was supposed to be and created a new database over the PL4 database in the “correct” location for the PL4 database.

However, why there are so many database files of the form PhotoLab_1520-2 in the PL4 directory on the E: drive I simply do not know!? Hopefully someone from DxO will enlighten us.

I had the same problem during PL5 installation. Moreover, I have lost all the keywords that I had entered with PL3 and PL4, and I have lost the projects too.
The strange thing is that PL5 installation created “C:\Users\user name\AppData\Roaming\DxO\DxO PhotoLab 5\Database” but was configured to use "C:\Users\user name\AppData\Roaming\DxO\DxO PhotoLab 4\Database. I checked that PL5 created the files in the wrong directory at launch before changing that.
The point is: I can’t find the key words with both PL4 and PL5, but I can find them with PL3 (which I still have).
Now I will wait a while before entering new key words…

rsp I am sorry to hear about your plight. I believe that some of us encountered problems because we had configured PL4 to use directories other than the default.

If you actually specified the location that you have indicated then you were using the default, either by design or by accident. Hence, in your case by using the default settings/naming for the location of the database PL5 was able to continue the “normal” naming convention and create the “C:\Users\user name\AppData\Roaming\DxO\DxO PhotoLab 5 \Database”.

However, PL5 should not then have destroyed the PL4 database, that should have remained intact in the PL4 directory! Even if PL5 didn’t successfully carry over the data into the new structures in the PL5 database, i.e. it just started with an empty PL5 database, the PL4 database should have remained intact!?

I believe that the complication for those of us using designated directories is that PL5 could not apply the “normal” naming rules so it simply used the PL4 directory and overwrote the database. This could have been avoided by DxO using a simple naming convention for the different generations of database(when all generations could have happily co-existed in the same directory if required) or by providing a proper upgrade procedure that asked for the name of the directory to be used.

However, this does not help you with your problem, neither does it explain why you suffered problems when you were not locating the database in some “obscure” location.

If you look at the first picture in my post just above you will see files identified as PhotoLab_1520-2.db, for example. I have a number of these in that directory and no idea why there are so many!? But by looking at them I “guessed” that the largest was probably the final PL4 database, which was actually the case and then successfully imported it into PL4.

Do you have any such files in your PL4 or PL5 directories or anywhere on your system?

If you do then it is possible you may be able to recover your PL4 data by importing into PL4/PL5 as I outlined in an earlier post. Please copy the entire directory somewhere safe before attempting any sort of recovery, if you are “lucky” enough to have any of the files I have described. Then start using the database backup commands whenever possible.

Unfortunately the current naming convention for backup copies only includes the date in the title, it would be good if it also included the time to avoid duplicate issues if more than one backup is performed per day!

I was lucky that I could recover my data, albeit I didn’t have anything in the database that I needed, the DOP files cover my needs adequately.

The PL5 DOP sidecars also contain the keywords but that will only help in the future if DxO provide a facility to import that data into the database and is of no help to you because that feature has only just been released.

I just ran a test importing a PL3 database into PL5 using the ‘File’ ‘DxO PhotoLab database’ ‘Restore a Dxo OpticsPro/PhotoLab backup’ and it appeared to work successfully!

Hopefully DxO will read these posts and react to prevent any such damage to other users and possibly help those who have suffered data loss.

It is late here in the UK. bye for now.

I had exactly the same problem and error messages. I found that the search path to the database was wrong under Edit - Prefereces - DxO Photolab Location.

The right path should be C:\Users\Sten-Ake\AppData\Roaming\DxO\DxO PhotoLab 5\Database of some reason it had been changed to C:\Users\Sten-Ake\AppData\Roaming\DxO\DxO PhotoLab 4\Database instead.

Since I use Photo Mechanic as a metadata editor and catalog instead of Photolab PhotoLibrary I also had to change all the references there (and they are quite a few for the Edit-funkction and all the references for preferred editor for all the different filetypes like DOP, JPEG, ARW (my RAW), TIFF and DNG.

Good luck!

Thanks for your message.
Well, that’s a lesson for me, actually, more than one :grinning:.
First: now I will always save the DB each time before upgrading and from time to time during the year. I was very confident in the upgrade process (it’s the 9th upgrade from OP7 to PL5, plus VP and FP upgrades) but anyone can make a mistake.
Second: I have to think about how I want to organize my key-words (hierarchy etc.) before I start attributing them to my images again.
I’ll rebuild the two projects that have been lost, I’m an amateur, these were rather small (less than 50 images), so it’s not a disaster, I’m just unhappy.

You can never have too many backups:

rsp I’m “glad” that the recovery task is not too big.

I currently copy (synchronize) my F drive, containing all my personal files (including photos and any associated sidecar files etc. and also the catalogue/thumbnail files files of a number of other photo editing packages) to multiple systems and backup drives at least once per week.

However I currently only backup my Email pst files from my E drive and periodically clone the C & E drive which were partitions of a single 240GB SATA SSD until I attempted to install PL5 when my C drive ran out of space for the last time and I cloned C and E to two separate 240GB SATA SSDs.

For speed the DxO database resides on the E drive (a SATA SSD) and was not being backed up, that will now change.

  1. At least daily use the backup facility in DxO to keep a history of updates to the PL database.

  2. Before installing any release (major or minor) or using a PL feature new to you take a backup with PL shut down (possibly unnecessary with daily backups) and secure (copy) the database directory (wherever it is located) to a backup location. In my case securing from the E drive to the F Drive which will then be backed up.

  3. On a regular basis secure the entire database directory to a backup location (in my case E to F) to create backup copies (with PL shut down).

  4. After securing the database directory, cull excess copies of the database backups otherwise things are going to get way too big (in my case that would mean removing from the E drive and, at my discretion from the F drive and the other backups as and when, my backups are created using Beyond Compare so they are synchronised copies of the F drive etc.)

So if it moves back it up, if it is just standing still back it up, if it is the day after yesterday back it up, basically you cannot have too many backups except for the space they occupy of course!!

Hello,
that was on my wishlist some time ago to implement a function similar to LR*s catalog function
image

Not a big challenge for developer, but big relief for user.

Guenter

Yes, and the LR (5.7) backup automatically gets a ‘time stamp’,
which makes it very easy to pick up the right one if needed.

Screen Shot 10-28-21 at 12.37 PM

Hi Wolfgang,

that’s correct und thanks for reminding me to delete some old backups. :smiling_face_with_three_hearts:

This kind of backups are the best…you forget that they are here nad the work on the fly

Guenterm Sadly I do not read German but could not find this screen in LR Classic until I shut down Lightroom!

and the options offer

In addition under ‘Edit’/‘Catalog Settings’/‘General’ the following options are available that would simplify the procedures for backup.

Backup Catalog:

  • Never
  • Once a month, when exiting Lightroom
  • Once a week, …
  • Once a day, …
  • Every time Lightroom exits
  • When Lightroom next exits