Database hell between PL4 and PL5

I had the same problem during PL5 installation. Moreover, I have lost all the keywords that I had entered with PL3 and PL4, and I have lost the projects too.
The strange thing is that PL5 installation created “C:\Users\user name\AppData\Roaming\DxO\DxO PhotoLab 5\Database” but was configured to use "C:\Users\user name\AppData\Roaming\DxO\DxO PhotoLab 4\Database. I checked that PL5 created the files in the wrong directory at launch before changing that.
The point is: I can’t find the key words with both PL4 and PL5, but I can find them with PL3 (which I still have).
Now I will wait a while before entering new key words…

rsp I am sorry to hear about your plight. I believe that some of us encountered problems because we had configured PL4 to use directories other than the default.

If you actually specified the location that you have indicated then you were using the default, either by design or by accident. Hence, in your case by using the default settings/naming for the location of the database PL5 was able to continue the “normal” naming convention and create the “C:\Users\user name\AppData\Roaming\DxO\DxO PhotoLab 5 \Database”.

However, PL5 should not then have destroyed the PL4 database, that should have remained intact in the PL4 directory! Even if PL5 didn’t successfully carry over the data into the new structures in the PL5 database, i.e. it just started with an empty PL5 database, the PL4 database should have remained intact!?

I believe that the complication for those of us using designated directories is that PL5 could not apply the “normal” naming rules so it simply used the PL4 directory and overwrote the database. This could have been avoided by DxO using a simple naming convention for the different generations of database(when all generations could have happily co-existed in the same directory if required) or by providing a proper upgrade procedure that asked for the name of the directory to be used.

However, this does not help you with your problem, neither does it explain why you suffered problems when you were not locating the database in some “obscure” location.

If you look at the first picture in my post just above you will see files identified as PhotoLab_1520-2.db, for example. I have a number of these in that directory and no idea why there are so many!? But by looking at them I “guessed” that the largest was probably the final PL4 database, which was actually the case and then successfully imported it into PL4.

Do you have any such files in your PL4 or PL5 directories or anywhere on your system?

If you do then it is possible you may be able to recover your PL4 data by importing into PL4/PL5 as I outlined in an earlier post. Please copy the entire directory somewhere safe before attempting any sort of recovery, if you are “lucky” enough to have any of the files I have described. Then start using the database backup commands whenever possible.

Unfortunately the current naming convention for backup copies only includes the date in the title, it would be good if it also included the time to avoid duplicate issues if more than one backup is performed per day!

I was lucky that I could recover my data, albeit I didn’t have anything in the database that I needed, the DOP files cover my needs adequately.

The PL5 DOP sidecars also contain the keywords but that will only help in the future if DxO provide a facility to import that data into the database and is of no help to you because that feature has only just been released.

I just ran a test importing a PL3 database into PL5 using the ‘File’ ‘DxO PhotoLab database’ ‘Restore a Dxo OpticsPro/PhotoLab backup’ and it appeared to work successfully!

Hopefully DxO will read these posts and react to prevent any such damage to other users and possibly help those who have suffered data loss.

It is late here in the UK. bye for now.

I had exactly the same problem and error messages. I found that the search path to the database was wrong under Edit - Prefereces - DxO Photolab Location.

The right path should be C:\Users\Sten-Ake\AppData\Roaming\DxO\DxO PhotoLab 5\Database of some reason it had been changed to C:\Users\Sten-Ake\AppData\Roaming\DxO\DxO PhotoLab 4\Database instead.

Since I use Photo Mechanic as a metadata editor and catalog instead of Photolab PhotoLibrary I also had to change all the references there (and they are quite a few for the Edit-funkction and all the references for preferred editor for all the different filetypes like DOP, JPEG, ARW (my RAW), TIFF and DNG.

Good luck!

Thanks for your message.
Well, that’s a lesson for me, actually, more than one :grinning:.
First: now I will always save the DB each time before upgrading and from time to time during the year. I was very confident in the upgrade process (it’s the 9th upgrade from OP7 to PL5, plus VP and FP upgrades) but anyone can make a mistake.
Second: I have to think about how I want to organize my key-words (hierarchy etc.) before I start attributing them to my images again.
I’ll rebuild the two projects that have been lost, I’m an amateur, these were rather small (less than 50 images), so it’s not a disaster, I’m just unhappy.

You can never have too many backups:

rsp I’m “glad” that the recovery task is not too big.

I currently copy (synchronize) my F drive, containing all my personal files (including photos and any associated sidecar files etc. and also the catalogue/thumbnail files files of a number of other photo editing packages) to multiple systems and backup drives at least once per week.

However I currently only backup my Email pst files from my E drive and periodically clone the C & E drive which were partitions of a single 240GB SATA SSD until I attempted to install PL5 when my C drive ran out of space for the last time and I cloned C and E to two separate 240GB SATA SSDs.

For speed the DxO database resides on the E drive (a SATA SSD) and was not being backed up, that will now change.

  1. At least daily use the backup facility in DxO to keep a history of updates to the PL database.

  2. Before installing any release (major or minor) or using a PL feature new to you take a backup with PL shut down (possibly unnecessary with daily backups) and secure (copy) the database directory (wherever it is located) to a backup location. In my case securing from the E drive to the F Drive which will then be backed up.

  3. On a regular basis secure the entire database directory to a backup location (in my case E to F) to create backup copies (with PL shut down).

  4. After securing the database directory, cull excess copies of the database backups otherwise things are going to get way too big (in my case that would mean removing from the E drive and, at my discretion from the F drive and the other backups as and when, my backups are created using Beyond Compare so they are synchronised copies of the F drive etc.)

So if it moves back it up, if it is just standing still back it up, if it is the day after yesterday back it up, basically you cannot have too many backups except for the space they occupy of course!!

Hello,
that was on my wishlist some time ago to implement a function similar to LR*s catalog function
image

Not a big challenge for developer, but big relief for user.

Guenter

Yes, and the LR (5.7) backup automatically gets a ‘time stamp’,
which makes it very easy to pick up the right one if needed.

Screen Shot 10-28-21 at 12.37 PM

Hi Wolfgang,

that’s correct und thanks for reminding me to delete some old backups. :smiling_face_with_three_hearts:

This kind of backups are the best…you forget that they are here nad the work on the fly

Guenterm Sadly I do not read German but could not find this screen in LR Classic until I shut down Lightroom!

and the options offer

In addition under ‘Edit’/‘Catalog Settings’/‘General’ the following options are available that would simplify the procedures for backup.

Backup Catalog:

  • Never
  • Once a month, when exiting Lightroom
  • Once a week, …
  • Once a day, …
  • Every time Lightroom exits
  • When Lightroom next exits

Hi Bryan,
I have an old Version 6.14, but that’s exactly what I see in KatalogEinstellungen


I have done the setting 6 years ago, and never touched…
At the moment I don’t know if it was a question or a stement by you

Guenterm It was a statement that was also a question and thank you for responding. I am not a Lightroom user and will probably not become one!

I am a hobby photographer that takes lots of photos of our garden and the gardens we visit, which we normally did for 2 x 1 week holidays in different parts of the U.K. before …

When on holiday I would have to take photos regardless of the weather (rain typically excluded) and light, position of the sun etc. and needed a product like DxO to make rapid “improvements” to the JPG photos I took (starting with free copies of OpticsPro with the deprecated OpticsPro 7 Smart Lighting my favourite feature). I only started taking JPG+RAW in 2018 so from 2003 to early 2018 all photos are JPG only!

The bulk of my photos have been taken on Panasonic Bridge cameras and an LX7 but from 2017 I moved onto the Panasonic 4/3s cameras and added an Olympus EM1 MkII about 1 year ago and a second hand Panasonic G9 replaced my G90 some months ago.

Hi Bryan,
you are welcome…and I’m also a hobby photographer. LR was my first software for Raw and JPEG development, and my first step for cataloging, kewording DAM stuff…I stopped working with LR after they have gone to cloud version.

Let’s wait and see what comes :grinning:

Yeah about the gardens in England I have heard a lot of intersting stories, or seen some documentation on german TV.

So always good light an a lot of intersting motives

Sunny greetings from Germany

Guenter

Guenterm Thank you for the sunny greetings it is currently very overcast here.

We have made two trips out this month hoping for some Autumn colour but it has been rather disappointing. Some trees simply have withered leaves and others do not seem to be changing colour as fast as they have in the past and the weather has now become very unpredictable!

Good news!
We had this afternoon a teamviewer session with a person from the support. It is solved: my projects and key-words are back!!!
Kudos to the support team!

2 Likes

@rsp that is really good news I am glad the DxO support team were able to help.

Please remember to take and secure backups, just in case!

In addition it is a good idea to keep a journal noting dates and times (and names) of backups, major keywording exercises, PL (and any other software you use on your photos or their metadata) releases etc.; effectively a simply “audit” trail so that it is possible to assess what has been lost if you need to fall back to a database dump.

I encountered a “bug” today which was actually something I did last December, i,e, it was obviously me experimenting and an unusual action for me to take and I do not remember the whys and wherefores at all! A simple journal/diary would have helped!!

You’re right, I used to do that at work. But… photography is NOT work in my mind.
Anyway, I’ll try to take notes from time to time and make backups every week from now.

@rsp the “lecture” was as much a reminder to me as anything else.

The database dumps are a good “chore” and help protect your investment in the creative editing you have undertaken, add a few comments to the end of the standard naming and they become a bit of a journal, take a dump just before a new (major or minor) release (appropriately named) and you are effectively logging the dates that you installed a new release.

The DOPs and the database can act as a store for the various attempts at refining the editing process but those edits are nameless (but not faceless). Creating a preset allows those attempts to be organised (and re-organised using a simple file manager) but they need to be secured as well so back them up to protect your “investment”.

I already have a place on the F: drive in ‘MY PHOTOS Other’ which I just updated prior to backing up that drive later today (it was very out of date but took seconds to bring it up to date!)

I have just copied the ‘MY PHOTO-Catalogs’ directory (it contains the DxO database files & backups, amongst others bits and pieces) from ​the E: drive (SSD) to the F: drive (HDD).

Projects I have no experience of whatsoever.

Rather than considering it a chore think of it as protecting the “joy” you experience in your photography and long may that continue.

Now to follow my own advice and start backing up my drives!!

And I’ll begin as of today my own weekly backups…
Thanks for your message.
Have a nice weekend!

Anyone using the PhotoLab database without sidecar .dop and .xmp files is playing Russian roulette. With five cartridges out of six chambers, not one.

Keywords and metadata are supposed to be stored in the sidecar .xmp. I’m worried a bit about .dop files being converted to PhotoLab 5 unintentionally. I’ll be stuck using both PhotoLab 4 and PhotoLab 5 for at least a year as most of my computers are on Mojave.

Keen to hear some more detailed sitreps from you, Bryan @BHAYT. Thanks for sharing what you’ve found so far.

1 Like

Alec @uncoy I have come across your posts with respect to DxO no longer supporting certain models/OS versions of Mac. With respect to my posts in this thread it started because some other users experienced problems when upgrading from PL4 to PL5. I had also had the same problem so I did some digging and wrote the posts, one of which included the phrase you quoted!

I have never worried about the database in the past and have never had to but on this particular release… There was nothing in my database that mattered (albeit I managed to recover it anyway) but for others it represented the results of a lot of effort.

In the case of @rsp DxO helped to salvage that investment and I am glad that they helped, if I was being critical I would probably suggest that it should never have happened in the first place but …

I was also under the wrong impression that the database did not hold an up-to-date copy of the editing data and that the DOP was the sole custodian but that was a very wrong assumption. Effectively the database is the prime custodian and PL goes to some lengths to protect the database entry, creating a ‘Virtual Copy’ whenever it is in doubt!

So what have I “discovered” so far

  1. Changing the Uuid at the end of the DOP does not cause an instant VC.
  2. Changing the Uuid located just after the ‘ShouldProcess…’ field causes an immediate VC. It is also probable that the file date timestamps are changed by the software I used to change the DOP and that alerted PL to the occurrence of a potential change.
  3. Changing the keywords in the DOP has no effect, PL5 currently appears to ignore the change and quickly changes the DOP back to the original value.
  4. Adding a keyword to the DOP and changing the Uuid (from 2 above) will result in a virtual copy with the new keyword.
  5. The ‘Name’ line in the DOP is not used as a checking mechanism, changing it will not have any impact and PL5 will change it back quickly (as part of a DOP update).
  6. I did cut and paste the editing from one photo DOP and pasted it into another DOP leaving the top and tail of the “old” DOP intact and managed to get PL5 to accept the new editing (with no VC) but using a preset is a lot easier!!

I would conjecture that a DOP will only be imported into the database if there is no entry in the database already otherwise it will come in as a ‘Virtual Copy’. What plans DxO have for the keywords in the DOP I am not sure but a command that allows for its import would be useful in the event that it is the only data available to a user.

I personally would like more controls to allow virtual copies to be “promoted” to the [M]aster status and the old master either to become a virtual Copy or be replaced entirely.

If you have read any of my posts during PL5 testing you will know that I have concerns about the underlying code. The database structures are incredibly “thin” compared with Lightroom, Photo Supreme etc. and others have commented on this but that only matters if there are any risks of system crashes at critical stages in the database update process or software bugs.

The latter brings us back to the reason for this thread in the first place!? The good news is that securing the database both physically and using the ‘Backup’ option in PL5 can go a long way to reducing the risk of the loss of investment.

Finally we have the question of the metadata itself being maintained in the actual JPEG and DNG and in ‘xmp’ sidecar files for RAW files. I know that there are PL5 users that want to be able to store metadata in sidecar files for JPG and others that want to store that metadata in the RAWs themselves. There are packages that allow for both options and currently PL5 follows the Lightroom line so metadata comes from and goes to the JPEG and DNG and to and from the ‘xmp’ sidecar for RAW files.

Many, many years ago a new release of Zoner damaged the exif data of some of my files, it was fixed quickly but the backup strategy needs to be able to cope with the fact that the data you are backing up may be corrupt and the backup copy is actually the intact copy! My backup strategy uses Beyond Compare which will warn about such occurrences but it is too easy to force an overwrite of the data and for speed I do not use BC in data compare mode, i.e. it uses size and date/timestamps for speed.

@sgospodarenko has recently posted an “annotated” DOP in response to requests from users in various posts.

Hope this helps.

1 Like