Fresh new database and project start

Hi all (and photolab gurus more specifically),

I will start a new fresh database from scratch with PL6. No import, everything from scratch inside photolab.
So I will use about every existing metadatas and keywords for indexing. I will use extensively projects too, now we have a way to organize them.

I have 2 main questions :

  • 1 - COMPATIBILITY :
    What to do and what not to do, to get this database as compatible as possible with other softwares. Are there some incompatibility issues ? what are they if any ?

  • 2 - EXTERNAL DRIVES :
    I work with several external and removable drives for photography, and not all are plugged at the same time on my worstation.

  • a) Will photolab keep trace of every data when plugging, unplugging, changing extrenal photo drive ?

  • b) I would prefer to put each cache and database folder on each relevant drive. Is there a problem with this (except the need to configure photolab config file pathes everytime external photo drive is changed) ?

  • c) What if I need to move (or copy) some images from one drive to one other (when both are plugged) ? is there a way to copy metadatas and projects datas too ? ( is copying .dop and .xmp files with raw enough ?)

Thanks for your responses.

PLEASE, THIS IS A THREAD ABOUT HOW TO USE PHOTOLAB, SO DON’T EXPLAIN HOW TO USE OTHER DATABASE SOFTWARES INSTEAD OF PHOTOLAB HERE.

Three question, one answer: PhotoLab 6’s database is only compatible to PhotoLab 6.

PhotoLab is made to be used on one drive mostly. Copying database files to and fro is asking for trouble. Same for having your photo archive on more than one drive. Many posts out there explain why, search the forum! Best practice: Leave the database and photo archive on one drive or one drive each. Everything else is backup.

2 Likes

@JoPoV one of the most important things is whether you are running DxPL(Win) or DxPL(Mac) as my post identifier shows I am running a Win 10 system and any tests I carry our will be pertinent to that platform and may or may not be relevant to the Mac!?

The database is SQLite (like most of the other such software that uses a database) and as @platypus has stated relevant only to DxPL.

This may or may not cause issues please see This image cannot be processed since the source file could not be found ( After I cloned the Drive ) for a “similar” situation were some “cracks” were found! I will investigate those posts again to remind myself of the exact nature of that particular problem.

I think that it does but that may actually work against you just depending on the workflow that you want to employ?!

There have been requests in other posts on the forum asking for a similar feature! My concern is that it potentially adds even more issues with drives coming and going because now there is a DxPL database doing the same!

I actually believe that DxPL can handle even the same (named) files, in the same (named) directories on two different drives connected at the same machine connection with the same drive letter! But as @platypus has stated there are potential issues and the DOP and any Xmp sidecar file are all that is at your disposal!?

‘Projects’ cannot be dumped out and read back in so they will remain in the database in which they were created!?

So I decided to try an experiment as follows

  1. Database hosted on a “normal” drive, the E: drive in my case which is my “extension” to the C: drive, both of which are SATA SSDs.

  2. Files on “identical” USB3 connected drives, i.e. same named files and held in identical directory structures. Drives were “old”, smaller SATA drives connected via USB3 to SATA convertors, labelled “VERTEX 4” and “INTEGRAL 2” to create a potential “worse case” scenario if any confusion is to exist!

  3. Started to experiment on my Win 10 system! The files were two JPGs and 3 RAWs and they were “discovered” in PL6.2.0 in the order “INTEGRAL 2” JPG then RAW followed by “VERTEX 4” JPG and RAW.

  4. Create 6 projects that contained
    1 INTEGRAL 2 JPG and
    2 INTEGRAL 2 RAW,
    3 VERTEX 4 JPG and
    4 VERTEX 4 RAW and
    5 all JPGs and
    6 all RAWs,

i.e. 2 pairs of ‘Projects’ (4 in total) specifically with only files from one or other of the two drives and 2 projects which contained mixtures from both drives!

The database ‘Items’ structure with all 10 files having been discovered:-

The ‘Folders’ structure showing how DxPL manages “transient” devices:-

The ‘UniqueId’ is related to the Volume Id. of the device and controls how DxPL “matches” the directories (folders) on the device with entries in the database, i.e. not just by drive letter but by the “Volume id” ('UniqueId) I believe! Please note the designation of ‘EFDOPVolume’ for these entries in the database.

The 6 ‘Projects’ to be used when a device or devices are absent and are re-discovered to see how DxPL handles such a situation:-

By using ‘Projects’ it is possible to check if DxPL is handling the “original” database entries for the directories or rediscovering them anew!?


The two devices with the “identical” images in the “identical” directory structure:-

What happens when a drive is “missing”:-

There are warnings but nothing particularly dramatic, at least not during my testing!

The database after the drives have been disconnected, re-attached with different convertors and “forced” to have different drive letters!!:-

To stress DxPL “a little” I changed to older convertors and forced the attachments to be assigned different drive designators and everything worked as it should, I am glad to say!

This test was with a database not co-located with any images because the images were mounted on removable devices. If you start co-locating with the data (not really a problem) but using removable drives then there will always be issues when the database you were using is not to be found because the drive is not attached.

At that point DxPL will create a new database and place it in the default location and you will then need to switch to the database on the drive which is actually attached!

With respect to copy/moving files between the drives then the danger has been covered many times in the forum under the banner of “Unwanted Virtual Copies”. This can occur when the following situation is encountered

  1. DxPL is encountering a directory (folder) which has exactly the same name as one already in the database. As the above tests have shown that must include the ‘VolumeId’ (or so my tests seem to show).
  2. The Uuid in the DOP of a file already in the database (on the same Volume, with the same hierarchical structure) must match the Uuid of the DOP otherwise the “new” DOP data will be added to the database as ‘Virtual Copy’ [1] and the original database entry will be designated as [M]aster!

It is for this reason that I deliberately set out to make the structures on both the test SSDs identical so that I could test to make sure DxPL(Win) did not get confused and start creating unwanted VCs.

Before I try testing any further, i.e. adding the databases to each SSD I would like more than a set of “wishes” that seek to allow you to make decisions about you work flow later, I guess that you may want to move the drives between systems?

So please try to target my testing to what you believe is the most likely workflow that you want to adopt so that I can try to replicate that as closely as possible, if I feel up to testing any further! The testing takes a little time but the write ups take a lot longer (and the re-tests when something unexpected goes wrong!).

I have two near identical systems which I can use to test a scenario that involves moving drives between systems where there is a real danger of “Unwanted Virtual Copies” and moving files with the database between systems where there is …? I tested that situation some time ago but was unaware of the ‘VolumeId’ at the time so those results are extremely suspect!?

Regards

Bryan

1 Like

So I continued with my tests as follows;

  1. On my Main machine I moved the database location to the INTEGRAL 2 drive and loaded an “empty” database.

  2. I then recreated much of what was done in the previous tests but only using images resident on INTEGRAL 2. This included creating appropriate ‘Projects’ and I then closed DxPL(Win).

  3. I then copied the database from the INTEGRAL 2 drive to the VERTEX 4 drive, simply to reduce the steps necessary for the test!

  4. I released INTEGRAL 2 from the Main machine and opened PL6.2.0 which could not find the database (on the removed INTEGRAL 2) so it automatically created a new one at the default location! I then specified the location of the database on VERTEX 4 and closed and re-opened DxPL and it found the database (which had originally been setup on INTEGRAL 2, in this case).

  5. I added the images from VERTEX 4 and created additional projects before closing DxPL and releasing VERTEX 4 on the Main machine!

  6. At this point the VERTEX 4 drive contains a database and image files with the database containing entries for images on both the VERTEX 4 drive and the INTEGRAL 2 drive.

  7. The VERTEX 4 drive was closed and released from the Main machine and plugged into the Test machine where DxPL was re-configured to expect the database on the VERTEX 4 drive (after a restart). The main difference is that without any cached items we have the following on the Test machine.

and the ‘Folders’ structure looks like

  1. Setting the edits of the RAWS to include ‘DeepPrime’ and setting the ‘Tag’ and the ‘Ratings’ we have the following on Test

10, Transferring VERTEX 4 back to the Main machine we have the following

So @JoPoV you can certainly do much of what you “requested” on Win 10!

Currently I have not tested keeping the ‘Cache’ on the same drive and need a potential workflow to fully determine what is and what is not possible using this scheme with respect to moving images between the two drives, i.e. between INTEGRAL 2 and VERTEX 4 in my test case!!

Please remember that opening DxPL on a machine where the last drive used is no longer present, i.e. a removable drive, will result in DxPL automatically creating a new database at the default location!

1 Like

Thank you for all the time you took to respond me.
I have been very busy with my work and have not been on the forum since.

And I still not have rebuilt my database.

I hope I’ll have time for this very soon.

So, again thank a lot for all your tests wich are very valuable for me.

@JoPoV you were starting a thread in Mid January and after that abandoned it for 2 ½ months. I don’t want to judge, f that’s polite, helpful for you or respectful to the ones replying. It just raises the question: how much time can you spare to build up this fragile thing DxO calls a “database”? Or anything else along this line?

@JoPoV Glad you are still with us, ignore @JoJu’s “indignation” on my and @platypus’s behalf and in particular ignore the comments about the fragility of the DxPL database, which is not especially “fragile” at all but has its design “wrinkles” and one of them involves the use of the “Volumeid”.

I believe (but no-one from DxO has ever explained the exact reason for the use of the field) that it is present to allow USB drives to come and go without confusing their contents with any other drive that might have the same directory structure and the same images, e.g. a backup drive. DxPL will treat each copy as independent and there will be entries in the database for each and every copy carefully managed by DxPL and the ‘Folders’ structure.

The database can be abandoned at any time since the DOP carries the necessary data to re-create the individual image and folder entries but not any ‘Projects’. ‘Projects’ only exist in the database and cannot be saved and recreated/reconstructed in any way, i.e. the database is precious for the preservation of ‘Projects’.

Please identify whether you are using DxPL(Win) or DxPL(Mac) because there are differences between the two products and where it is possible to move the database location about on Windows it is not possible (I believe) to do the same on the Mac (and as my forum id. indicates I am a Windows user).

Your intended workflow would also be useful.

Regards

Bryan

If you were not as busy to write endless manifestos @BHAYT you might have discovered one sentence of @JoPoV :

This and your “database can be abandoned anytime” don’t go well together.

Do I have to tell you everything? :grin: Oh, and it’s ok to ignore my indignation on your behalf, you would write two more chapters if there’s a living being around you could suspect as a potential reader - just don’t speak for @platypus as well :yum: keyboard-addict as you might be…

It’s ok you don’t want to judge, because you don’t know me neither my life.
Anyway reading some of your posts, it seems you’ve become a big troll on this forum …
Please, don’t respond, I probably won’t have time to read more for a long time anyway.

1 Like

So do you. I’d rather be a “troll” (by your definition, not mine) than guy asking for help and not answering to the helpers. :stuck_out_tongue: