So no. In my workflow The database is a temporary caching of the data including management of other temporary things like cache files for performance. That’s it.
I’m in line with that. With the looming fear of virtual copy doubling in the background and when using the photos on multiple devices, I usually delete the PL database before every start.
Plus, a file based “database” is a toy sollution as it does not in any sensilbly way scale beyond a single workplace. Move it (as an option) to a SQL server (at least the metadata) and we can talk.
Previous discussions about putting .DOP data into an .XMP file leaves PL open to issues if other programs damage or destroy the .DOP data in the .XMP files.
Personally, I rely on .DOP files to store all my edits. I would like to see JUST PL proprietary data in .DOP files and everything else in .XMP files. I do not want to rely on a single database when my photos are stored all over the place (internal drives, external drives, backup drives, archive drives etc.)
Until PL provides the facility to select different databases to work with (like Lightroom or Photo Mechanic Plus) I will not be using PL as my DAM.
That’s why I wrote somewhere correspondingly, stars and colour labels should be common ground with important software – and store them as xmp, not as proprietary data.
A single point of definition (for all images) is also a single point of failure (for all images). I favor the concept of sidecar files as the leading location for edits and a database as a secondary and disposable method only used for convienence / performance. This is also the concept of darktable and Capture One sessions (not C1 catalogs unfortunately). Sidecar files are also better for integration with an external DAM.
I understand the .dop sidecar as DPL’s way to store whatever information outside of the database - which is perfectly okay for me. It’s perfectly okay if DPL writes metadata to these files and I’m quite happy withe the structured text (instead of a binary or encrypted) file.
In a perfect world, we could live with either the database or the .dop files because both would contain the same information* throughout and drives would be infallible, software bug-free…
Enter Reality
The database can be corrupted by buggy software, drive failure or user action
The .dop files can be corrupted by buggy software, drive failure or user action
Database and .dop files do NOT contain the same information*
etc.
If DPL should be more resilient, the following should be done imo:
Database and .dop sidecars should mirror each other
Information should be fully updatable, either from .dop to DB or from DB to .dop in a controlled manner and: all issues like the one mentioned in this thread’s title should be fixed
Database maintenance should be established (consistency check, auto-backup…)
Note: information* mostly means the stuff that we’re interested in and that we can change and edit, but also the behind the scene stuff that is needed to keep the system live and healthy.
@platypus I am a little confused by your comments!?
What information are you referring to I would suggest that the DOP contains the vast majority of the information in the database, just in a slightly different form, The data in the database is just that i.e data the “metadata” that controls that data are the fields in the database that the data occupies, when that is externalised then some additional metadata has to be added but little or nothing is actually left behind!
Please identify what data is actually missing from the database data in the DOP, there is sufficient for the database entry to be recreated from the DOP, that statement is also true for the ‘Rating’ etc. fields that are currently “information” only (Write Only Memory) fields which PL5 currently does not use.
As for the corruption issues that is true but if DxPL manages to corrupt there own data then they can also corrupt the xmp data, albeit that might well be handled by library software that has a long pedigree for handling such data.
They do mirror each other in the only way that any external data can mirror a database, i.e. metadata plus data versus database fields + data!
It is controllable via the sidecar options in the preferences and the import and export options, these and the AS (auto sync ) and read from and write to commands are arguably mirrors of one another!?
Database backups are available but without the nagging prompts of other software, if you want the nagging prompts that can be very easily added I would suggest. As for consistency checks please read what follows below!
There are many things wrong with the PL5 release in particular but the issues highlighted in this post in this post are generally not amongst them! The issue of using the DOP [M]aster WOM (‘Rating’ , ‘keywords’ etc is a moot one and needs discussion with forum members!
Now to what might really be wrong with PL5 but which I have not seen any evidence of yet. The database software that PL5 uses is SQLite, the same database software that most other DAM etc. software uses, ACDSEE uses either DBase of one of the variants that sprang from DBase with a much longer heritage. This is possibly one of the reasons that people talk about loading 100,000s of photos into such a database successfully.
The only thing that worries me about PL5DB is that it is very “light” in structure compared with some other similar databases. The metadata is joined to the image data by a simple numeric key, i.e. the ‘metadatas’ structure is joined to the ‘items’ structure by a single number and you get from one to the other by an index. There appears to be no corroborating data between the two entries, they are linked by what “we in the database” industry would call an “unprotected” link.
The software I supported for 36 years provided such facilities directly to customers coders, it also supplied “protected” links and we could always store the associated data in a structure that could be accessed via an index.
The mechanism used by PL5 is very cheap on space but cannot be successfully policed (consistency checked) because there is no corroboration between the 2 structures, if a key was stored with the metadata e.g. the image name then some corroboration exists to be checked but in my testing I use the same image countless times!?
There are Uuids all over the place and one of those would be suitable to verify that the correct data has actually been accessed when a simple pointer has been followed. Without that **unnecessary duplication of data within the database it becomes difficult to police links between structures and potentially recover from any database damage or even detect that anything has gone wrong!
Much has been written about single sources of data and the dangers of replicate becoming replicake in posts on this forum but I am afraid there are places where duplication is arguably essential. I have not studied the databases of the other system in detail but they are way more complex that PL5 but in many instances they are also more sluggish than PL5 when updating!
PS:-@platypus I am sorry if the above is a little critical of your post, I concur with almost all you have written about improved directory management for PL5 in other posts but I feel that this one is being overly critical. You have not specifically identified what is actually at fault??
One area that has niggled me for a long time @sgospodarenko is the mostly useless item that heads up the ‘Advanced History’ palette, namely
That piece of information is held in the database and is present in the DOP and I believe I have seen complaints from a long time ago on the forum but I feel that it is essentially completely and utterly useless!!
What I want (very parochial!) is a record of the last preset that I applied, i.e. because arguably my edits will be based (to at least some extent) upon it, in addition the creation of any preset should also be added to the ‘Advanced History’ and I would like that stored (only the last one) and retrieved because those provide some “provenance” for the edits that I then see before me!
