Library module has many indexes to deleted photos - how to clean up?

[thomasev (Thomas Evensen)]

I am trying to clean up the library module, but PL does not allow to remove entries from the library if file is already deleted. I don’t know how I managed to create an inconsistence where PL library believes there is a file and there is not. Anyway, I have tried to reindex but PL still includes deleted files in the library…

All I want is to highlight the entries with no pictures within the PL browser and delete the entries. But PL does not allow it due to the file is already deleted…

My question is, how do I clean up the library?? And if I delete the database file I suppose I delete all my keywords… Or is the keywords included within the sidecar file?

[Joanna (Joanna Carter)]

In my opinion, the quickest way to sort things out would be to delete the database.

As long as you don’t use projects or persistent history, everything else should be recovered from the DOP files - as long as your preferences are set to automatically save metadata to sidecars.

[thomasev (Thomas Evensen)]

Thx, but what about all the keywords I have added? Are they within the sidecar files?

[Joanna (Joanna Carter)]

Keywords are known as metadata and are automatically written to the DOP sidecar files, as long as you preferences are correctly set…

You can also choose to save metadata like keywords to XMP sidecar files, for compatibility with other metadata management software…

[thomasev (Thomas Evensen)]

Hi and thank you very much for reply I will check my preferences and delete the database file… (after a backup)…

All the best,
Thom as

[platypus (Canon & Mac (Intel and M1))]

@thomasev

check my preferences and delete the database file

I’d also check that all the relevant image (raw) files have the corresponding .dop and .xml sidecars. Caution: You’ll lose edits and metadata of raw files without sidecars, if you delete the database.

[thomasev (Thomas Evensen)]

I have just deleted the database, reindexed and imported all sidecar files. I have folders for each year and date. The reinstall forced me to open every catalog, mark all files and import the sidecars… It seems like every adjustment is imported, but I have only .dop files and no .xml files… Are there additional info within the .xml files?

[platypus (Canon & Mac (Intel and M1))]

@thomasev

I have only .dop files and no .xml files… Are there additional info within the .xml files?

DOP sidecars should be sufficient with DPL 6. XMP sidecars could still contain metadata that you might have changed in other apps. If you follow a “single point of definition” regime, you’re all set.

For future prevention of data zombies: Do not move or delete files any other way than within PhotoLab.

[thomasev (Thomas Evensen)]

[quote=“platypus, post:8, topic:32686, full:true”]

@thomasev

For future prevention of data zombies: Do not move or delete files any other way than within PhotoLab.

In the future I will not delete any files but only within PL. The challenge with PL is it is not suitable to curate files. After a day I might have 1000+ pics and only 10+ are keepers. PL is very good working on the files you want to keep, but you need another application for sorting out the keepers…

[Stenis (Sten-Åke Sändh)]

You don´t need to delete any database.
Deleting the database is not at all a good advice.
There is data you will never be able to recreate via DOP-files.

A better approach is to reindex the folder you have deleted files in.
That will get you on track again in seconds.
It is also important to organize your folder tree under one topfolder.
If you do that it´s very easy to reindex all your data just with a selecting of the topfolder and an activation of the reindexing process.

Advices to delete the database usually comes from people not using the data base at all.
From what I have read you don´t seems to be one of them.

I definitely know that "External Searches get lost when deleting a database and probably your “Projects” too.

In the case you wrote about above you might be better off using another fast viewer to cull all those images and then just reindex just the folder you have been working in.

[BHAYT (Bryan [WIN10, PL6, VP4, FP6, G9 & EM1 Mkii, RTX 3060])]

@Stenis

There is data you will never be able to recreate via DOP-files.

@Stenis What data? As far as I know the only data that will not be recovered is ‘Projects’ (and ‘External selections’, unless you have had these options reset rather than set as shown here

when DOPs will not be written or read except by the ‘Files’/‘Sidecars’/Import and /‘Export’ commands!

@Stenis

A better approach is to reindex the folder you have deleted files in.
That will get you on track again in seconds.

@Stenis answering from memory, rather than a test, DxPL re-indexes what it finds on disk not what it cannot find!? So after the removal of a database re-indexing is a good way of recovering the lost data but re-indexing when data has been removed from disk will not cause DxPL to remove the data from the database.

@Stenis

I definitely know that "External Searches get lost when deleting a database and probably your “Projects” too.

‘External Searches’ are effectively ‘Projects’ in the database but with slightly different data entries and both will certainly be lost if the database is deleted.

@Stenis

In the case you wrote about above you might be better off using another fast viewer to cull all those images and then just reindex just the folder you have been working in.

@Stenis as stated above I an afraid that doesn’t work.

@platypus

DOP sidecars should be sufficient with DPL 6. XMP sidecars could still contain metadata that you might have changed in other apps. If you follow a “single point of definition” regime, you’re all set.

@platypus mostly correct but you have not mentioned the first discovery rules that are also controlled by

from PL5.3.0 release onwards.

If the option is set then metadata is freely exchanged between the image and DxPL whenever it changes in either location, from the image when changes made via additional software are detected by DxPL and from DxPL to the image when made using DxPL.

But, in addition, with this option set, on first discovery metadata will be taken from the image (and/or sidecar) and not from the DOP. Concern over the formatting of keyword metadata by DxPL which may adjust the keyword metadata formatted by other software needs to be borne in mind when selecting the appropriate setting for this option etc…

However, if the option is not set, then the exchange will not take place automatically but can be forced by the user using the ‘File’/‘Metadata’/‘Read from image’ or ‘Write to image’ commands AND on first discovery the metadata will be taken from the DOP and not from the image (complicated somewhat by DxPL4 and earlier DOPs!). If a DOP does not exist then the metadata will be taken from the image (which includes the xmp sidecar).

The option can be changed at any time so arguably if reset (not set) during the initial rebuilding stage of the database, via user initiated traversing of the required directories and/or re-indexing then edits and metadata will be taken from the DOP.

If there is a possibility that some metadata has been set externally that may not have found its way into some DOPs then ‘Read from image’ can be used to “top-up” the DxPL database metadata (and be written into any new DOPs).

The option can then be left unset or changed to set depending on what course of action the user wants for metadata handling but the setting will continue to control both the source of the metadata for all ‘new discovery’ situations that may be encountered and the automatic reading and writing of metadata from and to the image (the two functions cannot be split as a result of the decisions made by DxO, the wrong decisions in my opinion, there should have been two options available not a single “overloaded” option).

[thomasev (Thomas Evensen)]

@Stenis

A better approach is to reindex the folder you have deleted files in.
That will get you on track again in seconds.

I tried to reindex first, but still not existing files was presented with an question mark within the PL browser and not allowed to delete/remove because of “no file”…

[platypus (Canon & Mac (Intel and M1))]

Okay, Let’s track the handling of a missing file.

1. Here’s a starting point:
   

   

2. The option to delete the missing file can be selected:
   

   

3. There is a dialog to verify that you really want to remove the file(s)
   

   

4. When I click on “Remove” in the dialog above, I get this:
   

   

---

While steps 1 to 3 are quite normal and okay, the “remove” action fails because it tries to delete the file instead of removing it from the database or asking whether the file should

1. be moved to the trash or
2. be removed from the database

Check out what other apps propose

• Lightroom
  

  

• Capture One
  

  

---

From a user point of view, an option to remove a file from the database is absolutely necessary because it can help to keep the database clean of orphaned entries - specially if a search for orphans were possible.

Quick Win (attn. @StevenL )

• We could keep the current programming, if the exception (delete non-existant file) were intercepted by PhotoLab (that’s already the case) and present a dialog “Remove from database” (plus appropriate action) instead of an error message.

[thomasev (Thomas Evensen)]

@platypus

From a user point of view, an option to remove a file from the database is absolutely necessary because it can help to keep the database clean of orphaned entries - specially if a search for orphans were possible.

I do agree 100%. It should be easy for DxO to include a function for this in DxO PL. I am a long time user of LR and have never experienced similar issue with LR… But I have converted to DxO PL and DxO has some potential for enhancing the library module within PL…

[BHAYT (Bryan [WIN10, PL6, VP4, FP6, G9 & EM1 Mkii, RTX 3060])]

@platypus

While steps 1 to 3 are quite normal and okay, the “remove” action fails because it tries to delete the file instead of removing it from the database or asking whether the file should

@platypus while I understand that @thomasev is a Mac user Windows users may also stray into this topic so I repeated your test on my Win10 system with PL6.5.0 and the following occurred

After changing the name of one ‘Purple’ image to “.sav” I have

Removing the image (with a ‘Remove’ command)

and then

So although I believe that a ‘Remove from database’ command is important for all the reasons that we have discussed before what you found here is a “bug”, i.e. the Mac version should work exactly the same way that the Windows version works.

PS:- The fix is less than 5 minutes work, and the Mac coder appears to have forgotten to ignore the context of the ‘Remove’ and attempted to undertake a delete that could never succeed, when the result should simply have been ignored. Fortunately for the Win users the coder got it right one way or another, but that is no consolation for Mac users.

[platypus (Canon & Mac (Intel and M1))]

@BHAYT , your case is slightly different - apart from Windows.

Can you try to replicate the following:

• select folder A in DPL’s sidebar, then quit DPL
• move a file from folder A to folder B with Windows Explorer
• open DPL…which should still show folder A and an image with a “?” instead of a thumbnail image

Can you now delete the (nonexistent) image with the “?” ?

[BHAYT (Bryan [WIN10, PL6, VP4, FP6, G9 & EM1 Mkii, RTX 3060])]

@platypus

• open DPL…which should still show folder A and an image with a “?” instead of a thumbnail image

Can you now delete the (nonexistent) image with the “?” ?

@platypus No because the image has gone from the DxPL display of A!? This actually happened when I was preparing my post above with a ‘Red’ image and I thought the colour must have had something to do with it (lame joke).

With that test the image directory was in scope and so I switched to using the ‘Colour label’ search which produced the “orphan” scenario I was looking for and the test then gave me an orphaned image and the ‘Remove’ behaved as I expected and reported above.

I executed your test scenario exactly as written and when DxPL came up after restarting DxPL the image was not there.

I then moved the image from B back to A with DxPL running but “pointing” at the “owner” directory of A and B and then opened A and both images were present and opened B and it was empty?

I am using DxPL6.5.0 and will repeat the tests but with the following additions

1. I will remember the colour and do a search before and after the move from A to B.
2. I will put the image to be moved in A into a project.
3. I will do the tests with a clean database.

[platypus (Canon & Mac (Intel and M1))]

@platypus

Can you try to replicate the following:

• select folder A in DPL’s sidebar, then quit DPL
• move a file from folder A to folder B with Windows Explorer
• open DPL…which should still show folder A and an image with a “?” instead of a thumbnail image

I have to be more precise…

Modified step:

• open DPL…which should still show folder A and searching the missing file should show an image with a “?” instead of a thumbnail.

The thread is about orphaned database entries…and the added part in bold print is necessary to reveal the orphan with the question mark.

[Joanna (Joanna Carter)]

@platypus

• select folder A in DPL’s sidebar, then quit DPL
• move a file from folder A to folder B with Windows Explorer
• open DPL…which should still show folder A and an image with a “?” instead of a thumbnail image

On the Mac, upon reopening DxO, there is no image, just the message that the folder doesn’t contain any images.

If I drag back the image into the folder in Finder, without closing PL6, the image appears in the browser pane.

I also tried removing the file, in Finder, while PL6 was open and it immediately disappears from the browser pane.

[platypus (Canon & Mac (Intel and M1))]

@Joanna

On the Mac, upon reopening DxO, there is no image, just the message that the folder doesn’t contain any images.

If I drag back the image into the folder in Finder, without closing PL6, the image appears in the browser pane.

I also tried removing the file, in Finder, while PL6 was open and it immediately disappears from the browser pane.

Yes, all of this is as you describe. As long as DPL is open and showing the folder, the contents of which is renamed or removed, DPL keeps track.

The issue we have here is, that DPL is unable to remove images from its database in those cases, where files are shown as results from a search and the “found” files are not where they were when DPL indexed the folder(s) because those files were moved or deleted by Finder or any other app while DPL was NOT running.

How to keep DPL’s database true to reality

• always keep all your image files in exactly one folder
• always keep DPL running and pointed at said folder
• Note: scales well with < 100 image files