How do I get rid of "the lost images?"

Replicated your situation as best as I could and found the following:

  • PhotoLab seems to accumulate lost items with no reliable way back.

I duplicated a folder, opened it in DPL to make it add the files to the database, then closed DPL.
Renamed the folder, opened it in DPL to make it add the files to the database, then closed DPL.
Repeated this (also adding images to a project)…

Even though I tried to remove items from the database by deleting images from within DPL (delete files, quit DPL, moved files from the trash back into the folder and renamed it to a previous name etc.), I did not succeed to remove all the copies. Until DxO comes up with proper database maintenance, DPL just seems to collect whatever comes its way with no means to get rid of obsolete information.

:stuck_out_tongue_closed_eyes:

2 Likes

I wouldn’t say no means. There’s always the “nuclear option” :crazy_face::nerd_face:

Yeah, lots of problems were nuked in 50’s movies…now we’r much more reasonable (cough, cough)

2 Likes

Deleted the database and had DPL rebuild it from scratch. This took

  • 30 minutes for 25k images (no sidecars) on the internal SSD of my iMac
  • 60 minutes for the same load on the M1 MacBook Air

Your mileage may vary.

I took your advice @platypus and have now built up the projects in LrC, where I actually get some extra features — namely nested collections of collections instead of just a flat list. So now all my air show collections don’t need the words “air show” in their names, for example.

Also because LrC has colour labels, which I was already using — green means published — I’ve added all of the photos taken at an event/location to the collection and can then simply filter on colour to see un/published. I know I could use pick/reject in PL but I tend to use that ‘operationally’ for on-the-fly management that won’t be permanent.

1 Like

You certainly know that you can also use automatic collections, e.g. of photos with a green colour tag and other criteria.

Yes, I also have an automatic collection for green photos, but within the event collections it’s just as easy to filter on the fly, as required.

You do realise that Finder also allows you to set its own colour tags on files/folder? Then you aren’t stuck with not being able to find images in this or that software.

Then you can create a Smart Folder in Finder to just retrieve files marked with that tag only.

Thanks, I’ve used Finder tags and they’re not exactly the most efficient tool.

Given I am already in Lightroom doing keywording, it makes sense to me to just use it for collections, too. And hitting “8” is substantially quicker than setting a Finder tag.

I did consider coming up with a script to find the “Lightroom-says-green” files and also add a Finder tag, but then I remembered that Finder does not cache any kind of preview nor apply any default processing so makes a terrible viewer.

1 Like

Agreed. Especially because finder can add all colors to a file (but only 3 are displayed) and leaves too much of a gap between the file name and the color dot/s

Also, to exchange images with Windows users finder tags are rather lost efforts.

@Platypus I presume that your comments are based on viewing the internals of the PL5DB? I have not bothered to clear mine down so it will be full of all kinds of stuff however from a user perspective in Win10 there is generally nothing to see unless you index the files as I believe @zkarj was suggesting in the first post. Renaming (without indexing) simply shows the new directory when you navigate to it (I didn’t look into the database on this occasion so I am unaware what is going on’ or not’ there).

However when I changed the name of 2 of the files that had been indexed so they would no longer be treated as valid photos I received the following (please see snapshot) (the DOPs were still intact), i.e. two photos marked with an !. Even with the DOPs given new “identities” the screen remained the same.

When I finally navigated to the folder only two photos were shown and retrying the index search yielded only 6 photos, i.e. the two previously marked with an ! had vanished from view and from the search count!

Returning the photos to the folder by changing the name back to the original did not immediately change the ‘Search for Images’ result of 6 but navigating to the directory showed all 4 and returning to the search showed all 8.

I might be missing something but from a viewing perspective this appears O.K. with the caveat that if PL5 restarts on the display of a previous search the result it is not immediately refreshed so a misleading “picture” (sorry unintentional) might emerge until the folder(s) has/have been re-found by navigating to it/them when a subsequent search will show the true count.

Am I experiencing a WIN 10 versus Mac difference or am I conducting the right test in the wrong way or the wrong test entirely!? It has been a long day of plumbing and more of that to do tomorrow!!


!!

PS Colours what Colours! Set colours in FRV and shown in PM (3 JPG and 1 RAW), but Win10 PL5 does not appear to understand colours at all!!

Addendum:-
Projects work a bit like selections from the index, with deleted/missing photos being shown only as a thumbnail with an ! mark. However, navigating to the directory and then back to the project does not change the project display, the ! items remain even across a program restart.

To rid the database from image entries, there is only one way that seems to work now: Delete the image(s) using DPL’s delete command, which will remove the database entries and move the images to the trash.

If the command could be a) altered to remove database entries while leaving the files be (Lr can do that), we’d have a way to clean up the database…but b) DPL also checks if the file exists and c) prevents deleting missing files, which is another “problem” preventing DB cleaning.

Database cleanup requires these modifications

  • remove item from database, but not from the file system
  • remove item from database, even if the item doesn’t exist
  • check db item vs file system item and add or remove item while leaving the fs alone
2 Likes

@platypus I agree that the the file management in PL5 needs an upgrade along the lines that you state:-

In particular the first items I would like to see extended are (this could also be another situation where Win10 versus Mac comes into play)

  1. Provide a checkbox facility to enable “sticky” selection - a far safer way of selecting items from within a large directory without inadvertently cancelling items selected already. This could be either just a checkbox selection process or a full ‘Tagging’ process with associated commands, e.g. ‘Tag’ and select or move ‘Tagged’ or …

  2. Enable directory level selection including hierarchies of directories for “deletion”.

  3. All “deletions”(‘Remove’) to be extended to be files (with database) OR just database entries.

  4. Consider introducing ‘Move’ operations within PL5 at the file, files and directory level. Such ‘Moves’ should then also adjust the database to keep everything inline - but would need to move the ancillary elements with the file e.g. DOP and ‘xmp’ sidecar files etc. Most of us didn’t buy PL5 as a photo management system but as a photo editing system so we do need to be realistic about the “suggestions” we make. (it is a pity we can’t do strikethrough in forum texts because I wanted to do “demand” with strikethrough before “suggestions”).

But I am still unclear about the problem that you are describing in this topic and that you tested (I am sorry if I am being “thick” but I am a bit tired and finding it difficult to visualise the exact nature of the problem).

On my Win10 system the following happens;

  1. I have never experienced (so far) empty photo thumbnails. If I delete a photo I always have the thumbnail presented but with an ! mark. This could be Win10 versus MAC or the fact that I have not done large scale moves and somehow outrun the thumbnail cache. Where do you believe that the thumbnails are stored, e.g. in the database or as a reference in the database to the thumbnail cache or …?

  2. Apart from the “problem” I reported in my post about quitting PL5 while it is displaying a ‘Search for Images’ result and the count being potentially wrong on re-opening PL5 and showing ! images the only other place where I have also (so far) encountered the ! images is in a ‘Project’ which references ‘now deleted’ images.

  3. The ‘Search’ results appear to be updated after the folder with deletions is “visited” in PL5 BUT not if the deleted image is also part of a project!

  4. Externally deleted images (or moved/renamed etc.) will continue to have an entry in the project regardless until they are ‘Removed’ when they will vanish from the ‘Project’ and also from any existing ‘Search’ results.

  5. It is possible to ‘Filter’ on ‘! cannot be processed’ then select and remove from the ‘Project’, it is also possible to ‘Sort’ on ‘Processing data error’. Using these techniques it is currently possible to clean up some of the “mess” that can be created by editing outside PL5 which is currently the only alternative because PL5 file and directory management is on the “light” side.

PS Has anyone previously organised or considered organising online zoom, messenger etc. sessions to share desktops and conduct joint tests.

There are a number of dangers with such an approach, some of the discussions in the forum can become “heated” already, time zones, scheduling such activities with work life etc., in my case in particular broadband (very narrow band) bandwidth etc. just a thought particularly to explore Win10 versus Mac differences or you can have a “whip-round” and (buy) loan me a Mac mini so that I can test on both systems (once I have learned how to use a MAC (!!!) but that might just cause a divorce, which reminds me that I need to get back to trying to unblock the central heating, replace the rusting header tank and …!

Let me comment below

  1. I have never experienced (so far) empty photo thumbnails. If I delete a photo I always have the thumbnail presented but with an ! mark. This could be Win10 versus MAC or the fact that I have not done large scale moves and somehow outrun the thumbnail cache.
    ← OP’s thumbnails with a question mark arise from searching (which happens in the database) for something that has moved away from the location stored in the database. Example: The chocolate has been stolen, but the inventory system still thinks it’s in the shelf…
    Where do you believe that the thumbnails are stored, e.g. in the database or as a reference in the database to the thumbnail cache or …?
    ← Thumbnails are stored in the cache, not in the database, but the above is not the problem of the cache, but of the difference between what is and what the database has recorded (chocolate example)

  2. Apart from the “problem” I reported in my post about quitting PL5 while it is displaying a ‘Search for Images’ result and the count being potentially wrong on re-opening PL5 and showing ! images the only other place where I have also (so far) encountered the ! images is in a ‘Project’ which references ‘now deleted’ images.
    ← Sounds right. Looks like another sign of the chocolate inventory problem.

  3. The ‘Search’ results appear to be updated after the folder with deletions is “visited” in PL5 BUT not if the deleted image is also part of a project!
    ← Cannot comment this. While I worked with projects a few years ago, I almost never use them nowadays.

  4. Externally deleted images (or moved/renamed etc.) will continue to have an entry in the project regardless until they are ‘Removed’ when they will vanish from the ‘Project’ and also from any existing ‘Search’ results.
    ← When an image is removed from the database (using DPL’s delete feature), all references to that image should go away, which is not the case on Mac, as I’ve just seen in a short test. Also, an item that is remembered in the DB, but does not exist, cannot be deleted (removed from the DB) on Mac because of DPL, which simply pops up an error message:

  5. It is possible to ‘Filter’ on ‘! cannot be processed’ then select and remove from the ‘Project’, it is also possible to ‘Sort’ on ‘Processing data error’. Using these techniques it is currently possible to clean up some of the “mess” that can be created by editing outside PL5 which is currently the only alternative because PL5 file and directory management is on the “light” side.
    ← DPL on Mac can not delete absent items as shown in the screenshot above.

PS Has anyone previously organised or considered organising online zoom, messenger etc. sessions to share desktops and conduct joint tests.
← I have proposed this with private messages and found there was no need or want for such support.

@platypus Just about to go out to get the new header tank etc. before I resume my “day job” as a “plumber”!! Thank you for your response it appears that PL5 on the PC is somewhat better behaved than on the MAC on this occasion (although that could be a function of the DB software used?).

Because of the limitations of the file management in PL5 (possibly because they want to avoid all the work involved in not only doing the actual file management but also trying to keep the database inline with those changes) users will always have to resort to external file management and PL5 will then have to play “catch up”, but that “catch up” should be “elegant” and not leave elements hanging.

I think I would raise the issues as a ‘bug’. Sadly my investigations do not help @zkarj at all in this case because of the differences between the MAC and Win10 PL5.

If you want to use private messaging with me then by all means do e.g. to draw my attention to an item that I appear to have ignored and you want tested on Win10,

Regards

I do believe that DxO is moving in that direction. Question being: How many (paid) releases will it take to get there?

Meanwhile, add your input here:

@platypus will check that out later after I have collected my bits, bought some more copper, built a new platform for the new header tank, plumbed it in, removed the blockage (the real hard part), refilled the system and restarted the central heating - it could be a while!

Exactly, and that does not work if the file is no longer there.

Which either of these would fix.

I do think there is some caution to be applied to the last one, though. When I have elected to move images outside of PL, I have been able (with some difficulty) to tell it where they had gone and get them linked back up. If the database were to automatically “get rid of” any entries without matching files, this process would no longer be possible.

The simplest thing for DxO would be to get a copy of Lightroom (I’m sure that they have one) and see how Lr manages to prevent inconsistencies - and then add what is missing. :exploding_head:

2 Likes

This is one of the things with PhotoLibrary that has to be sorred out when people now really has started to use PhotoLibrary that really is version 1.0.

People that use PhotoMechanic or other similar software to manage metadata and cull images I’m pretty sure is doing that in the system that owns the metadata instead of doing it in Photolab. So that’s just another reason to get a more advanced solution if you are unsatified right now with these features. Still there are workarounds like reindexing it all or just the folders that are out of sync to solve the problems so it’s not something you can’t fix.