In this case you can see there are four of each filename. The last one in each set is the current one which is working perfectly. The first two in each set have impossible path names like /2017/09 and the third in each set just shows – for the path name.
They only show up in search, but as you can see, 3/4 of my search results are these useless “lost images”. How can I convince PL5 to forget about them?
Moving around images will definitely destroy the consistency between what is and what the database remembers - if moving is done outside of any database based app. While Lightroom offers means to add/remove moved images, DPL can’t.
I’d try to delete the missing items in Library view and, if it doesn’t work, have DPL index the whole photo collection from scratch.
Delete the database, start DPL and index all image folders. All unsaved changes will be lost though.
DPL5 took a while indexing 25k files during dinner. As for losing edits and metadata: I still stick to Lightroom and use DPL as a plugin, DPL is not vital for asset management in this case.
…use PhotoLab as a preprocessor or as a plugin to Lightroom… As of today, DPL cannot refresh the database, add or remove entries to make the DB fit reality, which practically disables DPL as an asset manager. You might want to test if DPL writes project information into sidecars. If it does, not everything would be lost.
Replicated your situation as best as I could and found the following:
PhotoLab seems to accumulate lost items with no reliable way back.
I duplicated a folder, opened it in DPL to make it add the files to the database, then closed DPL.
Renamed the folder, opened it in DPL to make it add the files to the database, then closed DPL.
Repeated this (also adding images to a project)…
Even though I tried to remove items from the database by deleting images from within DPL (delete files, quit DPL, moved files from the trash back into the folder and renamed it to a previous name etc.), I did not succeed to remove all the copies. Until DxO comes up with proper database maintenance, DPL just seems to collect whatever comes its way with no means to get rid of obsolete information.
I took your advice @platypus and have now built up the projects in LrC, where I actually get some extra features — namely nested collections of collections instead of just a flat list. So now all my air show collections don’t need the words “air show” in their names, for example.
Also because LrC has colour labels, which I was already using — green means published — I’ve added all of the photos taken at an event/location to the collection and can then simply filter on colour to see un/published. I know I could use pick/reject in PL but I tend to use that ‘operationally’ for on-the-fly management that won’t be permanent.
You do realise that Finder also allows you to set its own colour tags on files/folder? Then you aren’t stuck with not being able to find images in this or that software.
Then you can create a Smart Folder in Finder to just retrieve files marked with that tag only.
Thanks, I’ve used Finder tags and they’re not exactly the most efficient tool.
Given I am already in Lightroom doing keywording, it makes sense to me to just use it for collections, too. And hitting “8” is substantially quicker than setting a Finder tag.
I did consider coming up with a script to find the “Lightroom-says-green” files and also add a Finder tag, but then I remembered that Finder does not cache any kind of preview nor apply any default processing so makes a terrible viewer.
Agreed. Especially because finder can add all colors to a file (but only 3 are displayed) and leaves too much of a gap between the file name and the color dot/s
@Platypus I presume that your comments are based on viewing the internals of the PL5DB? I have not bothered to clear mine down so it will be full of all kinds of stuff however from a user perspective in Win10 there is generally nothing to see unless you index the files as I believe @zkarj was suggesting in the first post. Renaming (without indexing) simply shows the new directory when you navigate to it (I didn’t look into the database on this occasion so I am unaware what is going on’ or not’ there).
However when I changed the name of 2 of the files that had been indexed so they would no longer be treated as valid photos I received the following (please see snapshot) (the DOPs were still intact), i.e. two photos marked with an !. Even with the DOPs given new “identities” the screen remained the same.
When I finally navigated to the folder only two photos were shown and retrying the index search yielded only 6 photos, i.e. the two previously marked with an ! had vanished from view and from the search count!
Returning the photos to the folder by changing the name back to the original did not immediately change the ‘Search for Images’ result of 6 but navigating to the directory showed all 4 and returning to the search showed all 8.
I might be missing something but from a viewing perspective this appears O.K. with the caveat that if PL5 restarts on the display of a previous search the result it is not immediately refreshed so a misleading “picture” (sorry unintentional) might emerge until the folder(s) has/have been re-found by navigating to it/them when a subsequent search will show the true count.
Am I experiencing a WIN 10 versus Mac difference or am I conducting the right test in the wrong way or the wrong test entirely!? It has been a long day of plumbing and more of that to do tomorrow!!
PS Colours what Colours! Set colours in FRV and shown in PM (3 JPG and 1 RAW), but Win10 PL5 does not appear to understand colours at all!!
Addendum:-
Projects work a bit like selections from the index, with deleted/missing photos being shown only as a thumbnail with an ! mark. However, navigating to the directory and then back to the project does not change the project display, the ! items remain even across a program restart.
To rid the database from image entries, there is only one way that seems to work now: Delete the image(s) using DPL’s delete command, which will remove the database entries and move the images to the trash.
If the command could be a) altered to remove database entries while leaving the files be (Lr can do that), we’d have a way to clean up the database…but b) DPL also checks if the file exists and c) prevents deleting missing files, which is another “problem” preventing DB cleaning.
Database cleanup requires these modifications
remove item from database, but not from the file system
remove item from database, even if the item doesn’t exist
check db item vs file system item and add or remove item while leaving the fs alone
@platypus I agree that the the file management in PL5 needs an upgrade along the lines that you state:-
In particular the first items I would like to see extended are (this could also be another situation where Win10 versus Mac comes into play)
Provide a checkbox facility to enable “sticky” selection - a far safer way of selecting items from within a large directory without inadvertently cancelling items selected already. This could be either just a checkbox selection process or a full ‘Tagging’ process with associated commands, e.g. ‘Tag’ and select or move ‘Tagged’ or …
Enable directory level selection including hierarchies of directories for “deletion”.
All “deletions”(‘Remove’) to be extended to be files (with database) OR just database entries.
Consider introducing ‘Move’ operations within PL5 at the file, files and directory level. Such ‘Moves’ should then also adjust the database to keep everything inline - but would need to move the ancillary elements with the file e.g. DOP and ‘xmp’ sidecar files etc. Most of us didn’t buy PL5 as a photo management system but as a photo editing system so we do need to be realistic about the “suggestions” we make. (it is a pity we can’t do strikethrough in forum texts because I wanted to do “demand” with strikethrough before “suggestions”).
But I am still unclear about the problem that you are describing in this topic and that you tested (I am sorry if I am being “thick” but I am a bit tired and finding it difficult to visualise the exact nature of the problem).
On my Win10 system the following happens;
I have never experienced (so far) empty photo thumbnails. If I delete a photo I always have the thumbnail presented but with an ! mark. This could be Win10 versus MAC or the fact that I have not done large scale moves and somehow outrun the thumbnail cache. Where do you believe that the thumbnails are stored, e.g. in the database or as a reference in the database to the thumbnail cache or …?
Apart from the “problem” I reported in my post about quitting PL5 while it is displaying a ‘Search for Images’ result and the count being potentially wrong on re-opening PL5 and showing ! images the only other place where I have also (so far) encountered the ! images is in a ‘Project’ which references ‘now deleted’ images.
The ‘Search’ results appear to be updated after the folder with deletions is “visited” in PL5 BUT not if the deleted image is also part of a project!
Externally deleted images (or moved/renamed etc.) will continue to have an entry in the project regardless until they are ‘Removed’ when they will vanish from the ‘Project’ and also from any existing ‘Search’ results.
It is possible to ‘Filter’ on ‘! cannot be processed’ then select and remove from the ‘Project’, it is also possible to ‘Sort’ on ‘Processing data error’. Using these techniques it is currently possible to clean up some of the “mess” that can be created by editing outside PL5 which is currently the only alternative because PL5 file and directory management is on the “light” side.
PS Has anyone previously organised or considered organising online zoom, messenger etc. sessions to share desktops and conduct joint tests.
There are a number of dangers with such an approach, some of the discussions in the forum can become “heated” already, time zones, scheduling such activities with work life etc., in my case in particular broadband (very narrow band) bandwidth etc. just a thought particularly to explore Win10 versus Mac differences or you can have a “whip-round” and (buy) loan me a Mac mini so that I can test on both systems (once I have learned how to use a MAC (!!!) but that might just cause a divorce, which reminds me that I need to get back to trying to unblock the central heating, replace the rusting header tank and …!
I have never experienced (so far) empty photo thumbnails. If I delete a photo I always have the thumbnail presented but with an ! mark. This could be Win10 versus MAC or the fact that I have not done large scale moves and somehow outrun the thumbnail cache. ← OP’s thumbnails with a question mark arise from searching (which happens in the database) for something that has moved away from the location stored in the database. Example: The chocolate has been stolen, but the inventory system still thinks it’s in the shelf…
Where do you believe that the thumbnails are stored, e.g. in the database or as a reference in the database to the thumbnail cache or …? ← Thumbnails are stored in the cache, not in the database, but the above is not the problem of the cache, but of the difference between what is and what the database has recorded (chocolate example)
Apart from the “problem” I reported in my post about quitting PL5 while it is displaying a ‘Search for Images’ result and the count being potentially wrong on re-opening PL5 and showing ! images the only other place where I have also (so far) encountered the ! images is in a ‘Project’ which references ‘now deleted’ images. ← Sounds right. Looks like another sign of the chocolate inventory problem.
The ‘Search’ results appear to be updated after the folder with deletions is “visited” in PL5 BUT not if the deleted image is also part of a project! ← Cannot comment this. While I worked with projects a few years ago, I almost never use them nowadays.
Externally deleted images (or moved/renamed etc.) will continue to have an entry in the project regardless until they are ‘Removed’ when they will vanish from the ‘Project’ and also from any existing ‘Search’ results. ← When an image is removed from the database (using DPL’s delete feature), all references to that image should go away, which is not the case on Mac, as I’ve just seen in a short test. Also, an item that is remembered in the DB, but does not exist, cannot be deleted (removed from the DB) on Mac because of DPL, which simply pops up an error message:
It is possible to ‘Filter’ on ‘! cannot be processed’ then select and remove from the ‘Project’, it is also possible to ‘Sort’ on ‘Processing data error’. Using these techniques it is currently possible to clean up some of the “mess” that can be created by editing outside PL5 which is currently the only alternative because PL5 file and directory management is on the “light” side.
← DPL on Mac can not delete absent items as shown in the screenshot above.
PS Has anyone previously organised or considered organising online zoom, messenger etc. sessions to share desktops and conduct joint tests. ← I have proposed this with private messages and found there was no need or want for such support.
