My very first post for PL5 Beta testing back in early 2021 was to express concern about the fact that PL4 and PL5 Beta and now PL5.2.1.4737 (the latest release) have never successfully detected changes made to an image file (JPG) changed by ExifPro @alex (an old, now free, basic metadata program).
Why “rake over old coals” as the expression goes, and the answer is because PL5 does not perform as well as other programs with ExifPro and throughout PL5 Beta testing I (appeared to) encounter situations where a test came to an end prematurely because PL5 (Beta) failed to detect a change in the metadata! While I feel that PL5 is perhaps not as bad as it was (not entirely sure about that statement) I have had tests stop because PL5 or PL6 Beta has failed to detect a metadata change!?
Summary of yet more tests @sgospodarenko :-
All tests were conducted with JPGs and in the presence of the ‘Conflict Resolution’ bug (i.e. no ‘S’ detected or set or …):-
-
PL5 always fails to detect metadata changes made to JPGs by ExifPro because ExifPro makes the changes without changing the ‘Date Modified’, with AS(ON), but (some) other software does better.
-
PL5 “apparently” detects keyword changes made to JPGs by ACDSee and Exif Pilot with AS(OFF) (spoiler - because they make metadata changes by creating a new copy of the image file by deleting and creating a new copy rather “blatantly”)
-
PL5 displays ‘Image file missing’ after a Zoner change to metadata with AS(ON) (spoiler - because Zoner makes metadata changes by creating a new copy of the image file using an very, very, very … convoluted process involving the use of intermediate “.jpgs”)
-
Temporary DOPs in directory after Photo Mechanic metadata changes with AS(ON) but only sometimes (spoiler - because PM makes changes by creating a new copy of the image file by a slightly convoluted process that involves the use of temporary “.jpgs” sometimes picked up by PL5 )!?
So the the “clue” to issues 2, 3 and 4 is that the software in question make updates to JPGs for metadata changes by creating another copy by one means or another and these processes catch PL5 out!
In truth much of the software available does the same/similar thing but some (seem to) cause more grief to PL5 than others, principally because they use “jpgs” as intermediate files which are “spotted” by PL5 if it is running and “focussed” on the directory in question!
All of the above issues are arguably a “problem” with the software that is making the changes but PL5 has to co-exist with these products so I would argue that it is incumbent on PL5 to do its best to overcome the problems. Better research might have resulted in PL5 avoiding some of the “Traps” but then I only recently discovered FolderMonitor, which has me helped understand the (general) nature of the issue!
I have included a Footnote containing some comments from the author of IMatch discussing this issue back in 2014 and he identifies the issues caused by a common method of updating metadata, namely the creation of a new copy of the file but potentially via the creation of a temporary file (one or more!?) and the impact that can have on the ‘change detection’ process.
The problem is at its worst for programs that are looking to provide immediate, real-time recognition of metadata changes, e.g. PL5 (and Zoner), other software does this to an extent with ‘Ratings’ but frequently after a short delay, and some software require an ‘F5’ refresh or clicking on a file entry or the photo or thumbnail display to actually cause the program to (re-)read the data and display the current values, i.e. after the “dust has settled” from the changes made by another program.
PL5 offers either the instant, real-time recognition and display of the change [AS(ON)] or the ‘Read metadata from’ [AS(OFF)} rather than a middle ground approach to hide the fact that capturing the events too soon can lead to problems!
The problem with posts like this one:-
These tests take a while to run and to document and result in large posts (in my case) that most users don’t have the time and/or patience to read, particularly when the issue doesn’t affect their workflow, plus they are not in a position to fix the issue anyway!
If DxO also ignores them (the posts and the problems) then what is the use, other than exercising my aging mind (a potentially worthy cause but not when my wife catches me “wasting” my time engaged in “naughty” activities, i.e. answering a DxO forum post instead of more “worthwhile” activities!!)?
However, these issues need to be taken seriously if the whole DxO metadata handling saga is to be anything more than someone’s (failed) “vanity” project. But I will certainly be reducing the amount of such testing going forward unless DxO start taking the issues more seriously!
The nature of the problem:-
The key to the problems are how the various software packages go about changing ‘keyword’ settings in particular and how that impacts (or not) PL5.
FolderMonitor:
This exercise was made possible by the use of FolderMonitor 1.3.0.0 from Nodesoft which uses the following feature from Microsoft FileSystemWatcher Class (System.IO) | Microsoft Docs to capture folder events as I suspect does most of the other software on test here, including PL5 (I believe), this or a variant!
It would be good if FolderMonitor (FM) has increased granularity with the the timer and I am a little worried about the fact that a “modified” entry seems to mostly be two entries rather than one but my only real regret is that I didn’t find this software a lot sooner!
PIE image of all entries in Subdir-07 after the completion of the tests (some have been done more than once):-
PL5 Sequence:-
The PL5 sequence is about the shortest of all, except ExifPro and that includes updating the DOP!?
ExifPro:-
This is arguably the best behaved of the programs but PL5 ignores it completely but FolderMonitor (FM) and Zoner do detect the changes made successfully!
The sequence recorded by FolderMonitor (FM) is as follows (please note only one “modified” entry in this case!?)
Exif Pilot:-
During testing for the knew ‘Conflict Resolution’ a tester made a change with Exif Pilot and no ‘S’ icon was shown apparently suggesting that PL5 had missed detecting the change but the reason is a bit more complicated than that!
Exif Pilot creates and updates a “.tmp” file that is to become the new copy of the image but then deletes the old image before renaming the “.tmp” file to the name of the original image. Not the safest of strategies and one which has PL5 completely “fooled” with AS(OFF)!
PL5 sees this as a brand new image and adopts its “normal” strategy of taking the metadata from that image regardless of the AS setting, effectively the PL5’s equivalent of the “import” process so the ‘S’ icon will never appear in this situation!
ACDSee:-
ACDSee falls into the same “trap” as Exif Pilot with the following sequence (arguably the least safe of all sequences). Quite what the two modify steps are I don’t have a clue but the outcome is the same as for Exif Pilot no ‘S’ with AS(OFF) (tested when the new bug was absent!)
Adobe Bridge:-
It appears to create and then modify a “.tmp” file that contains the new updated image. This is followed by the creation and deletion of another “.TMP” file, presumably to check that is can do that successfully before using the “.TMP” file name to rename the original image. It then renames the “.tmp” file to the original image name (updated image now in place) before deleting the “.TMP” file.
This procedure appears to cause no problems with PL5 and testing renaming by hand seems to work fine with AS(ON) and AS(OFF).
Lightroom:-
Lightroom uses is similar strategy to Adobe Bridge and appears to be acceptable to PL5 in both the AS(ON) and AS(OFF) situations.
Photo Mechanic:-
Photo Mechanic is perhaps safest sequence creating an arbitrary “.tmp” file and deleting it to check it can be done, followed by creating a temporary “.jpg” and modifying it. This temporary “.jpg” is to become the updated image but unfortunately it is a “.jpg” and is sometimes “seen” by PL5 and a DOP is created and left behind! The original file is then changed to a temporary name, but as a “.jpg” before the other temporary “.jpg” is renamed with the original image name and the original file (as a renamed “.jpg”) is then deleted.
The disadvantage of the PM strategy is that the it uses “.jpg” files which are then “visible” to other image handling software, albeit only for a very short space of time!
Zoner:-
It is because Zoner is so good at the real-time capture of metadata changes, particularly ExifPro changes, that this whole exercise started. However, it has the worst and most convoluted sequence of all the programs and using “.jpg’s” for some of it’s intermediate files is a bad idea!! PL5 detects one or other of the temporary “.jpg” files and the result is “lost images” in PL5!
FOOTNOTE:-
IMatch author posts in an IMatch forum about detection issues:-
The following is from the IMatch author in response to a user query back in 2014! I must emphasise that a lot of the testing that I have done has been with multiple programs open at the same time and most have been well behaved with respect to the amount of time they lock a record.
The times being proposed in the posts below of 10 seconds appears “extreme” and the logs above show that the software that updates by recreating the record takes a very small amount of time but PL5 sometimes creates DOPs for temporary jpgs created by Photo Mechanic, always seems to capture temporary Zoner images and ACDSee and Exif Pilot create new entries in PL5 with new keywords etc. because PL5 sees them as “new” and performs its “standard” “import” process for a new entry!
"Re: Folder Rescans
« Reply #1 on: July 11, 2014, 08:02:47 AM »
-
IMatch does re-import files (and metadata) only when the “last modified on disk” timestamp of a file differs from the timestamp in the database. When IMatch imports a file, it saves the “last modified on disk time stamp”. When Windows sends the message “Something in this folder has changed” or “file changed”, IMatch detects updated files by comparing these time stamps. IMatch does not re-import unchanged files.
-
When you rescan a folder via + IMatch always rescans the folder and all sub-folders.
and in additional posts;
Quote
though it may take 10-15 sec. to become aware that another application has saved something to this folder.
This delay is intentional and required. Some applications keep files locked for some time after saving them. Some applications save to a temporary file in the same folder, and if the save operation completes, delete the original file and then rename the temporary file to the original file name. And similar schemes. To handle these weird cases (e.g. LR or PS or Office) IMatch waits for a short while, and then checks how long no update was performed in the folder. If things are quiet, it starts to rescan the folder because it is then fairly safe to assume that the other application is finished with whatever it was doing.
Quote
However, if I save something to a another folder, and IMatch is not running at the time, then when I later open IMatch and the database, IMatch will seemingly not realize that the folder has changed.
IMatch does not rescan all folders in your database every time you open it.
It would take very long to do this. Instead it compares the folder timestamps with the folder timestamps in the database. Folders with new time stamps are added to the background processing queue and are rescanned automatically. Unless you disable this under Edit > Preferences > Background Processing."