PL5.2.0.4730 Conflict Resolution only detects externally generated conflicts

It´s Camera Bits and not Camera Labs

Camera Bits Forums - Index

Was it this article you referred to Bryan where they advise people not to use the two way sync of performance reasons?

Image Alchemist | Sync Metadata Between Photo Mechanic And Capture One • Image Alchemist

By the way, thanks a lot for your empiric testing efforts. Intresting!

Bryan, I have no time to anonymise data just now and my data is anyway published widely on the net already so I am happy to send it to me if you just kan mail me your address.

Isn´t the problem with the loss of the “ratings” explained by the fact that Ratings is EXIF and the rest IPTC? Everybody is talking about XMP but not all are using the namespaces of EXIF and IPTC in XMP, instead they are using the older pre XMP way of reading and writing IPTC. At least in Photo Machanic it´s an active action to write IPTC to both places.

… and it´s not a surprice Photolab seems to interact best with Adobe Bridge and Lightroom. DXO have spent quite a few years to polish these hot links but have just started to be aware of that a lot of users are using other tools than the ones Adobe manufactures.

1 Like

@Stenis I have sent you a message with my email address and thanks for access to an image.

While that is certainly possible the strange thing is that when the keyword changes the products find both the ‘keyword’ and the ‘Rating’. It is possible that they are looking at another file attribute, such as size and if that doesn’t change the image is ignored!?

Sadly, while I knew what was appropriate on the old platforms I worked on and what interrupts and file attributes could be used and interrogated etc. I do not know how the programs are structured to “detect” a potential change without “buzzing” which typically shows as excess CPU usage?

As a consequence I do not know what attributes etc. Bridge is using versus what Zoner and PM are using versus what PL5 is using. If PL5 was coded like Zoner or PM it would be better than it currently is, if it could be codded as Bridge is coded then it would be better still!

The PL5 “report card” still reads “needs improvement” and will continue to do so until they decide to explain in “words of one syllable” what the current mechanism looks like and what alternatives they consider might be even better or why whatever Bridge is doing is actually considered “bad practice”!

The cpu table from Process Lasso is a little scary. ExifPro has consumed the largest proportion of time followed by Creative cloud and I am not sure whether CC is actually consuming resource on behalf of Bridge or …

Some hours later during which time the machine has been unused!

@Stenis thank you for the images. I now have RAWs with no embedded xmp, xmp sidecars with a lot of data, some “ancient” DOPs and I have forced the sidecar data back into the RAW so I can create any combination that I want for testing, when I can be bothered - not testing fatigue just forum writing fatigue!

Once again, thanks for some real data which I will “mess” about with as soon as I feel up to it, hopefully later today!!

You are welcome! Hope you will see some useful patterns.

Stenis

Providing a more complete Conflict Management function:-

I had started to put this in a separate topic but I am not sure I posted it correctly and it really belongs here! I believe that it is as straightforward as I suggest below to provide a more complete Conflict Management function, I hope!?

@sgospodarenko & @alex in this topic I complained about PL5 ignoring changes made to metadata which could not be written back to the image because AS(OFF) and I also requested that the direction of the mismatch could be signalled by DxPL as it is with LR.

I originally suggested >, < or <> to indicate where the mismatch was coming from, i.e. PL5, the image or both! In the meantime I have changed that original suggestion to incorporate the direction indicators as part of the in the ‘S’, in such a case the current double-headed ‘S’ icon would signal the “both” situation.

Recent posts from DxO seem to suggest that what has currently been developed is all the development proposed for Conflict Resolution!

While trying to reproduce a fault I had encountered (as in the above identified post) I started thinking about the database field used to “persist” the ‘out of sync’ status (the ‘IsOutOfSync’ field in the ‘Metadatas’ structure) and decided that far from being a complicated development to extend to the full set of cases it is actually a very straightforward task, unless I have got something very wrong, so here goes!

  1. PL5 currently detects a mismatch situation when the metadata in an image changes but AS(OFF) means that the change cannot be captured and entered in PL5 so the ‘IsOutOfSync’ flag is set to ‘1’ and that controls the presence or absence of the ‘S’ icon from then on.

  2. Please extend the ‘IsOutOfSync’ situation to all cases where the data appears to have been changed but cannot be read or written to the image or the database for whatever reason, AS(OFF) being the main one but other situations may well arise with AS(ON)! Is there really any good reason to restrict the setting of the 'S’ynchronize icon to only situations detected with AS(OFF)!?

  3. Assign an ‘IsOutOfSync’ value of 1 or 2 in PL5 whenever an out of sync situation is detected,
    1 ‘IsOutOfSync’ = 1 when external metadata does not match PL5 and cannot be rectified by reading that data and adding to the database (the only situation which is currently recognised and flagged),
    2 ‘IsOutOfSync’ = 2 when the PL5 data does not match the external metadata and cannot be rectified by writing the data to the image. A new situation that would answer my “bug” report in PL5.2.0.4730 Conflict Resolution only detects externally generated conflicts

  4. Check the program ‘IsOutOfSync’ value with the database ‘IsOutOfSync’ value,
    1 If the database ‘IsOutOfSync’ value = 3 simply ignore the situation because it is already marked
    2 If the database 'IsOutOfSync’value = the ‘IsOutOfSync’ then simply ignore the “new” value because it is already marked.
    3 ELSE Simply add the current value of ‘IsOutOfSync’ to the database value of ‘IsOutOfSync’ and set the icon accordingly and set the database ‘IsOutOfSync’ to the new value, i.e. = 1 (image > PL5) , = 2 (PL5 > Image), = 3 both have been encountered!

Unless I have made a mistake it is basically as “complicated” as that!? I have looked at different scenarios, e.g. multiple changes to image metadata while PL5 is shut down followed by restarting PL5 and immediately changing metadata in PL5 but before such data can be changed, PL5 should have detected the earlier changes to the image and marked the images with an ‘S’ before allowing the user to access the directory.

To ‘Compare’ or not to ‘Compare’:-

To avoid potential metadata loss it is imperative that PL5 detects such changes to the image metadata in all cases. My suggestions for a ‘Compare’ function, i.e. a user initiated request to compare the metadata in PL5 with that held in the image file(s) looking for any mismatch may not be as easy to do as the icon handling I have suggested above.

I believe that PL5 currently detects potential mismatches/changes to metadata solely on the basis of timestamp comparisons. Whether any actual data comparisons ever takes place in the current version of PL5 is something I cannot test. If no such comparison code exists then there could be a lot more new work involved to provide a full blown compare function!

But, if PL5 essentially manages automatic updates by recognising mismatches in timestamps then what I am asking for is to simply “trigger” that existing process manually rather than automatically!
Having such a facility would then help close the “window” of uncertainty that follows situations I have encountered during testing, when PL5 has either not detected the change to image metadata or has ignored any changes that might have been detected!

Identifying the different mismatch scenarios:-

I like the idea of using the arrow head pointers on the ‘S’ icon to indicate the “direction” of the mismatch, they are a little small for tired old eyes like mine so it might be useful to change the icon colour to identify the three states of 1 (external image data ahead of database), 2 (database ahead of image data) and 3 (both situations identified!), possibly in addition to using 3 versions of the ‘S’ icon!

If scenario 3 is detected then some loss of metadata is going to occur, with the current implementation this situation and the risk will simply not be recognised and the user will not be informed. I consider that the whole point of using a computer product is to automatically detect such situations whenever possible and help the user as much as possible to make the “best” choice, even though it will be a compromise in this case!

Detecting and reporting Scenario 2 should focus the user to “flush” the changes made in PL5 to the image as soon as possible (or reverse the changes made), thereby helping to eliminate the occurrence of Scenario 3 because at that point the loss of some metadata is “inevitable”!

Handling the “Pop-Up” for resolving 'S’ynchronisation conflicts:-

Currently the pop-up when clicking on the ‘S’ icon highlights the ‘Read from Image’ option, if the above is implemented then the following should take place;

  1. For scenario 1 highlight the ‘Read from Image’ option.

  2. For scenario 2 highlight the ‘Write to Image’ option.

  3. For scenario 3 highlight neither option but when an option is selected by the user emphasise the potential loss of data in the database or in the image, depending on the user option chosen, and request confirmation before proceeding with the chosen option.

Again, editable metadata can be in the following carriers

  • image file
  • .xmp sidecar
  • DPL database
  • .dop sidecar

Proposal

  • Any difference should be signaled
  • Upon clicking on the signal, a window showing the difference(s) should open
  • The user should chose, which source should be copied to the other carriers

Note: Mark the image file carrier as read-only as long as it cannot be written to.

The proposed functionality goes well beyond anything I’ve seen in other metadata aware applications and could give DxO a metadata feature that were at the height of DPL’s image tools. I’d still require better database maintenance functionality though.

I suspect the DxO’s problem is that they are relying on the database being the main points of information.

To me the original file should never be written to. When you first click on a new image DxO should write a new DOP file and a new XMP file. Only then should the database read the DOP file and XMP file so that DxO can do its own housekeeping. The editing done to the image should then update it DOP file and any editing to the exif data to update that XMP file. This should then update the database. This should hopefully cause less corruption of the database and people having to delete it and no need for synchronisation on and off. Obviously the off causes corruption.
Of course, if the file has already been edited in a different programme, then it should read the existing XMP file and create its DOP file.

1 Like

Why not? Both Nikon’s and Canon’s own software write both editing data and metadata to their RAW files.

I have been writing keywords, descriptions, ratings and Finder Tags to original RAW files for years and never, let me repeat that, never had the slightest hint of corruption.

However, if I had relied solely on the database and not had DOP files in PL, I would be in right state, because of the number of times moving files (often necessary when reorganising) is tedious in PL but, if I do it in Finder, it creates an almighty nightmare.

As others have said, if you are not happy with writing to RAW files -

  • image edits should only ever live in DOP files
  • metadata should only ever live in XMP files.

The database should only be used for caching image and metadata changes whilst the app is open and should be scrapped every time it is closed.

Of course, there is still a need for a separate database for other things, but not for the two mentioned here.

But it should never copy anything from the XMP into the DOP, as it presently does.

2 Likes

The writing back to the original file is probably just one of my fads. I’ve always felt that the original file should be pristine. I know that you like to write your data back to the original file which always makes me cringe, thinking that if it ever gets corrupt that’s a valuable file lost. As I say one of my fads.

I agree image edits should only be in the DOP file and the metadata in the XMP file. Photolab should only use the database for its own housekeeping as I have already said. And I 100% agree with you. No XMP data should be copied to the DOP file.
Possibly, as somebody has already said. The database to delete itself at the end of session and make a new one at the start of the next session. Quite a bit of programming though.

1 Like

You are at odds with others who want the original RAW file to be amended, inline with the JPG, TIFF, DNG files that are changed as a matter of course by most software! Personally I agree but don’t really care either way.

I don’t understand this!? The DOP will be written as and when there are any changes worthy of note made to the image editing and/or the metadata (max delay in writing DOP of 20 seconds) and the new XMP file is incorrect if one already exists, as well it might!

Typically PL5 has access to its own database first and the same applies to the DOP but it must potentially “fight” for access with the image files! What I was suggesting in my post above is to use the extended Conflict Resolution model I am proposing above to flag any situation where the image cannot be updated with the ‘S’ icon!

I am lost again, how many genuine “corruptions” are users experiencing or are you referring to mismatches between the PL5 view (database and DOP) and the image data. Although other packages seek to minimise the risk of a mismatch there is always that risk! Without the extension to the conflict management then the product has a serious weakness.

If your proposal is to change the order of updates so that PL5 will never update the database if the write to the image cannot be successful then AS(ON) would be the way to go with that scheme because otherwise you are going to have updates stuck in “limbo” until the user finally executes a ‘Write metadata to disk’ which might never come (particularly if the user forgets the updates that need to be flushed back to the image!).

The order of updates does not need fixing what needs fixing is the following;

  1. A full flagging of ‘out of sync’ situations however/whenever they arise, information is power but ignorance is just chaos, in this case.

  2. Consider a better scheme for comparison between image metadata and PL5 metadata (I have a much bigger response underway for you @platypus)

  3. Encourage more people to use AS(ON) by not playing with the metadata
    1 Do not add hierarchical keywords to simple keywords ever!
    2 Allow users to specify no adjustment to be made to the metadata (gets complicated when users start to use PL5 even for minor adjustments)
    3 Reconsider the way PL5 adds keywords, I am no metadata expert but consider the changes made in PL5.2.0 were a big step in the wrong direction @Marie @Joanna.
    4 Do not make changes to metadata handling that leave those that used the previous version “up the creek without a paddle”. Actually start asking users, beware reacting to posts without asking for a wider consensus @sgospodarenko and provide an option to continue using the previous method!

Looking forward to it. Please keep your post short and/or make a summary of what you propose.

1 Like

I think I already said in my first post that if the XMP file already exists. It should read from that.

I personally have not had a corrupt database, but then I do not add keywords, but it seems that many people delete the database. Whether it is through past corruptions or just overcautious. I don’t know.

1 Like

I have been a user of PhotoLab since version 1. My metadata requirements are much simpler than many others users. I have been using the most current version of PhotoLab almost every day since late 2017. I never delete the database and I have never had a problem related to it.

Mark

So, you’re not bothered if any of your non-RAW files gets corrupted when you write metadata to them? :stuck_out_tongue_winking_eye:

Don’t forget that RAW files are containers and only part of its contents is the image data block. The metadata block is entirely separate. Apart from camera manufacturers like Nikon and Canon, who already write, not only metadata, but image editing data, there are also multiple apps that have been writing metadata to RAW files for years and not once have I seen a report that folks files have been damaged.

Like all the best apps, I use ExifTool, which has proved itself time and time again.

The main benefit of writing directly to RAW files is that I can use Apple’s Spotlight search engine to directly find which images contain which keywords, etc, without having to first find the XMP files and then find the RAW files from there.

I used to think I was paranoid, but now I know they’re out to get me :crazy_face:

2 Likes

I’m not taking photos to get containers for metadata. And I experienced a couple of corrupt RAW data (not only because a converter tried to write into the file), so the image was lost - not only the rating or keyword or GPS. This kind of stuff I can get again. No matter what, each edit process including altering of RAW files is a two sided risk - the image can be destroyed as well as the metadata. The latter is expendable although not increasing the fun.

I had a few damaged files…but I also got time machine backups…


…clamped to the iMac.

1 Like

And yet, every time you press the shutter, that is exactly what you are getting.

That is all going to depend on the software you used. There are some that can get it wrong and there are the rest, which use ExifTool.

And yet, you wouldn’t hesitate to let PL (or any other app) write metadata to you non-RAW files - or aren’t they as precious? I know lt TIFF scans of LF film are very precious and yet PL5 can write directly to them.

ExifTool first creates a copy of the original file, writes to that copy and, if the write fails in any way, you will find the renamed original sat right next to the original. All you need to do is delete the corrupted new file and rename the original.

Anyway, most people with any common sense keep a backup - see @platypus comment

I love the secondary “solid state” 25KG wireless hard disk. Heavy! :sunglasses:

1 Like

This container is filled or not by the camera when writing the picture data. Every additional editing afterwards comes at a risk of destroying something not reproducible with something expendable.

I’ve read enough threads about “wrong going” metadata synchronisation here in DxO forum to not trust an application which refuses to take responsibility of “accidentally” (by the app) destroyed images - like all other software also tries to exclude liability for what they do to my files. Liability is a big issue with all this gameboy apps - so many things can go wrong and the damage and it’s costs always lays on the user.

Name one single person who can’t print an image because it’s lacking metadata, like GPS-Data, ratings, colour labels, keywords, even exposure data. I can count millions who never heard of the term “metadata” and use their images.

What’s the value of TIFF scans? Compared to the LF negative: close to 0. If you lose the negative or had it destroyed, no TIFF scan will ever bring it back. What’s the value of metadata compared to one single beautiful image you bring back from a travel, from a place you only saw once in that light, in that weather, with the person or animal in it? Less than 0.

Metadata only have the purpose to give additional information, like the booklet of a CD or the sleeve of a long play record. The music it contains isn’t getting more or less beautiful just because that info is missing or wrong. But I would not squeeze a longplay record into a typewriter to write the titles, composers or artists on it.

And that metadata has no big value to the people of DxO and some other app companies show very clearly, else we would have less troubles.

Have you never added a colour or text tag to files in Finder? That changes the metadata in the file or folder.

As I reinforced earlier - if you have an up-to-date backup, whether that be Time Machine or a simple clone, you are never going to lose an image irretrievably. I have Time Machine running 24/7 and, if I need it, it takes less than a minute to recover a file, no matter what format.

And, once again, I will reiterate that ExifTool is astoundingly safe and I can find no reference to it having corrupted files.

Of course. But there are also loads of folks who couldn’t manage their collections without metadata, especially keywords. Just because you can’t cope with a concept doesn’t make it wrong.

When you’ve spent a week editing a scanned image, you really don’t want to lose all that effort. But, then I have a constant, real-time, backup, so I could care less about a file being corrupted.

Of course it will. With both a clone backup and a Time Machine backup as well, the chances of losing everything isn’t worth thinking about.

The value is, 20 years later, I can search my archive by keyword or tag or star rating and find an image a whole lot quicker than sifting through boxes of negative sleeves or even folders of images, looking for an image that I might or might not recognise.

But you could write on the sleeve with a pen or pencil. I have designed a label for my negative sleeves, upon which I write the exposure details, filters used, N+/- info, etc. But without an index, it still takes a long time to find the image I am looking for - but at least I don’t have to view the neg to see what the image contains. Instead, I use the metadata I wrote on the sleeve, which I can then transfer to the scanned TIFF file’s metadata.