Write dop files less frequent, in predictable order, with control over virtual copies

Currently, DxO Photolab writes .dop sidecar files for no apparent reason and with entries in unpredictable order, and it creates virtual copies if dop files are changed externally.

This makes it hard to work with synced directories on two different computers and/or with backups or version control software.

My request is to:

  1. Write (modify) .dop files only if really necessary
  2. Keep entries in a constant (predictable) order to enable comparison
  3. Let the user control whether he wants to create virtual copies after remote edits

Examples:
Since Photolab is a resource hog, I process larger sets of images on a powerful desktop machine. Then I copy the results back to my portable main PC. When I only visit the folder later on this second machine, Photolab re-writes all dop files and puts the entries in a different order so the version control software flags all files as modified and I can’t check easily for real differences (e.g. with a diff tool).

In my opinion, Photolab shouldn’t even change the file modification time unless there is a change of settings to avoid uncertainty about the date of the last change (e.g. whether the exported image is current).

But even if settings are modified: Why on earth do you change the order? The text format could be so useful for a quick check what has been changed between two revisions, but by mangling the file on each write you can’t have a quick look with your favorite diff tool (I recommend Beyond Compare).

If I make then modifications on one of both machines and copy/sync them to the other, things get even worse: Photolab creates virtual copies. This seems to depend on the storage type (removable or network drives don’t get virtual copies?). The same applies if I want to revert changes by restoring dop files from backup or SCM. It’s very hard to get rid of this virtual copies: Which one is the old and which is the new one? And it is error-prone to delete a hundred of copies.

It’s a “nice try” to guess the user’s intention depending on the drive type (removable or network), but it can fail as I demonstrated. Please give the user control over the behavior.

Find attached two dop files containing the same settings, if you have no diff tool, use http://mergely.com/editor or similar.

Oliver

before.dop.txt (7,4 KB)

after.dop.txt (7,2 KB)

Hello @obetz

Actually it’s all under your control - you can modify the Preferences and get exactly what you want:

And then export/import the sidecars manually when you really need it:
export_import

Regards,
Svetlana G.

Do you suggest I should set the settings as your screenshot shows or disable automatic read/write?

I showed you the default state. If you need to control the sidecars creation manually, you should disable them and then create the sidecars manually as displayed on the second screenshot.

Regards

Creating the sidecars with an explicit (manual) step would be even more error-prone than the current behavior.

You would lose more often the information about the time when the settings were edited last.

With manual export, it would be yet harder to work with synchronized directories or to manage dop files with SCM or backup software.

For me, the sidecar file is the main storage place. The image and the sidecar are a pair being handled by processes and tools outside Photolab, e.g. move, rename, backup.

Oliver

1 Like

I experimented with working this way - but, unfortunately, PL does not provide any warning before shutting-down if changes were made that have not been written out to a sidecar file (such as, say, Word or Excel would do on termination if a document had not been saved).

John M

Hello John,

Yes, that’s true - with manual sidecars saving the latest changes are not saved with closing the application until you do that manually.

We’ll think about it

Thank you
Regards,
Svetlana G.

1 Like

automatic save seems to be the natural way to me.

Please reconsider my initial suggestions, I think they are well thought out (over several months) and not that hard to implement.

Oliver (main occupation firmware engineer)

Yes, this is also a DAM topic: How to detect and handle differences between database and sidecars?

Normally, if the user deactivates auto write back for metadata into sidecars:

  1. An indicator is shown on the image tile, if write back is pending for that file
  2. A ‘write back pending’ category is offered, where the user can see all writeback pending files at once.
  3. A command is offered to write back all pending files in addition to the manual write back command for the selected files.
  4. An OPTIONAL warning on application shutdown is offered, to notify about pending files with the possibility to execute 3).

This was the direction database -> sidecars.

In the oposite direction there should be an indication, that the sidecar has changed externally and the database is out of sync. Here the user can use the ‘Import sidecar’ command. The auto read of sidecars should be OPTIONAL after they were read once into the database, to not destroy the database content by unintended external changes.

this sounds much more complicated than my suggestions.

It may sound complicated. But it works this way (more or less) in tools like Lightroom, Acdsee and IMatch. No need to reinvent the wheel IMHO.

Yes, I’m agreeing with @asser on this one - as follows;

Also, @obetz’s suggestion for consistency in order in which correction settings are saved to the sidecar/.dop file (from one version of the same sidecar/.dop file) has merit too.

John

I meant “complicated to implement in Photolab”. Since DxO already “invented a wheel”, the question is whether they should (or want to) fine-tune it or change it.

After all, I’m not that happy with the catalog-centric workflow of Lightroom. Putting everything in a single database has not just advantages.

A central database is mainly useful for the DAM function, but Photolab is no DAM (and I doubt it should be unless the developers are bored because they have no more challenges in the editing area).

Imagine a solution without central database, only with sidecar files. No need to import/export. The edit information is always in the sidecar file and you need to care only about the image+sidecar pair.

Backups and moving around in the folder structure (or between computers) is much simpler this way: No inconsistencies, no duplicated settings etc. You can then use synchronization tools, cloud storage etc. without any problems.

Anyone who tried to use Photolab with a cloud synchronized folder knows what I mean.

2 Likes

Yes, I understand you fully. The sidecars are also my way to store information. Database/catalog is used only for file and metadata indexing for fast search and recursive file enumeration over hierarchy trees. There is no problem, if I loose my catalog.

The virtual copy creation problem I have reported long ago. I was told, that it is expected behavior.

My proposal above was under the aspect, that people request more DAM in the feature request section and support for XMP round tripping. If you can read/write XMP, DOP becomes obsolete.

Well, if we customise and recustomise images time after time, we can either get one image with all edits or several snapshots, one each per customisation round.

Customising images on different computers usually means that you work with different databases too. Therefore databases get out of sync.

As of now, DPL shows no history/protocol of edits. This has been requested earlier, but has not found its way into the product yet. Even if we had protocols, we’d need an additional function to sync databases. Or sidecars containing a protocol of all edits which could then be displayed in the protocol. Whether the protocol would need a timestamp per entry or just a separator between sessions seems small matter compared tho the overall change.

If we consider such a change and add DAM to the mix, the database would be the one centrally important element of management and you’d certainly NOT want to loose the database then. Deleting the database to work around an issue would then be a no-go.

Sounds really complicated. For me all changes to a single image are atomic. No merging for a single image needed.

The timestamp of the last change indicates the sync direction.

Timestamp inside sidecar > timestamp for that file inside database, database is out of sync => indicate, allow replacement of database content with sidecar content after confirmation through user (External change use case)

Timestamp inside sidecar < timestamp inside database for that file, indicate writeback pending for that file.

All databases can still be lost, after sidecars were updated. Last timestamp wins. No merge, just replace.

1 Like

@Asser, sorry for the late reply. DxO told me also that this is the expected behavior, but this is only half of the truth:

If the files are on a removable or network drive, no virtual copies are created but the external changes are silently integrated (if I remember correctly, based on experiments last year).

This means that the choice is already present in the code but the DxO developers didn’t consider use cases where local volumes get remote edits (e.g. cloud sync software).

I just want a user interface to control the creation of virtual copies on local drives.

(and no unneeded writes and consistent order of entries)

Oliver

I never want virtual copies to be created automatically through external changes. Virtual copies should only be created, if I request it through command execution in the (context) menu.

This problem also occures after rename on local disc, with the following sequence:

  1. Open A.RAW+DOP in PL.
  2. Close PL
  3. Rename A.RAW+DOP to B.RAW+DOP
  4. Open B.RAW+DOP in PL
  5. Close PL
  6. Rename B.RAW+DOP to A.RAW+DOP
  7. Open A.RAW+DOP in PL

=> Two virtual copies
=> External rename is unsafe in general

then you might want to spend a vote

I am out of votes currently, and I have adapted my workflow, so that I never have to rename or move a file. So this bug/expected behavior is under control.