Missing DOP files

I have a problem with DOP files which are going missing. I first noticed this with PL5.5, so I reinstalled 5.4 and it went away. I have noticed it again with 6.0. Computer has windows 11 latest update.

When working in DxO, first I import raw files (CR3) into a single working folder which I use for all processing, located on the C drive. Files are culled, and the ones I want to keep are processed in DxO. I then export high quality JPEG files into one of my long term storage folders, located on a separate internal drive (labelled D). Finally I move (drag and drop) the raw files and the DOP files in Windows File Explorer from the first C drive folder into a third folder, also on the D drive. Thus, I end up keeping both an original raw copy, the adjacent DOP (in the same folder as the raw), and the final processed JPEG (separately).

My problem is that when I drag and drop raw and DOP files from the C drive to the D drive, Windows starts telling me with some files that it cannot find the DOP file to move. I end up losing at least some DOP files. If I try and reopen the file in DxO again, it only shows the original raw without any changes.

It seems that DxO is creating dop files, which exist for as long as the file is on the C drive, but is perhaps not always properly saving them. Does DxO save DOP files continuously or only when you close the program? I am wondering if the problem might be that I sometimes move files before DxO is shut down (because I want to continue working with it) so it doesnt have the opportunity to write the files to drive. However, DOP files are always visible in windows explorer on the C drive before they are moved. Alternatively, is there some step I should be taking with the latest version of DxO to force it to write the DOP file to drive permanently, before exiting the program or moving the files around in Explorer? Or is there a way to transfer the raw plus DOP into the final storage folder from within DxO itself, which would solve the problem? I never had this problem with earlier versions of DxO PL, which I have been using for years.

This is not the end of the world as I still have both the original raw and the high quality JPEG (and often I don’t look at the raw file again). However, it is frustrating to have to re-process files on the occasions that I need to particularly with lots of control point changes etc.

Any help appreciated.

Thanks
SP

…better do it within DPL. It should make sure that the couples are moved or copied correctly and it will also keep the database consistent.

If you regularly delete DPL’s database, you can also use Explorer to move the couples. Keep in mind that this might lead to some loss of metadata (not all that is stored in the database is transferred to the DOP files), but I don’t know the exact situation with DPL on Win.

Others have reported the exact same problem here in the forums. What I do is close PhotoLab or change its working directory before moving the camera and sidecar files.

@SimPel your problems doesn’t make a whole lost of sense for the following reasons

  1. DxPL writes DOPs at a maximum of every 20 seconds or so we have been told!
  2. It also write DOPs on close.
  3. You can force a write at any time using
    image
    and typically it will warn you

    If memory serves me correctly it does this even when No DOP exists!?

But the above forced write should not be necessary and you start out actually dragging the file plus an existing DOP only to discover it has gone when you get to the destination!? But you have explicitly stated that it has always worked for you in the past but not on PL5.5 and PL6?

Please do me a favour and perform the following tests;

  1. Close DxPL (i.e. no DxPLs of any version open) and do your standard procedure and see if the problem occurs.
  2. With DxPL open, and remaining open, force the DOP write as above and perform the procedure and see if the problem occurs
  3. With DxPL open, and remaining open, perform the procedure and see if the problem happens!

Then we can see if we can “surround” the problem.

Regards

Bryan

@SimPel

as @platypus said, the best is to move things within PL, and everything from the database will also be kept.

Otherwise, close PL before you move data ( don’t steal from a running program :frowning: while it tries to keep the database in sync ).

@Wolfgang certainly wise advice but exactly how with DxPL(Win) do you move data within the product!?

The is a command to ‘Create Folder’

image

I can copy but only by drag and drop (as far as I can see)

Dir 8:-

Options available

Dir 9:-

Dir 9:-

After Drag & Drop

The reason that moving may not be available (may because I can’t find the command doesn’t mean it does not exist) is because of the amount of work necessary to delete all the data elements from the database and re-introduce that data elsewhere!

@platypus(Mac) and @Wolfgang(Win10 please show me(Win10) where the command is placed, and yes I agree with the issue of not moving data with an active copy of DxPL using it but somehow snatching a DOP from within an operating system Move command as described by @SimPel seems very strange.

click / hold Shift + drag & drop = Windows basics

@Wolfgang to be honest I have rarely used Windows commands directly, I use FastStone Image viewer to copy and move Images, Xplorer 2 and FreeCommander (Lifetime Edition) for all the basics alongside utilities added to the right mouse click, and Beyond Compare to keep everything under control and check everything is where it should be!

But I have obviously led a sheltered and protected life although it has never held me back, until now. I can hack the DxPL database but not use a primitive OS command to move files!!

Actually I use that to move messages to folders in NEO, an Outlook add on but …

Thank you for the information!

Thanks all for your responses. I am trying Bryan’s suggestions and a few other things to see if I can get a reproducible set of results, and will report back.

On a slightly different question (but related) I was not aware of the database and its function until this email chain (I had seen it in the File menu, I guess, but had not really taken any notice). I naively thought that DPL simply stored changes, metadata etc in the dop files.

Is someone able to explain what the database does, and how does it relate to dop files? If I delete a dop file, does DPL still have all the relevant data in the database and can it be recovered? Also, what is best practice in terms of using the backup (which I have never used), given that I keep and routinely back up the dop files?

Thanks again

I have now isolated the issue, so I think I know what happens, although not why it happens.

I have discovered there is a difference between copying (drag and drop), and cut-and-paste. I incorrectly stated in my first post that I was copying the files whereas I think I was actually cutting and pasting on those occasions when I noticed the problem.

To test, I started with a set of new files (not ever opened in DPL before), did some random processing applied to all files in the Customise tab, and then went to Windows file explorer to move the files (raw + dop) to a new folder. If I used the copy function the files transferred without issue, both raw and dop. Re-opening the files in DPL in their new directory gave me the edited version - ie the files had copied correctly. I tried this three or four times and was successful each time with DPL open or closed.

However, on one occasion today I accidentally used the cut-and-paste function, and immediately started getting the Windows ‘cannot find the file’ error message for several file pairs when pasting into a new directory. The final result was that all raw files were cut-and-pasted fine, but only some dop files. I cannot see any logic in which dop files were moved, but (to pick one example) I would have 15 raw files cut and pasted but only 8 dop files, and the lost dop files completely disappeared from both source and destination folders. I tested this several times with DPL open and closed and it only was a problem for me when I cut and pasted files in Windows File explorer when DPL was open.

So obviously there is some issue associated with dop files that causes errors when cutting and pasting, at least when DPL is open and the relevant folder is selected in DPL Customise.

This may not be worth worrying about further, and may be more a Windows problem rather than a DPL problem - I’m really not sure. The important precaution seems to be (as others have said) to only move files after DPL is closed (or move from within DPL), and to only use the copy function not cut and paste in Windows file explorer.

SP

…have you used copy-and-paste instead of cut-and-paste to see if it makes a difference?

@SimPel The database is arguably the “beating heart” of DxPL and the DOP is essentially an external copy of the database contents with respect to an image. Both the database and DOP can survive without each other but if DxPL has a database entry but no DOP it will typically re-create the DOP!

Providing that an image is not already in the database then presenting an image with a DOP will result in the DOP being read and used to (re-)create a database entry. There are potential issues surrounding this process when there is already and entry in the database, which I can cover if you wish, but I need to keep this post short to get on with clearing up after some DIY!

The metadata is stored in the database with a copy in the DOP and will be transferred to and from the image automatically with AS(ON) and manually with a ‘Files’/‘Read from image’ and ‘Write to image’ with AS(OFF) (and also with AS(ON)).

AS(ON) and AS(OFF) are my shorthand for

This would be AS(ON)

and manual transfer by

image

Database saving and Reloading:-

It’s actually pretty simply; PL is “tidying up” - and it’s very quick to do so !

  • As soon as it detects an “orphaned” sidecar/.dop file it deletes it - This occurs when File Explorer happens to have moved (= Copied & Deleted) the .RAW file before its associated .dop file.

  • Sometimes, you’ll find that both files are moved successfully - This (lucky result) occurs when File Explorer executes the move the other way around (sidecar/.dop first).

The solution is to always move images around from within PL - - OR, only move {RAW+Sidecar} combos from outside PL when PL is NOT running.

Note: The irony of all this is that, in the past, there were user complaints asking "Why do I have all these orphaned sidecar files lying aroundWhy doesn’t PL clean them up properly !?! " :smiley:

John M

Yes - by ‘copy’ I mean copy and paste (either control+c / control+v or using the icons in File Explorer). Cut and paste is ctrl+x and ctrl+v (or using the File Explorer icons).

Thanks John-M and Bryan for helpful explanations.

@John-M and @SimPel I believe it is possible to avoid the problem simply by not having the directory involved in the moves selected (‘in scope’) in DxPL, i.e. either simply navigate to another directory or have an empty directory where you always “Park” DxPL when exiting the product, the latter is useful for testing when you want to return to a directory only when you choose not when DxPL is re-opened.

DxPL is waiting on an interrupt from the operating system that alerts it to a change in the directory ‘in scope’ and whatever files are within that directory, given the speed of reaction to a ‘Rating’ change with AS(ON), DxPL is very quick to detect and quick to react!

I have only previously seen the problems of this speed with extraneous DOPs when changing metadata via Photo Mechanic in JPGs. Photo Mechanic does this by creating a temporary JPG first before deleting the old one and then changing the name of the new one to the original name.

DxPL spots the change in directory status immediately and “sees” the temporary file and creates a DOP for the temporary file, which it fails to clean up when the temporary file vanishes just after! IMatch deliberately waits to avoid such problems but as a consequence appears slow and ponderous when I use it for tests and I feel that its delay period is too long.

This is fairly standard practice and it is certainly what ExifTool - the leading metadata tool that a lot of apps use internally - does.

Of course, it also does the same for any image file or associated XMP file.

For example, editing the file _HLN0032.NEF with ExifTool creates a same named edited copy and also renames the original to _HLN0032.NEF_original

This is why I have no qualms in placing metadata directly in RAW files.

But, if you specify the -overwrite_original_in_place parameter, that *_original file is deleted as soon as the edited copy has been verified.

If PL is reacting that fast, it means it makes it very unfriendly to most metadata tools, unless those tools take the risk of editing a file in place without a security copy.

It would be interesting to know the strategy and tool DxO uses.

@Joanna don’t think that DxPL uses any external tool but may well use a library such as Exif2 perhaps.

I have done tests with IMatch setting metadata, and IMatch uses ExifTool for changing images and DxPL has always worked well with no extraneous this or that!!

But with Zoner (my license has now lapsed and I am not renewing so I cannot reproduce what happened with Zoner and DxPL) there were missing images in DxPL because of the weird way that Zoner went about a metadata update, i.e. DxPL had detected the actual image which Zoner then deleted!!

Having made the comments above I felt duty bound to verify my statement so we have “three little JPGs sitting on a wall …” sorry 3 JPGs in a Directory ‘in scope’ (as I’ve chosen to describe this)

image

with FolderMonitor also “watching” the directory! The above is the result of the first test where I set 4* in Photo Mechanic and no problems whatsoever and I wondered what had changed!?

So in PM I set JPG2 to 5* then moved to JPG3 and thought it should be 3* and then went back to JPG2 and set 2* and then returned to JPG1 and set 1*, and in the directory I found only one “rogue” DOP

and an annotated log from FolderMonitor shows

Which I find a weird order but …

With most programs there is no issue, but with JPGs and PM you occasionally get some detritus possibly because of timing issues and FolderMonitor participating might actually decrease the chance of this happening!