I’m having difficulty deciding whether to adopt .dop files as part of my workflow and would like advice.
In Lightroom I never used sidecar files. Now that I’m using DxO Photo Lab and Affinity Photo with a touch of Photo Supreme for a smallish number of files where I need DAM it seems a good point at which to review what I’m doing regarding sidecar files.
I’ve searched for information about .dop files but have found very little. Is anybody aware of a source of information? Lightroom has so much help it is surprising me how little there is to support DxO PhotoLab. Of course I might just be missing it, I’ve only just discovered this DxO forum.
Hi Ken - - Following is just my view … Others may have different ways of working.
With PhotoLab, each sidecar.dop file is continually updated with all corrections made to its matching image-file - and with the status of the image from PL’s perspective (eg. Successfully exported, etc).
If you have these Edit\Preferences settings;
… then PL will act accordingly.
PL also maintains a database, here;
… which, as I understand it, is essentially a central store for all the same info/data contained in the sidecar files.
My preference is to rely ONLY on the sidecar/.dop files … even to the extent that I regularly delete the database file. Why?
My preference is to have all correction history directly related to each image (stored in its matching sidecar/.dop file) - so that it is safely retained even if I were to move the file-pair to different storage locations without the “knowledge” of PhotoLab …eg. using 3rd-party tools, or as part of a backup process.
I want to be able to rename my image files (AND their matching sidecar/.dop files) using 3rd-party tools.
There’s too much reliance (for my liking) on PhotoLab knowing everything about the name & location of all my images in order to properly and reliably maintain the database … I prefer the simplicity of a matching pair of {Image + Sidecar.dop} files, which I can move and/or rename as I may wish.
Then again, if your experience is with LR - and you never “fiddle” with images outside the awareness of PhotoLab then I’m sure the database method is just fine (but others are better to comment on this than I am).
Regards, John M
PS. A warning about renaming {Image + Sidecar.dop} files;
The “image-name” portion of the two file-names must always be identical.
Provided you maintain this “two file-names must always be identical” rule then it’s safest to do this with PL NOT active.
If PL is active when you rename {Image + Sidecar.dop} files then the name-change must be enacted to BOTH files (effectively) simultaneously … and that’s probably not something you could achieve with File Explorer !
The .dop sidecars are specific and usually ignored by other applications. Therefore, they mainly serve to preserve the customising of an image.
I only use .dop sidecars for testing purposes, my settings exclude their use. I also delete the database from time to time, which means that every single output image is unique. I can live with it because my main tool is Lightroom and because no two things are equal…
Thank you John for your most detailed reply. It has given me plenty to think about. I’m using a Mac but I don’t think that should make much difference.
Your point about renaming is most pertinent. I used to rename on import via LR but now am using a third party app called Name Mangler. Your warning is a most serious consideration.
The fact that PL is defaulted to using .dop sidecars when first installed suggests to me that DxO think that is the best option as do you. I understand that the PL sidecar contains all of the edits that I’ve made in PL. I presume that those edits are readable only by PL but I believe that Photo Supreme (DAM) can read ratings and a few other basic things from the .dop sidecar which might be useful. These seem to be good reasons to use the .dop sidecar.
However I know that Photo Supreme can also produce its OWN xmp sidecar files so that should the user want to move to another app then all of the hierarchical keywording etc can travel with it. Also I used to use an app called FastRawViewer which was a good way of culling photos. However that produced its OWN xmp sidecar too! Having 3 programmes that could potentially have their own sidecar file was beginning to alarm me so I retreated to having the Correction Settings for .dop sidecars unchecked and relying on the PL database which as you point out is not without its disadvantages. I stopped using FastRawViewer and am taking extreme care with my file renaming using Name Mangler although I haven’t encountered any problems with that yet.
Do you know what happens if a file has multiple sidecars?!
I’m aware that in the past I’ve run into trouble with PL where I’ve done something to the file that causes PL to refuse to read the file because presumably I’ve done something to it in a third party app that it doesn’t like. I’m unsure whether that has any bearing at all on the sidecar question but it’s something that is always at the back of my mind with PL. Simple seems to be best.
Regarding the renaming where do you place that in your workflow? I was intending doing that before I used PL but having abandoned FastRawViewer for culling I was doing the keeper / reject bit in PL with renaming occurring mid PL i.e. post edit, pre export.
That now seems to be a possibly dangerous move having considered your advice! I shall go away and give thought to what you say. Thank you for your input. If you have any further thoughts that you have would be most welcome.
Thanks for replying Platypus! The interaction with other applications is most interesting. I’m quite nervous about them as my current software choices have the ability to produce several sidecars, hence my current reliance on the PL database. Do you use sidecars in LR?
Does that mean you don’t use virtual copies or is the data related to those in the .dop file?
Yes - That’s my understanding too … because they’re PL-specific (eg. Prime Noise reduction, Clear View, etc)
If you mean in the sense that you might have one for, say, FRV and one for PL - - then that’s OK … each will have its application-specific file-extension that will be recognised/used only by the related application.
That should occur only if you explicitly change contents of the sidecar.dop file with, say, an external editor - - I very much doubt that any 3rd-party application would recognise a sidecar.dop file in such a way as to update it (?). I have never experienced this.
I tend to perform any renaming before I present my images to PL for corrections - - but I might also “tidy-up” afterwards, as well. I use FRV too - - and have no problem using it in tandem with PL … It’s a MUCH faster way to review/cull your images than using PL (which is, in part, why I don’t have much interest in PL having DAM capabilities).
A most reassuring reply. Your approach seems to be fairly laid back and flexible which suits me just fine.
I’ll keep a look out for any problems reading files but again shall relax about my concerns. I can’t really remember where they originated, I think from DxO originally but I might be wrong.
Excellent. I’ll restore FRV. I’m sure you’re right and I know where to come if I encounter any problems!
I’ve done some quick previewing with FRV and NO problems so far as you suggested might be so. No sight of a sidecar file which is strange as I was being inundated with them a month or two ago.
Well that appears to have solved my initial review part of my workflow so a big thank you John
Every now and then, I save the sidecars in Lightroom.
I rely on Lr’s reliability and make regular backups with Time Machine and a clone every few weeks.
As for the .dop sidecars, i mostly don’t use them but found it interesting that they also contain informations about virtual copies. I don’t regularly use virtual copies though.
When I was using Lightroom I didn’t use sidecars as I was advised that they could cause problems and that unless I wanted to move files to other Lightroom installations on a regular basis I would be better off backing up the LR catalogue regularly. As I was working alone on one machine I never went the sidecar route. That was back in the very early days of Lightroom (I started with LR) and may have reflected the development stage of the software at that time.
My questions here on the forum are an attempt to understand ‘why?’ I’m doing things as I develop my new workflow.
It’s interesting what you say about the .dop files carrying version history. I haven’t tried it yet but I gather the Photo Supreme DAM that I occasionally use can read some parts of the .dop sidecar and as Photo Supreme manages virtual copies for different purposes e.g. email, web etc, that might be a feature I’d consider using in the future. As I would only use that feature occasionally (if it actually exists) I might be better to just produce the .dop files needed at the time.
I’m becoming increasingly interested in the idea of NOT keeping my edits, simply producing output I’m happy with as JPEGs, TIFFs etc. and if I want to work on a file again in the future just start afresh. After all when I stopped using LR I was unable to take my editing with me along with the RAW files. Having been steeped in the LR DAM environment for what seems to be a very long time there is a sense of freedom in thinking that I could make life a lot simpler and perhaps spend more time taking photographs. I know that no DAM wouldn’t suit everybody but perhaps it would suit me. I’m still thinking!
…or try to see what benefits you have from (not) using the sidecars.
I started my photographic journey with chemical photography. No sidecars, no database. If I wanted to keep a “recipe” I had to make a drawing that illustrated the steps. Never did, but then again, I am not Ansel Adams.
Ah yes, I do think I need to have an open mind at the moment. I’m still not clear about the two alternatives:
OPTION 1. using sidecars 100%. Simple. Advantages, well I don’t know at present.
OPTION 2. saving my DxO edits in the DxO database and just producing a sidecar when I need it. Slightly more complex. Advantage ‘out of sight, out of mind’. Disadvantage ‘out of sight, out of mind’ means I might not use them. Hmm…
As you say using them should clarify whether they are the way to go. Do your sidecars ever get detached from their associated RAW?
I have spent more time with FastRawViewer and can see that sidecars are created when I add amp metadata. When I open DxO PhotoLab then PL reads the RAW files perfectly as John-M suggested BUT doesn’t read any of the ranking data in the sidecar. That gives cause for thought.
I remember going to an Ansel Adams exhibition some years ago. I think it was him who said that 12 good photographs in a year would be a good haul. More cause for thought!
A friend had the materials and a darkroom space and I can remember still the images appearing out of ‘nothing’. Sadly I couldn’t afford to follow up my enthusiasm. One day …
The benefit of sidecar files is that they help prevent any alteration (or corruption) of the the original raw files while storing information needed by applications such as DxO PhotoLab.
Using sidecar files introduces issues of managing images and sidecars together. This is best handled using some kind of DAM (digital asset manager) that can automatically and transparently keep track of these sets of files. (I use IMatch 2018 https://www.photools.com/ as my DAM, which I find very effective.)
Different vendors take different approaches to sidecar files. I believe DxO has the correct approach of using a specific type of sidecar (.dop) for storing its own data for processing individual images (including virtual copies).
Adobe established the .xmp sidecar file to store more general information about the image, including metadata (descriptions, ratings, copyright, etc., etc.). The Metadata Working Group has issued specifications and guidelines http://www.metadataworkinggroup.org/specs/ on how xmp and other metadata should be managed. Any application that handles metadata should be in compliance with these standard/guidelines.
Unfortunately, although Adobe established the .xmp standard and released it for use by others, they have also made their own proprietary variations (e.g., Lightroom includes image processing data in the .xmp sidecar files). This creates a number of issues including making larger .xmp files and potentially introducing incompatibilities with other programs. I believe some other vendors are guilty of similar issues although I don’t have any direct knowledge of this.
DxO PhotoLab (and Optics Pro before that) reads Ratings from .xmp files (shown as gold stars) and also carries metadata from the .xmp sidecars over to its exported files (e.g., .jpg files). This was an excellent decision that makes PhotoLab a useful and valuable tool that complies with MWG guidelines and helps establish and maintain vendor-independent workflows.
A benefit of using .dop sidecars (assuming the user has a DAM or other way to manage groups of related files) is that it provides more flexibility in structuring and rearranging file structures or hardware as needs change. It also eliminates a single point of failure should something happen to the PhotoLab database.
Hello jch2103. Thanks for your detailed reply to my query. Apologies for my delay responding.
Yes. I can see that writing data to an individual .dop file is potentially better than writing to a single PL database. Corruption of database much more serious than corruption to single .dop file. Is that correct?
I have Photo Supreme as I’m on a Mac and believe that iMatch 2018 is Windows only. I’d looked at iMatch seriously before discovering the limitation, looked impressive. Is your workflow to import files to iMatch before you develop in PL?
I have been confused by multiple .xmp files so it makes sense to me at a personal level.
I’d heard that this was possible but when I’ve tried to do that between FastRawViewer and PhotoLab the image ratings don’t appear to feed through from FRV to PL. I assume I’m doing something wrong but exactly what I don’t know.
This is a very big plus. When you’re moving files around in your DAM software do the sidecars move automatically or do you have to be proactive in making sure the pairs stay together?
Any corruption (or any other major problem) in the database affects all of your files. Of course, you have multiple backups of everything anyway, right?
Yes, I handle files in IMatch (adding metadata, adding ratings, etc.) before sending them on to PL for processing. The resulting sets of files (master raw and output files) I also manage in IMatch. Makes life much simpler. By the way, at least some IMatch users Macs with a Parallels / Windows10 platform.
I’m not familiar with FRV, but perhaps there’s a setting that assigns ratings to .xmp sidecar files?
Exactly. That’s the benefit of a DAM. You may of course need to do some minor customization of file handling if your workflow differs from common practices.
I overdid the backups originally. Many drives, multiple copies of files, began to be a bit of a nightmare. Getting it back under control has given me freedom to get out with a camera more now. But yes, backups still essential.
Photo Supreme seems to be working well although currently I have only small sets of photos to test as I move away from LR. I’ll look into iMatch again in the future if I have problems. Yes, Parallels is an option I hadn’t thought of.
Well… I think I’m applying ratings to sidecars but shall go back and check that I really am!
As I’m moving to a new DAM I’m at a good stage for switching to sidecars hence my original question.
