Option to preserve metadata in XMP sidecars

Yes, you are talking about XMP writing (rating is part of xmp sidecar), we should thinking to covert all (or max) cases possible. Something like:

  • What happens when we have virtual copies? and rating is not identical for each virtual copy. In XMP sidecar, we can store data for only one image. (therefore in the previous EA, we set the Master image notion, this can help workflow to overwrite XMP sidecar.
  • We should add preference options and let user decide for automatically overwrite XMP sidecar or not

We will keep your feedback and fine tune it when we implement this xmp writing feature
Thank you,
HT

1 Like

Those star ratings are primarily for rating incoming images. Using selection ratings to rate virtual copies should be discouraged. Virtual copies should employ some other kind of rating or comment system (colour labels?). If I were the UX designer, I’d lock the ratings on virtual copies to the rating of the master file to avoid the issue

This an unnecessary and confusing preference. If a photographer has chosen to use XMP and DOP sidecars the XMP rating should always be overwritten.

Consistent behaviour is often more useful to the end user than flexible behaviour. Steve Jobs stressed that it’s the developer’s job where possible to make intelligent choices for the end user. Decisions carefully made save users time.

The deeper issue here is the conflict within DxO to make PhotoLab some kind of monolith program like Lightroom which demands that every other program be subordinate to it and runs off of a huge database or to carry on with a RAW developer which interacts well with other software and stores its data within the file system (.dop and .xmp files). Iridient Developer is an example of clearly the latter. C1 is an example of the former. PhotoLab doesn’t know which one it wants to be and has been walking the fence for the last couple of years.

For most existing users (we’ve all paid a pretty penny for the privilege), playing well with others and compatibility with other software and storing settings separately from a database are extremely important. The Lightroom users (most of whom would never drop €200 in a single purchase for any software, particularly if it doesn’t have the name Adobe on it) targeted by DxO marketing would probably prefer the monolith.

The question is does DxO throw its existing users to the wolves in the pursuit of potential users in the hopes of attracting users who 1. would not pay that kind of money for software 2. are mostly Adobe die-hard junkies 3. the last people to leave Adobe software behind. Never forget the existing Photographer’s Package includes a full copy of Photoshop. Phase One just had a go at pulling users out of Adobe, with an extended half price sale to bring in Sony and Fuji users. I’m curious to know if they made a dent in in the Lightroom userbase or not with that big promotion. Unless there are plans to radically change pricing and positioning, siding with the existing paying users would be both ethically better and commercially wiser.

DxO PhotoLab’s situation with its userbase reminds one of the husband of ten years abandoning his exquisitely beautiful and scintillating wife to chase after a common or even vulgar young nanny. Of course this illogical situation happens almost every day. I.e.: Jude Law and Sienna Miller or d’antan Hugh Grant and Elizabeth Hurley.

PS. The above is written with a smile, not a frown. To amuse, not to offend.

4 Likes

Agree with Alec:

  1. Virtual copies shouldn’t have separate star ratings, and
  2. changes star ratings should automatically be written / overwritten as XMP to the dop sidecars.
2 Likes

I disagree. A virtual copy in my workflow can be a totally different picture (different crop, perhaps B/W, …). Often I export even two versions of the the same source, but they may have different ratings from each other and from the other versions that I don’t export.

3 Likes

I would like to have the possibility for PL to Read only XMP files because PL is not gonna be my DAM anytime soon and I do not want any software to mess with the XMP, especially as long as you are debating those topics. I do NOT want to be an XMP feature beta tester :nerd_face: Thank you.

3 Likes

Marc, If you don’t make changes to XMP properties (ratings, keywords) in PhotoLab, there’s no reason for PhotoLab to write to XMP. If you are making changes to XMP properties, I’m afraid you and everyone else like me who do not use PhotoLab as a DAM do want those changes written. Otherwise we don’t have any kind of sync with our existing applications.

Your point that DxO should proceed with considerable care here not to damage existing data (zeroing XMP files for instance) is well taken.

2 Likes

Since I wrote the note below, I have done bit more exploring.

  1. I find that when you export a sidecar to a Nikon raw image, the exported file name is “filename.NEF.dop”. Although this file looks very much like an xmp file, Exiftool indicates that it is not a xmp file but a txt file.

  2. I have tried every combination of settings I can find but I cannot find any way to transfer color settings, star settings, or any other metadata between dop files and xmp files. this means that I can find no way to use Photo Mechanic to set metadata and then import it to PL3 or vice versa. I may be missing something, but I just cannot find a way.

  3. Metadata that is embedded in a jpg file by Photo Mechanic 6 Plus does not show up in any way in PL3.

This is all very confusing. I hope the developers might be able to shed some light on this. However, I find it disconcerting that it is so difficult to use other programs that all interchange metadata by xmp files that do not play well with PL3.

1 Like

I too have been very confused by this discussion. However, after digging around a bit, here is what I suspect is happening.

  1. All of the external programs/apps mentioned above read and write the metadata for raw files to sidecar, usually with an extension xmp. The format of this file is an industry standard ( Since early 2012, the XMP specification part one, which defines the XMP data model and core namespaces, is an open, international ISO standard (16684-1).) This is how Photoshop, Lightroom, etc. handle metadata. I use Photo Mechanic 6 for my DAM and have the preferences set so that the metadata for raw files are stored in such a sidecar. It is possible, but not a good idea, to embed the metadata in the file. If you do this, it is my understanding that PL3 will not process the file. Metadata for non-raw files (e.g., jpg, tif) is always embedded in the image.

  2. If the preferences are set, PL3 creates a sidecar with the extension .dop. This file turns out to also be an xmp-formated file. You can verify this by using Phil Harvey’s command line tool Exiftool.
    (the simple command to display the xmp data in a file is ‘exiftool filename.ext’. This file is thus a second, separate, and DxO proprietary xmp file. (I note the both files are, in fact, txt files, so you can read them in any text editor.)

  3. What is happening in the discussion started by Brian is that when he goes to IMatch and changes the number of stars, the information is stored in the industry-standard xmp file described in (1) above, while PL is reading their xmp file described in (2). Hence, PL does not see the change. And, vice-versa, when he changes the stars in PL, IMatch cannot see the change.

  4. The consequences of this disparity are significant. For those who use a cataloging program external to PL (e.g., Neofinder, Photo Mechanic 6 Plus, …), to the best of my knowledge they index only the xmp files described in (1). I have not looked in detail at how the metadata that one enters in PL is handled, but some of these data are set up to transfer to Lightroom, but certainly not all. Also, if you need to store IPTC data, as far as I can tell, the is not done in a dop file, but is fully supported by the programs that write xmp files. Caveate–I may misunderstand some of these details and certainly may stand to be corrected.

  5. For my personal workflow, I turn off the creation of the dop sidecar and rely on saving all of my parametric edits in the PL database. I do all of my classification and cataloging in Photo Mechanic 6 Plus (still in beta test as of this writing, but available to all customers). I can then find the images I want quickly and easily. Many readers of this forum do the same thing using other DAM systems as well. Once sorted and selected, I create an empty project in PL3. Then, in Photo Mechanic, if you right click on one of the selected images, you get a choice to ‘Edit Photos With …’. I have set up DxOPhotoLab3 as one of my choices (you get a pull-down, so you can add as many choices as you wish). One click, and all of the images show up in the project perfectly. As noted elsewhere in this forum, you may be able to drag and drop the selected images in your DAM onto an empty Project in PL3, but I have not tried this.

Since I back up my work nightly from my working drive to two separate external drives (a primary and a secondary backup), I also want to backup all of the parametric data in the PL database. Thus, I also copy this database (as noted elsewhere in the forum, there are a variable number of database files–anywhere from 3 to 5 have been documented) to a folder on all three of my hard drives. (My working drive only has images and their derivatives–all of my programs, apps, word processing, etc. are yet on a separate hard drive.)

Hope some of this helps in understanding what appears to be happening with PL and metadata.

1 Like

Hi,
I can’t remember what this thread was all about, but after a trial and some reading about Photo Mechanic I can say, that you can configure it to save data to the XMP sidecars in a way, that Photolab will be able to read it.

Following topic of the Photo Mechanic documentation page shows the settings necessary for compatibility. Set it for Adobe Products and you should be fine.
Photo Mechanic documentation - Settings for compatibility

You can only use the star rating and the keywords only.
Unfortunately (at the moment) it is a one way street: settings in Photo Mechanic will show up in Photolab after you saved them.
You will not see the color labels as Photolab doesn’t support them.
Unfortunately if you change a star rating in Photolab or add a keyword it will only be saved to the database (or the star rating also to the .dop sidecar) so normally you shouldn’t do it. It will hopefully change to something more useful in the future.
If you export the edited photo, the exported file will have this information too.
The pick/reject (green/red) colors in Photolab are only useful inside Photolab.

In my experience the .dop sidecars are more useful and easier to backup.
These are not XMP files. You probably confuse XMP (a specific namespace inside XML) with XML (a markup language format). The .dop seems more like a JSON, but is rather something else. The categorisation as a text file from Exiftool is correct.
It might only be in the Photo Mechanic Plus Beta, but they were also talking about handling the .dop sidecars for renaming/moving.

Photo Mechanic handling of dop sidecar

You will not really be able to move information from a .dop file to a .xmp file unless you write some special program for it. But if you only change data outside Photolab and use above mentioned settings, you will be able to see them inside Photolab.
I definitely see changes made to star ratings and keywords in Mylio and IMatch inside Photolab.

1 Like

Hi Zoltan:

Thanks for your comments. As far as I can tell, all of my PM6 settings are for Adobe compatibility as you suggest. I still have the issues as I noted.

Regarding PM6, Camera Bits allows you to associate dop files with images that they support, but so far as I can tell that does not mean that they support transfer of metadata with those files. It just means that if there is a dop sidecar in the folder with a raw image, the dop file will not be shown with the thumbnails in the contact sheet.

Bob

Like Robert, I would really like PhotoLab to write back any changes to ratings.

In my workflow I rate all my files before bringing them into PhotoLab (just contemplating the slow tedium of rating files one by one in PhotoLab makes my eyes water), in my case in FastRawViewer. That said, among four star and five star files I view in PhotoLab, I’m often tempted to change the grades either up or down. It’s frustrating not to be able to do so.

The way I work around it @RWHendricks is to do a final pass of four and five star images within PhotoLab adding the green pick flag to photos I’d like to process and export. I play with the order when I apply the flag. The green pick flag is really for photos I’ve processed to the end and would like to export. Poor picks revealed while editing (limitations of the RAW file usually) get a red reject sign. Of course none of this makes it back to FastRawViewer (or PhotoMechanic in your case) to my great frustration.

How easy it would be for PhotoLab to apply any changes to star ratings or keywords to sidecar .xmp files, just as it reads them when ingesting a new folder.

I certainly would not go without the PhotoLab .dop sidecars. Any database corruption or a folder move gone astray and you’ve lost years of hard work grading your photos. Take advantage of what DxO does offer us – backup of our corrections work alongside the original image files.

3 Likes

I think DXO should consider cooperating with one of the major DAM software companies, my preference would be IMATCH or PhotoSupreme. Enabling access to processed thumbnails and interoperability of metadata would benefit both companies.

DXO would not lose anything as their own database capabilities would cater for the customers who don’t need an external database whilst the cooperation would allow them to compete more effectively with programs like Lightroom. A win-win situation.

This is a bit like Capture One cooperating with Fuji, Sony and Nikon to offer free raw processor software. The camera companies don’t have to develop and maintain software whilst C1 gains potential customers for their paid software. These days companies need to work smarter not harder, an old maxim that’s still true today.

Ian

5 Likes

DXO just needs to stick to the latest recognised international standards. That way it remains compatible with all the main DAM packages.

6 Likes

Absolutely.

DxO should go back to writing changes to star ratings and keywords (and any pick flags) XMP. That’s where this kind of rating information is stored, in XMP files. Not a database, not .dop files.

Frankly I’d like Photolab to offer a preference to disable the database. That would mean rereading each folder on open (all the .dop files and all the .xmp files). Image previews could be used and kept on some kind of a weekly or thirty day revolving cycle for speed.

The database creates nothing but trouble for those of us (most of us) trying to simply use Photolab as a RAW developer. The pseudo-DAM which does not write to XMP and does not refresh changes to XMP sidecars is just a nuisance and results in lost information and inconsistency.

At the very least, DxO developers should give us a folder refresh option in Photolab to read any changes to XMP sidecars.

Please play nicely with others DxO developers and write your ratings, flags and keywords to XMP.

6 Likes

This is exactly how I use PhotoLab, Alec. I have no interest in any of PL’s DAM fetaures - - so, I do not need to depend on its database: Therefore, I have enclosed PL in a “wrapper” that first deletes the current database, before invoking the PhotoLab application.

As you say, this means PL needs to read & interpret all my corrections from sidecar/.dop files for each folder that I present it to - - which can be a slow process if the folder contains many (100s or more) {image+sidecar} combos. This works fine with my workflow, tho - as I work with no more than 10s of images at a time, in a dedicated Work-in-Progress folder - after which I move all components of the images (RAW + Sidecar + Exported RBG) to a separate storage location … and then repeat with the next batch.

There’s already a means of limiting the number of image-previews held - via the Cache settings in Preferences … albeit, based on cache size - not by time/age.
In my case, I also delete the cache (at the same time as I delete the database) before invoking PL … as its not needed with my batch-processing Work-in-Progress workflow.

John M

1 Like

John, thanks for sharing your workaround in detail and for the tip about cache. It would be wonderful if it would be possible to run Photolab in a way which doesn’t require these database-deleting handstands. With SSD ubiquitous now, there’s really no need to keep file level information (in the hundreds at least, thousands might be an issue)

I move more files than you (I might have 70 files in my five star folder for processing) and have run into issues when I don’t separate five star files into a separate folder before processing (those folders could have a couple of thousand images: sport). Even with the database, performance on the big folders can be an issue. Really better to separate selects before processing.

The reason I’m so keen to see star ratings, flags and IPTC copyright tags back in XMP is that I do make adjustments to those attributes when narrowing those selects from 70 to 40. It’s very frustrating that metadata changes made in Photolab are not shared back to PhotoMechanic, Lightroom, ApolloOne or FastRawViewer (I’ve used all of those programs within the last year to work on metadata, I’m leaning to consolidating on the new Photomechanic Plus which includes catalogues but continues to leverage the OS file system and external XMP files). Unlike Photolab, all of these programs store metadata in the XMP sidecar.

DxO really should cooperate better with other programs and share star ratings, flags and keywords with others via XMP. Just sucking the information in at the beginning and hiding it in proprietary files is a really Adobe/Microsoft style move. It’s very alienating and disappointing.

5 Likes

If you mean(?) there’s no need to continue with sidecar/.dop files - then I don’t agree with that; I reckon the complete flexibility of the {sidecar+image} relationship is a key factor of PL.

Absolutely, yes … that’s the way to go, Alec.

Regards, John M

Hi John, I mean the opposite. SSD are very fast so there’s no need to store information in a database (theoretically faster than reading two hundred .dop and .xml files for a total of six hundred files read to show a folder of two hundred images). Thanks to SSD high I/O transaction threshold there is very little performance penalty here other than loading the preview images, which happens whether or not the XML info (star ratings, flags, copyright info, headline/title, description, coloured labels – last three not available in the Photolab EXIF editor currently) is stored within the XML file where it belongs and not in the Photolab .dop file where it does not belong. DxO is breaking standards here.

It is almost criminal that DxO refuse to store metadata in the XML and only in the .dop file. It breaks photo application interoperability at no benefit to DxO users. DxO does so much right and then they sneak something like this in which frustrates existing users and makes non-users think, oh this might not work with my existing workflow. Worst of all it wasn’t always like this. DxO Optics did not try to bury info Wrong, issues with playing well with XMP files have bogged down DxO for the better part of a decade. In the past, DxO refused to even read XMP file ratings and Optics had its own rating system.

3 Likes

High time that DxO breaks the broken paradigm and starts not to just read incoming XMP star ratings but write its own ratings and select flags back to the XMP file where it belongs for proper interoperability. IPTC keywords, headline, description/caption, copyright info also belong there. Photolab should stop worrying about a proper DAM and worry about supporting keywords, headline/title, description/caption in their mini-EXIF viewer-editor just as they support copyright info (a successful implementation, btw).

The IPTC fields most DxO Photolab users need access to are exactly those ones, with perhaps adding location (although there are just too many location fields to make location an easy add). Users who want more than headline/title, description/caption, copyright info and keywords are probably require something like Photomechanic or iMatch which really can not and should not be shoe-horned into Photolab. Lightroom’s own EXIF handling is inadequate, while CaptureOne has gone overboard with IPTC fields:

While it pains me to say something nice about the people who made subscription-only software mainstream, Lightroom seems to have the right number of fields.

As far as I know Lightroom writes that metadata into the XMP file (the preference has to be set on a per-catalogue basis which I find too granular and likely to cause user-error: should be set once globally by a user).

For tif or jpeg files, the information is stored within the tif/jpeg which is okay as those file formats are intended to carry XMP/EXIF/IPTC information and the applications which understand XMP know to look within the file too.

Since Photolab users work almost only with RAW files (I did try to finish my Fujfilm jpegs in Photolab for a couple of months but found the RAW to TIFF in Iridient Developer to Photolab to jpeg output workflow far too heavy) the important part here to keep in mind is writing that metadata and those metadata changes back into the XMP files. It’s somewhat less of an issue than I imagined as Photolab will write metadata changes into its own jpeg/TIFF output.

In my own workflow I start working with metadata again post-Photolab (was using Adobe Lightroom 4, currently trialling with much better results Photomechanic Plus) so any changes to metadata should be in the XMP so I can see changes made to metadata in Photolab. The big issue is with star ratings as I may wish to browse that selects folder with Photomechanic or any other fast efficient RAW viewer (does not include Photolab) and see changes made to star ratings or flags. It’s really a matter of principle. There’s no good reason for DxO to behave badly here by storing shared data like star ratings and select flags, and perhaps later keywords and captions when the accepted vessel for this information is an XMP sidecar.

I hope this information and these examples help some others to puzzle through the challenges of integrating Photolab into a DAM and metadata friendly workflow. Even more, I hope it inspires DxO developers to fix Photolab’s lapses with metadata (store metadata once and for all in XMP sidecars where it belongs; for bonus points, add support for headline/title, description/caption, keywords to the existing EXIF editor).

PS. If DxO improves the EXIF editor, storing metadata in XMP sidecars suddenly becomes much, much more important. Right now the most prudent way to handle metadata in DxO Photolab is to more or less ignore it and not touch it.

3 Likes

Good morning all,

maybe some of you will vote for this feature request Possibility to add or change metadatas - DxO PhotoLab / Which feature do you need? - DxO Forums

Thanks in advance and enjoy rest of the week

Guenter