Hello Zoltan,
PSU can embedd metadata to:
NEF
NRW
TIF (RAW versions some Canon older cams)
ORF
CR2
PEF
You just tick a box in preferences and it will embedd - but to those formats only. Everything else ise xmp-sidecars.
So PSU definitely uses other technology to embed the metadata then Exiftool (at least I assume so).
I guess I will have to ask what IMatch exactly writes back to the files as a simple comparison of metadata values was not really helpful, there were no real value changes, but the files were definitely changed.
Zoltan - you are correct. They use their own technology to embedd, based on inform from the developer.
Hi,
sorry for the late reply - I’ve been travelling this week and didn’t get manychances to check in on the discussion here, but thanks for all the additional investigation, Zoltán and Sigi.
I’m not sure if it’s still something of interest at this point, but I took your CR3 file, @bekesizl, and edited it in IMatch as you asked, adding a GPS location, some copyright info and a couple of keywords:
2019_07_14_6059_edited.7z (3.0 MB)
Hi,
you are not late, thank you for the file.
I have made further tests and I can now definitely say, the problem is in the cooperation of the way IMatch uses Exiftool and the way DxO Photolab reads the CR3 file.
Although I am not an expert on Exiftool or Metadata, I have some general understanding of computers and programming.
Your file was not readable by PL3.
I ran following command on both original and edited files and compared them in Notepad++
exiftool -a -G1 RAWFILE.CR3>exportlist.txt
I can see, that beside the “Artist” tag, the tag “Copyright” was added:
The tag “Sub Sec Time” was removed and “Focal Plane Y Resolution” was changed:
GPS Location was added:
These changes are displayed in the composite tags too:
At the very start of the file however I see, that then endianess of one Exif part was changed:
Now I tried to make these changes with Exiftool myself.
You can run Exiftool with the arguments saved to a file, which makes complex edits like this easier.
Command line
exiftool -@ command_in_argfile.txt CR3_FILE.CR3
command_in_argfile.txt contents (GPS coordinates not exactly the same, just similar):
#Basic metadata
#Copyright
-copyright=Copyright © 2019 all rights reserved.#GPS
-GPSLatitude=51.504578
-GPSLatitudeRef=N
-GPSLongitude=0.1254579
-GPSLongitudeRef=W
-GPSAltitude=7.96
If I run these commands on the original file, the export can be opened by PL3.
The export is not exactly the same, but Mario will never tell us the exact commandline used by IMatch. The changed data is inside. Endianness was not changed.
I made one more test.
I took the argfile from this link:
source for more complex Exiftool argfile sample
It contains also keywords and much more.
Running Exiftool with the “argfile” on the file, the CR3 can be opened by PL3. Keywords are not visible, but most other information is.
So I will post this also to the Exiftool Forum. Maybe Phil Harvey might have an idea what command can cause the endinannes change and what command you could run on your files after IMatch to be able to read them, until DxO solves the problem.
I guess, I won’t post these results to the IMatch forum. I have the feeling, that Mario is not very fond of my findings.
1 Like
We might know more on Monday, Canada time.
Last post from Phil:
Hi Zoltán,
The only way to change the byte order should be to delete the EXIF then add it back again. That would be a very dangerous thing to do to a RAW file, so I would be surprised if iMatch was doing this. But I’ll take a look at the files on Monday and see if I can answer some questions then.
- Phil
If Phil thinks there might be a problem with the way IMatch writes metadata to CR3 files then I wouldn’t hesitate to post what he says to the IMatch forum .
I’ve been a very happy IMatch user since 2006 and I can assure you that Mario is very receptive to fixing bugs in his software. Just this morning I submitted a bug report to him which he had fixed within 2 hours.
He just needs to know a problem is actually in IMatch and not some other software before he’ll spend much time diagnosing it.
Interesting findings for sure - curious to hear what Phil has to say about it…
Thanks again, Zoltán for diving into the gritty details here and bringing it to Phil Harvey’s attention!
If it turns out the issue is indeed with IMatch, I also think Mario will pick it up.
Great finding!
This may be relevant: https://exiftool.org/forum/index.php?topic=9074.msg46839#msg46839
Phil Harvey:
I do not recommend changing the byte order of the EXIF because proprietary information may be corrupted.
and
I just released ExifTool 10.90 which allows ExifTool to be used (again) to change the EXIF byte order of a JPG
So the answer from Phil is here. He found out, that the GPS data is in another byte order and that is causing the problem.
First, there is a workaround for correcting the files:
exiftool 2019_07_14_6059_edited.CR3 -o fixed.cr3 -gps:all= -tagsfromfile @ -gps:all -exifbyteorder=little
A note from Phil to the correction:
(note that without the -o fixed.cr3, DxO somehow won’t recognize the original file after it is fixed, even though it is binary equivalent to the newly created file fixed.cr3 – it must be caching some information about the file, and I don’t know how to clear the cache)
To @Marie and responsible DxO staff:
“But be aware: In the past DxO has taken shortcuts in their reader routines (ie. instead of properly parsing the file format, it seems they they were assuming a fixed offset for some data), which would explain their policy of not supporting edited raw files.”
“The DxO programmers are cutting corners that will make more work for them in the long run. They must be using a dedicated code to read the GPS from this file instead of a general EXIF/TIFF reader which would have handled the byte order difference with no problem.”
We have a saying at my workplace: you never have the time to do it once good, but you always have the time to do it a second time.
So please, clean up the CR3 file reader.
I will post this also to the IMatch forum without comments, but you, the IMatch users will have to persuade Mario to make a solution on his side.
3 Likes
Great to hear that the issue has been identified - thanks again for driving the effort, Zoltán!
What I’m wondering about though is: why does this issue with the changed byte order of GPS data also affect files that have no GPS data assigned to them? Does the byte order also change even though the corresponding metadata fields don’t contain any value? (Sorry if that’s a silly question, I’m not so well versed in the technical background questions…)
That is something that probably Mario at IMatch can answer.
In my original file for this thread there was GPS data, but maybe IMatch is writing always something there or the block contains also other data.
On the IMatch forum Mario said, that IMatch is not willingly modifying the byte order, so maybe there is something he wants/can do. My first try to send him an original file was not successful. If my second one doesn’t succeed, I will send him a file by email from home in the evening.
OK, understood, thanks again! Let’s see if Mario is willing to look into it…
I think we reached the end of this thread on the IMatch end.
There is a new version of Exiftool which should solve this problem for new files.
Phil Harvey: I have just released ExifTool 11.89 which is patched to create new GPS information using a default byte order that is the same as the existing EXIF boxes. This should hopefully solve the DxO incompatibility problem. Mario plans to include this new version in his upcoming IMatch release.
It is planned to be included in the IMatch 2020 version.
Phil Harvey: But to avoid problems with DxO, I have just released ExifTool 11.89 which will create new GPS with the same endianness as the other EXIF metadata in a CR3 file.
Mario: Excellent.
I’m just in the final stages of building the first official IMatch 2020 version. I will include the latest ExifTool version in that build.
So if you are using IMatch and cameras with CR3 format, you should upgrade if the new version is released.
Until then, hope for DxO fixing the CR3 reader or use the workaround.
1 Like
Hello @Marie,
information for the programmers:
Phil Harvey: In a CR3 file, if you delete all GPS information and add it back again then the byte ordering may change. The reason for this is that the GPS is stored separately from the rest of the EXIF in this type of file, so it isn’t necessarily the same byte ordering as the rest of the EXIF. Unless you specify another byte order when writing, the ExifTool default for the newly created GPS is big-endian. I find it surprising that DxO can’t handle big-endian GPS, but there you go.
Fix, please.
Yes, please!
I have run a short test in IMatch which shows that the problem is not related only to GPS data. CR3 files straight from camera without GPS data are accepted by PL4. After adding metadata through Breeze Browser Pro (e.g. keyword, Title etc), they are still accepted by PL4 and show the following ExifByteOrder:
ExifByteOrder : Little-endian (Intel, II)
ExifByteOrder : Little-endian (Intel, II)
ExifByteOrder : Little-endian (Intel, II)
ExifByteOrder : Little-endian (Intel, II)
In Imatch, CR3 files edited externally to the app require write back, after which they show an altered list of ExifByteOrder:
ExifByteOrder : Little-endian (Intel, II)
ExifByteOrder : Little-endian (Intel, II)
ExifByteOrder : Big-endian (Motorola, MM)
ExifByteOrder : Little-endian (Intel, II)
It seems that the act of Write-back in IMatch introduces the big-endian line into exif.
Which version of IMatch are you using?
The Exiftool patched for this DxO bug is only inside the newest 2020 version released yesterday.
I didn’t purchase in the end so I can’t test.
I just installed IMatch 2020 (since I only bought the 2019 version a few months ago, I was eligible for the free upgrade) and gave it another try writing metadata to a CR3 file (to the exact same fields with which I encountered the issue originally) and this time around Photolab can open the file with no issues, and is displaying the updated metadata written by IMatch 2020.
However, those CR3 files to which I had written metadata using IMatch 2019 still can’t be read by Photolab - no surprise there, since this is dependent on DxO fixing the issue regarding PL not properly reading the byte order/endianness of CR3 files on their end…
1 Like
Hi Phil,
I checked it again if maybe a simpler workaroung is possible with the newest Exiftool version, but unfortunately not.
You have to copy the complete GPS group from the original file and write it once again in a new file.
And here I finally understood what Phil and Mario were saying about the ByteOrder mark. This is a -v3 output from Exiftool and the directory with the byte order “Big Endian” is containing the GPS data, the byte order is marked clearly.
So DxO, please correct this bug.
What is Photolab using the GPS data for anyway? I see a coordinate display in the customize workspace, and you are probably exporting it too, but currently I see not other use inside the software.
Zoltán
1 Like