Win 10 DxPL6.0.1 Can't handle large directories - thumbnail refresh fails

BHAYT · October 30, 2022, 1:13pm

@DxO_Support-Team The following has been moved from Update to V6 Elite issues - #16 by BHAYT to this new topic to keep track of what seems a serious problem with DxPL 6 that was not present with DxPL5 when handling directories with a lot of images, one with over 11,000 and the other with just 1,000!!

I used PL550 and PL601 with empty databases to execute a load (discovery) of images, i.e.

A directory containing multiple copies of the same images from a shoot where I left the “burst” mode set, some 6,900+ RAWs and 4,800+ JPGs.
I had already copied the bulk test data of 11,959 images to my Q: drive, an NVME mounted on a PCie card.
Both versions were set to locate their databases on the NVME drive, PL5 already was configured that way but, in the previous test I documented in the original post, PL6db was located in the default location on a SATA SSD.

Now for the problems:-

While testing the PL6 handling of such a large directory I encountered a number of problems

Sending a group of 5 images from FastRaw Viewer (FRV) to PL6 the following was encountered and no amount of ‘F5’ would change the situation!

It took a PL6 restart to give the following

I edited an image in another package and decided to edit the same image in PL6 and encountered the following problem while trying to locate the image

and

no amount of navigating away and back would clear the problem

It required a PL6 restart!

Returning to the directory today I started scrolling through the images and all appeared O.K… Jumped to the last image and scrolled upwards and all went well until

Once PL6 gets into this state nothing short of a program restart will get it going again and there is no guarantee that somewhere in the 11,959 images that this might not happen again. This is absolutely a stress test for PL6 (and for me as well)!!

Navigating to a directory of 1,000 images (without a program restart) and clicking on various thumbnails produced the image and the following, please note the ‘Loading error’ when clicking on a ‘Loading please wait’ thumbnail!?

After a restart of PL6 (on the 1,000 image directory) all went well until I scrolled passed where PL6 had got to with the thumbnails and this happened!

I navigated away to another directory and then back and scrolled down without scrolling ahead, i.e. just keeping up with PL6 and

But PL5 does not have the same problems!:-

I repeated the test on PL5 and no matter where I scrolled PL5 immediately produced the thumbnail albeit some were left with the arrow icon until the image is selected

Whatever has been done on PL6 seems to have undone the excellent performance on PL5!

BHAYT · October 30, 2022, 1:18pm

Repeat of the tests with renamed image files (for better navigation of test images):-

@DxO_Support-Team I renamed the two bulk test groups as “BULK-xxxxx” and “Bulkxxxxx” so that I could show at what point things went wrong.

Cleared the PL6 database and re-discovered the “BULK” images and scrolled down the thumbnails in step with PL6 until

After discovery was completed scrolling down from the start of the thumbnails the problems start at the same point as previously identified

Navigating away and back changes nothing and it will require a program restart to change things (and, sadly, not necessarily for the better)!

Once discovery is complete.

PL5 shows none of these problems, in fact it was still running with the directory in focus during the name changes etc. and works just fine.

LJC · October 31, 2022, 3:43pm

Hi Bryan,

Problem number 2 on your list, the issue with thumbnails hangs with message “loading please wait”, is the issue I also have and I have logged a support ticket with DXO support last week.

IanS · October 31, 2022, 4:48pm

Have you checked your memory use when you get this problem?

I had an issue in a beta version where in a long process session of around 4,500 images, memory use would continually increase, (until ~20/32Gb) and then the system locked and gave me similar results to those you describe. Closing and restarting DXO was needed. I wasn’t able to chase this down during the Beta for personal reasons but just wondered if it was in any way related to what you are seeing?

BHAYT · October 31, 2022, 5:34pm

@LJC I owe you and @ralbrux an apology for misunderstanding the nature of the “load” that failed. I took it to be an initial load and that worked well (or seemed to) hence my response, then I repeated the test with timings to see what, if any differences, occurred and that all seemed to go O.K.

But then I started looking at what had been loaded and quickly ran into the problem with PL6 “running out of steam” and thought at the time that might be the issue you had described!

The good news is that it is eminently reproducible and was not present in PL5 so who changed what, why and how and can they please put the product back the way it was @DxO_Support-Team!

@IanS No I didn’t but I could try, although I managed to run into problem with the 1,000 Bulk images let alone the 11,000+ BULK images.

Both my Main and Test machines are essentially the same, i7-4790K, 24GB memory, 1050 graphics (1050Ti-4GB on Test and 1050-2GB on Main) an although I wasn’t monitoring the memory usage PL6 still kept “working” but it was just not going anywhere with the directory thumbnails once it had “stopped displaying” them. I can run another test or two to see what additional symptoms might be showing.

This problem needs addressing urgently because while breaking with 11,000 images may be “vaguely” acceptable (but it didn’t happen with PL5) breaking with 1,000 images is totally unacceptable!

@LJC Thank you for posting and “linking” your support request to this post.

Regards

Bryan

LJC · November 15, 2022, 3:05pm

Hi Bryan,
Just got an email from DXO support, issue causing the “loading, please wait message…” has been identified and will be fixed in the next release of the program, version 6.1. There is no date given for the release of 6.1 yet.
Ljc

BHAYT · November 15, 2022, 3:33pm

@LJC Thank you for the update lets hope that the release is not too long a wait!

Take Care

Regards

Bryan

roogirl · December 1, 2022, 9:14am

@BHAYT Thank you for posting your testing and letting others know it’s an issue with PL6, not previous versions. We are new to DXO software and obviously didn’t do enough testing before purchase as this thumbnail issue is horrible. First thought it was me, then storage drive, then using standard preset for all RAW and JPG files due to this page (https://lenscraft.co.uk/photography-blog/photolab-performance-problem/). Now I hope update is soon AND resolves it. For me it doesn’t matter on # of images in a folder (1-2000), type (CR3, NEF, JPG, TIF), or if I’ve been in that folder this session or in previous software restart. It seems to hit a certain buffer of thumbnails and then hangs. If I walk away for hour, return and scroll though images, sometimes it starts loading them.

BHAYT · December 1, 2022, 12:45pm

@roogirl Thank you for the post the issue was first discovered by others and I initially misunderstood what the fault was, oops! @LJC raised a support request and hopefully the problem will be fixed soon (there is talk elsewhere of a January release)!?

The tests here were then conducted when I realised my misunderstanding but starting with the largest directory I have of over 11,000 images, created to explore another issue some time ago and then used to explore how other software copes with such a large directory!!

PL5 coped slowly but perfectly but PL6 didn’t and fails with smaller directories as you have discovered!

I just ran a quick test by accessing a smaller directory of 300 images and another of a similar size and then accessed a large directory of 997 (JPG) images and PL6 quickly “choked” on those.

Closed PL6 and re-opened on the large directory and it “coped” for a while before “choking” again. However, while the thumbnail rendering stalls and then finally stops working completely it is possible to access an image and apply presets and edits and the DOP is successfully stored!

But I then passed an image from FastStone Image Viewer, from the RAW directory (of the same images in the JPG directory tested above), and it immediately encountered

so the “poisoning” (“indigestion” is perhaps the better analogy) is cumulative as you indicate in you post!

The problem comes if you are trying to use DxPL for work or need to process a batch of images urgently, i.e. can’t wait for the fix!

Depending on the platform you are using DxPL(Win) or DxPL(Mac) one “workaround” is to use another piece of software to a provide the navigation and selection process and pass the images to PL6, effectively breaking any “large” directory into a series of smaller ones!

When they arrive in DxPL these can then be turned into ‘Projects’ so they can be processed and re-discovered easily, e.g.

It is possible to navigate the individual ‘Projects’ (Batches) or the whole structure and this does not completely “sidestep” the issue but may make working on the larger directories at least possible!

I used FastRawViewer which has a 25% off offer which is due to end today but other viewers can pass large batches to DxPL and all will encounter another bug on DxPL(Win), an empty ‘Customize’ screen! Use ‘Ctrl-Tab’ to go to ‘PhotoLibrary’ and the same to return to ‘Customize’!!!?

PS:- XnViewMP can also do the same and shows the number of images selected.