Provide the option to use PhotoLab without a database

I for one don’t want two sidecars. I would be totally okay with all DXO data to be stored in an XMP sidecar. But if not then the keywords should also be stored in the DOP sidecar.

I’m with you on that sentiment - - The database does enable some efficiencies within a session - but, like you, I have no need for a database to store anything between sessions (again, like you, preferring the “flexibility and safety” of sidecar/.dop files).

Accordingly, I wrote a simple wrapper that deletes both the database and the cache before each time PL is invoked. (Something as simple as an old fashioned batch.bat or command.cmd file will do the job).

A quick/simple way to do so is via Ctrl+Tab - - which toggles between the 2 modes.

In which case, as @platypus suggests; you might like to check-out DxO’s new “PureRAW” tool.

John M

Thanks for the suggestion, but unfortunately that doesn’t fit my use case. I probably shouldn’t have used the term “RAW developer”, as that implies I’m converting files into another format and editing them further elsewhere. I’m not; I’m using DXO to generate final output as JPEGs, which in most cases means running with default image adjustments, but sometimes involves cropping, white balance, or level adjustments before export.

I’ll usually process an entire folder with the defaults at once, adjust any that I already noticed need some manual work, then export the lot. If I spot anything that needs additional tweaking later, I’ll pull it back into PhotoLab for additional tweaking, and re-export. I’d say around 95% of my images get automatic adjustments only, and perhaps 4% get a bit of manual adjustment in PhotoLab. Less than 1% get exported to TIFF and pulled into a separate raster editor; That’s usually only done when I need to combine a few images becuase somebody moved or blinked at an inopportune time.

Basically, if you took OpticsPro, added DeepPRIME, and added the extra camera/lens modules created since the rebrand, that’d probably be the sweet spot for me right now. Extra features such as version history are nice ideas, but not if they don’t follow the image around, and come at the expense of a big performance hit at startup.

It’s lightning fast at everything except for startup on my box. It’s an AMD 5600X, 32GB 3200MHz RAM, NVIDIA 1080Ti, with the database and PhotoLab itself on a high speed NVME SSD, and photos on SATA SSD storage. This morning, it took around 14 seconds to open the application with the database present on disk. The moment I removed the database, the startup time was cut in half. It’s not horrific, but this isn’t really appropriate behaviour. I see other people reporting multi-minute startup times on their systems, and I can believe it. I didn’t even realise how much it annoyed me until I removed the database and realised how quickly the application should start on my hardware.

Maybe removing the database outright isn’t the solution, but something is wrong with the design if we see this level of performance hit when loading the application. Whether that means providing the option to disable the database, having PhotoLab purge old, unnecessary data more aggressively, or being more selective about what data is read from the database before opening the application, I’m not 100% sure. Right now, for me, no database is better than bad database behaviour.

Thanks for the heads up, that might be useful.

I don’t switch tabs often, as normally work with a single folder in a typical session. I suppose it just slightly bugs me as an unnecessary extra click between waiting for the application to load, and actually being able to start processing images. I guess with a quicker startup time, I’d be less impatient, and it wouldn’t annoy me as much. I don’t remember it being as annoying in the OpticsPro days, but maybe that’s just me.

I think that’ll be my strategly going forward, either at application startup, or at system logoff. I’m a sysadmin by trade, so I’m confortable enough with scripting. It’s just mildly annoying to have to write a script, find a place on disk for it where I won’t lose it in future, and schedule it to run on both my main machine and laptop. I can do it easily enough, but it’s obviously a lot slower and more difficult to set up than “just tick a box”, and requires a level of technical knowledge that not everyone will have.

2 Likes

Let me make sure I understand this correctly. You are frustrated that PhotoLab takes 14 seconds to open with the database and half that time without it.? How many times a day do you start PhotoLab that the 7 seconds becomes such an issue for you? Before reading your response I assumed your startup was taking a minute or two, or more, before you could begin editing…

I have a i7 6700 machine with 24 gig of ram and a GTX 1050ti,.PhotoLab, the database, and the cache are on an SDD drive, but the images are on a 2 terabyte HDD. PhoroLab 4 startup takes the same 8 to 10 seconds regardless of whether I delete the database or not. .

Mark.

Yes, that is correct. “Frustrated” is probably an overstatement, “impatient” is probably a more accurate description. I felt a 14 second startup time on fast, modern hardware was rather on the slow side, but tolerable given the capabilities of the software (although I wonder if that would creep upwards over time). I am by no means saying this is some sort of showstopper issue for me personally, although I suspect others are worse affected.

That said, with my IT nerd hat on, I can’t help but feel that losing an unnecessary 7 seconds or more on every application startup, simply because DXO PhotoLab is slow at performing unnecessary database operations every time it opens, is a negative. There’s been a lot of research done into things like web page load times, and the takeaway is, from a user experience point of view, seconds matter when you’re waiting for your system to finish a really short task and hand back control to you. If the software startup is being slowed down that much by the database code, there’s got to be something really suboptimal about the design of the software, and how it interracts with the database on startup.

I don’t want to rant excessively here. Like I said, in my case, I’ve probably made it sound like a bigger deal than it is; It’s a minor performance regression on my machine, nothing more. I’d personally would rather not have the database around for my particular workflow if it makes things slower, and clearly I’m not the only one.

I do think there’s some merit to having the option of storing the edit history in dedicated sidecar files, rather than the database, though.

Edit: Interestingly enough, re-opening the application once it’s created a new database seems to be consistently around the 7 second mark for me. The slowdown also doesn’t seem to be proportional to database size on disk, as the slower database I deleted was about 20MB in size, and I’ve got the new one up to around 70MB by viewing every photo folder I have in turn in DXO.

That suggests that it’s something else, possibly relating to number of files processed, files that are in the database but have moved or been deleted outside of DXO, or some sort of weird edge case that causes a performance regression. I’ll play around with bulk processing and then deleting a bunch of copies of my images, and see if I can identify a reproducable cause for any sort of performance hit.

1 Like

OK, here’s the latest stuff I’ve tried, starting with a blank database. I’m unable to reproduce the slower startup with a new database at the moment, so I’m leaning towards some sort of as yet undetermined edge case bug as the cause.

My original database had been building since around January, when I set up this particular PC. Since then, I’d worked on a few individual photos on a few of my photo folders in PhotoLab, as well as some RAWs that I’d copied into temporary folders for processing (both for DeepPRIME benchmarking, and for EXIF tweaked copies of some iPhone XR files to get them to process in DXO). The temporary folders would have been removed after I’d finished with them. The total size of the “bad” database was probably around 30MB, and regrettably, I no longer have a copy of it. Total opening time with the “bad” database was around 14 seconds.

After creating the new database, and clicking onto each of my photo folders in turn, DXO would be aware of approximately 6,500 images, and the database grew to around 70-80MB in size. Application opening time remained at around 7-8 seconds after this, so database size and number of images doesn’t appear to affect things in any significant way.

I then created a temporary folder on my desktop, copied around 3100 RAW files into it, and opened DXO. I selected the temporary folder, selected DeepPRIME for noise reduction across all files (so that some changes had been made), then reverted most of the images back to HQ for denoising. I ran the export to JPEG, and closed PhotoLab afterwards. PhotoLab took a couple of minutes to close when everything was done, but re-opening PhotoLab afterwards remained at around 7-8 seconds, although the database had grown to around 140MB. I think that eliminates “number of photos that have version history stored in the database” as a potential for causing slow startup as well.

Finally, I removed the temporary folder from the desktop. Again, PhotoLab opened within around 7-8 seconds, and the database size immediately dropped back to around 80MB, so I think we can safely assume that DXO are purging old data from the database for files that they can no longer find on disk. I think that eliminates “lost” files as a possibility for slowdown as well.

In summary, I’m now running out of ideas. I know some people have reported slowdown after upgrading from one version of DXO to a newer one. I’m on the latest release now, so I don’t know if that’s a factor, but maybe that’s something for me to keep an eye on. Otherwise, I’ll try to keep an eye out for either gradual degradation, or any point where opening DXO suddenly gets a whole lot slower.

If I run into any issues with additional database related slowdown, I may log a support ticket, and send DXO a copy of the database file to investigate. I’d suggest that if anyone else suffers a performance hit to startup over time, and finds it resolved by removing the database file, that they also consider sending through the old database to support, to see if they can make any sense of the problem. The database files seem to compress really well, by the way, so you can easily stick the database files in a ZIP file if you need to send them from A to B whilst keeping file size down.

So, it appears you are now saying that the database does not cause a performance hit at startup, correct? The earlier “issue” might have been due to the operating system, other software, or memory use variations and not a bug in PhotoLab. I have known for a long time that after rebooting my computer, the first time I start PhotoLab it takes almost twice as long compared to subsequent restarts. Is that a bug?

Mark

I think that the PhotoLab startup time was consistently longer before the DB was cleared, but I’m not 100% certain. Since being cleared, the database is not slowing down the launch time, and I’m unable to artificially create a scenario where it does so. Hopefully that’s the end of any performance issues, but I’ll try to keep an eye out for any patterns, and if things start to slow down, I’ll try to dedicate some time to figuring out why that might be.

I think it is important when making observations regarding things like software performance not to draw absolute conclusions based on insufficient data. One of the dangers of that is those conclusions may cause general confusion and create negative perceptions about PhotoLab in the minds of those reading them.

In your earliest post you stated, “You can add me to the list of people who consider the database to be actively harmful. In my case, all it does is significantly slow down the opening of PhotoLab, unless I purge it periodically.” (The bolding is mine) That was a very damning assessment of PhotoLab which, even based on your own additional research, may not be accurate. Words matter…

Mark

You are correct, and that’s a fair point. I definitely misidentified the scope of the issue; I was under the distinct impression that I was dealing with a gradual and recurring performance degredation issue, and should definitely have verified that the issue was recurring, rather than jumping to conclusions. Today’s testing has shown that my initial hypothesis was incorrect.

I’m still a little concerned that there are situations in which the database can get into a state where it does slow down application startup, and wish I understood the particular set of circumstances that can cause that to happen. Right now, without knowing what can cause a database problem, we don’t know how to avoid a database problem. Having the option to disable use of the database means there’s no way to develop a database problem, and hence no risk of any hit to application startup time.

It is entirely possible that the database can cause some performance issues depending on circumstances and the platform PhotoLab is running on, but I think the jury is still out on that.

I have been running PhotoLab Elite, along with the Viewpoint 3 and FilmPack 5 Elite plug-ins, ever since Photolab1 was released in late 2017. I use PhotoLab almost every day. In all that time I have never had any issues related to the database. PhotoLab has always been very stable and responsive on my Windows 10 midrange desktop computer.

I had mentioned in an earlier post the possible contribution of the operating system on performance. My personal observation was that when I upgraded Windows 10 1809 to release 2004 both PhotoLab startup and DeepPRIME processing improved by several seconds. After upgrading to 2004, my startup time averages between 8 and 10 seconds. Prior to that upgrade the average startup was 12-14 seconds. That is a pretty significant.performance improvement.

Mark . .

The way I look at the database, it should be effectively a cache used to speed the performance. All the info required to rebuild the database should be stored in the sidecar files. This is ALMOST true. But not mainly because of keywords and edit history. The database should be self cleaning in that it should automatically removing the cache from folders that have not been accessed in a while.

Since the DAM is somewhat limited, we manage our images outside of DXO primarily. Whether it’s by folder structure or with another tool. DXO should accommodate this strategy. (It mostly does…)

2 Likes

This is one possible way. And I still keep up the idea to make the database and cache management manageable by the user. One way could be to offer some kind of “timer” or a switch in the settings:

1 Like

Yes. Or a “days old” concept. I.e. purge any data and image cache for images that haven’t been touched in more than “n” days. Where N can be 0, 7, 30, 90, or whatever.

I hate that every imaging program seems to think they can only survive, if they imitate Lightroom! One major reason why I chose PhotoLab was that it came without that obnoxious database. I do not want a database. I want a great raw developer! This is what PhotoLab excels at. The metadata managing options in Lightroom are a joke. And now PhotoLab follows their lead? PLEASE, give us an option to turn this off. There are far, far better and much more powerful image asset management programs available than either Lightroom or PhotoLab. I believe it is a huge strategic mistake to believe that PhotoLab needs to compete by imitating what other programs do. Managing images is different from editing them. One single program will never excel at doing both.

6 Likes

Are you using the * .dop and * .xmp sidecar files ?

Yes please.

As a Photo Mechanic user I have no interest in PhotoLab’s DAM options, so all the database does for me is create issues when using outside methods to rename or organize photos.

I appreciate that some people want DAM features, but surely we could have the option to switch off the database and use dop files exclusively. Or as @Platypus suggested, at least add the option to automatically purge the database at quit.

Purging the database manually is an option, but a very long way from ideal.

2 Likes

I do it the other way around; I have a “wrapper” that deletes the database (plus the cache & thumbnails) just before it invokes a new PL session … A fresh database is created each time PL starts (when it finds none already exists) - and it uses this to facilitate processing for the current session. Rinse & repeat.

Used a rinsing wrapper for a while with OpticsPro and early PhotoLab versions. It provided best editing conditions. I use manual rinsing now, because I want to be able to continue working on a job without risk of losing things. This risk has diminished considerably with newer versions of DPL for macOS, but on Win, you still lose history and projects when you delete the database, right?

1 Like

History is not retained between sessions on the Win version - so that’s not an issue.

Yes, you do lose projects (because the virtual linkage between project-member images is lost when the db is deleted).

There used to be other issues too - but, now, I think pretty much all other meta-data is held in the sidecar/.dop files (tho, there may be some that I don’t know or care about).