• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Catalog on NAS?

Status
Not open for further replies.

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
OK, this has probably been asked before, but.......

I've just upgraded my network and storage setup - I have a Windows 10 desktop PC in my office (dedicated concrete 'man-shed' in the back garden). I have Cat 6 GbE for internet and the like, plus a dedicated 20,000 Mb/s link (2 x 10Gb-SR in LACP over fibre optic) to a Synology RS3617xs with 12 x 4 TB HDDs in RAID10. NAS is in the house for security reasons and the fact that it's FAR TOO LOUD to tolerate in my office.

Rather than running my catalog on the local PC and having to back it and the backups up to the NAS, and then having the NAS sync the backup to the local storage/cloud, is there a really good reason why I couldn't/shouldn't run the catalog from the NAS on an iSCSI target volume?

My plan going forward is to keep photo folders and catalogs together in one place if possible and keep the catalog sizes relatively small.

I've read lots of arguments about network speed limitations being a reason to not have the catalog networked, but my network transfer speeds are significantly faster than even a local SATA SSD would be. Are there other good technical reasons why it's not a good idea?
 
Other factors are: (1) Lightroom now accesses the catalog in SQLite WAL (Write-Ahead Log) mode: changes are written to a .wal file and later applied to the .lrcat catalog file. ... SQLite provides a function to flush changes to storage. I don't know if Lightroom uses it, and if it actually causes the changes to be written all the way back to the server.

I used several tools (watch, inotifywait, wireshark) to observe catalog operations while having LR sync develop settings across >3K photos:

- Windows does set oplocks on the .lrcat* files, enabling client caching.​
- Data is frequently flushed to the .lrcat-wal file on the server.​
- Occasionally, SQLite checkpoint operations apply the WAL data to the .lrcat file on the server.​
- Other than the initial oplocks, no other lock requests are sent to the server. (According to the documentation, WAL mode limits database access to the (Lightroom, in this case) processes on a single PC, and these processes coordinate through shared memory.)​
- SQLite also creates a shared memory-mapped file (.lrcat-shm) in the catalog folder. According to the documentation, this file contains no persistent content and is not used for recovery.​
- SQLite creates temporary files, but none were observed in the catalog folder, so I presume that they're in a temp folder on a local drive.​
This is all good behavior. Since "one test is worth a thousand expert opinions", it would be interesting to force Windows to crash (see notmyfault) while LR is performing various catalog modifications to measure the relative risk of corruption with the catalog on a network drive vs. a local drive. I suspect it's negligible.
 
Glad to see the detail; you have the energy many of us lack to dig into it.

While you are set up, I'd be curious to hear how much activity you see while lightroom is otherwise idle (but with a good sized catalog). I've always worried more about people who might just leave it sitting, especially since Cloudy came along and people want it up as a receiver for sync'd photos from mobile. Then the power goes out (or whatever).

Of course, as to testing, as they say -- it's hard to prove a negative. You can just draw some statistics around it.
 
While you are set up, I'd be curious to hear how much activity you see while lightroom is otherwise idle (but with a good sized catalog). I've always worried more about people who might just leave it sitting, especially since Cloudy came along and people want it up as a receiver for sync'd photos from mobile. Then the power goes out (or whatever).

Opening a catalog with >30K photos and then letting it idle for 30 min. resulted in 1670 SMB2 ops total on the lrcat* files on the server. No Write ops after open. LR just does a Find op on the catalog folder every 10s while it's idling. My LR isn't syncing with cloud, though.
 
Opening a catalog with >30K photos and then letting it idle for 30 min. resulted in 1670 SMB2 ops total on the lrcat* files on the server. No Write ops after open. LR just does a Find op on the catalog folder every 10s while it's idling. My LR isn't syncing with cloud, though.
That's interesting. I thought it was always doing stuff in the background, but maybe only if there are things to do, like purging 1:1 previews that have expired (I realize that's not in the catalog, not sure if it updates anything there), address lookups, or such.

Thank you forchecking.
 
That's interesting. I thought it was always doing stuff in the background, but maybe only if there are things to do, like purging 1:1 previews that have expired (I realize that's not in the catalog, not sure if it updates anything there), address lookups, or such.

Thank you forchecking.
Since I thought your concern was about the catalog, I only checked ops on the three lrcat* files. Running
Code:
inotifywait -r -m --timefmt "%H:%M:%S" --format "%T %w %e" -o events.txt Master

on the server shows that LR searches the Backups, Helper, and Previews folder trees once/hour while idle.
 
Since I thought your concern was about the catalog, I only checked ops on the three lrcat* files.
It is really, that was the only example of a known idle time thing I could think of. :whistling:
 
I would like again to make the point that @gorlen is easily in the top 10%, if not the top 1%, of technical skills in this forum. For most everyone else, you probably don't know how to cope with some Windows issue created by using a NAS, that @gorlen just fixes without even thinking about it. SQLite was not designed to work over a network, including a NAS. I personally have experienced problems with the personal finance manager Quicken, when keeping the Quicken data files on a network share. Quicken also uses SQLite.

We all know that catalog corruption very real, and sometimes even Adobe can't even recover a corrupted catalog. Get a larger HDD, get an external HDD that plugs in via USB, but please don't use a NAS. Your time and effort are more than worth the expense.
 
SQLite was not designed to work over a network, including a NAS.
From SQLite CVSTrac :

Using SQLite on a Network
SQLite database files may be shared accross a network using a network filesystem. This is never a particularly efficient method and may have problems (depending on the filesystem, or may simply not be available. These are alternative techniques for remote access to SQLite databases.

It appears that accessing a Lightroom catalog on a network filesystem has gotten a bad reputation because, years ago, catalog corruption was common due to software bugs which are now rare -- the software involved has matured.

Furthermore, a Lightroom catalog presents a less demanding situation than what SQLite's design supports. First, the catalog is locked when in use, restricting access to a single Lightroom instance, so the catalog is not being shared by two or more machines; sharing is limited to the Lightroom processes running on the same machine.

Second, catalog corruption was usually attributed to faulty network file locking software. In LrC 7.3 /7.3.1 released in April, 2018, the catalog was converted to WAL mode, which does not use network file locking.

Third, performance is no longer such a major issue. Since the catalog is not shared across machines, it can be cached in local memory, which is now plentiful. Gigbit/s Ethernet is common/inexpensive, and network equipment providing higher speeds (2.5 - 5 Gbps) over the same wiring (CAT5e) is becoming available as well (though still expensive).

My point is that using a catalog on a network file system is no longer an inherently bad idea, as are some of the other methods proposed in various forums for sharing catalogs. It would be useful to repeat the tests done by Adobe in 2007 with catalogs on both network and local drives to measure the relative risk of corruption. And as I posted previously, Adobe could maintain a compatibility list of NAS/fileserver products they've tested, as they already do for GPUs.
 
It appears that accessing a Lightroom catalog on a network filesystem has gotten a bad reputation because, years ago, catalog corruption was common due to software bugs which are now rare -- the software involved has matured.
I don't think this is relevant.
SQLLite is a single user database. It has no login security to prevent multiple uses from opening the same database file (a Lightroom Catalog for instance). With the file on the network, two or more users are able to make changes. These concurrent changes violate data integrity. User 1 can be changing the same records that user 2 is accessing. The result is a corrupted database. In addition to data tables, there are indexes that get rebuilt by the database engine. Then you have two database engines updating on the same index tables and you quickly trash the database.
A Client/server database like Oracle manages users and prevents different users from accessing the same table rows by locking at the row level. This is not possible in the single user database as row level locking is not even implemented
 
I don't think this is relevant.
SQLLite is a single user database.
From the SQLite documentation at Appropriate Uses For SQLite (emphasis mine) :
Situations Where A Client/Server RDBMS May Work Better
  • Client/Server Applications

    If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because of the latency associated with most network filesystems, performance will not be great. Also, file locking logic is buggy in many network filesystem implementations (on both Unix and Windows). If file locking does not work correctly, two or more clients might try to modify the same part of the same database at the same time, resulting in corruption. Because this problem results from bugs in the underlying filesystem implementation, there is nothing SQLite can do to prevent it.

It has no login security to prevent multiple uses from opening the same database file (a Lightroom Catalog for instance) ...

From Lightroom catalog cannot be opened (emphasis mine):
When Lightroom Classic is running with a catalog open, it creates a [yourcatalogname].lrcat.lock file next to the [yourcatalogname].lrcat file. This file ensures that there is no other access to the catalog in use. When you exit Lightroom Classic, the lock file is deleted automatically.
 
I don't think this is relevant.
SQLLite is a single user database. It has no login security to prevent multiple uses from opening the same database file (a Lightroom Catalog for instance). With the file on the network, two or more users are able to make changes. These concurrent changes violate data integrity. User 1 can be changing the same records that user 2 is accessing. The result is a corrupted database. In addition to data tables, there are indexes that get rebuilt by the database engine. Then you have two database engines updating on the same index tables and you quickly trash the database.
A Client/server database like Oracle manages users and prevents different users from accessing the same table rows by locking at the row level. This is not possible in the single user database as row level locking is not even implemented
To Clee's point. While the original poster may understand the risk of multiple users accessing (or trying to access) the Lightroom catalog, I daresay that some number of forum participants, as well as the vast majority of Lightroom users in the world, simply do not. Inevitably, there will be "Adobe bugs" reported when two or more family members try to do Lightroom work from different systems "to save time," or some such.

I'm enough of a tech-geek to appreciate the points in this thread. But if there is one thing I learned in my "day job" career as a software product manager, it's that you can't expect your users to be technically proficient about your product. Otherwise, a company's support team would consist of just one, very unhurried person.

Phil Burton
 
To Clee's point. While the original poster may understand the risk of multiple users accessing (or trying to access) the Lightroom catalog, I daresay that some number of forum participants, as well as the vast majority of Lightroom users in the world, simply do not. Inevitably, there will be "Adobe bugs" reported when two or more family members try to do Lightroom work from different systems "to save time," or some such.

I'm enough of a tech-geek to appreciate the points in this thread. But if there is one thing I learned in my "day job" career as a software product manager, it's that you can't expect your users to be technically proficient about your product. Otherwise, a company's support team would consist of just one, very unhurried person.

Phil Burton
I generally agree with you, but this thread is about Catalog on a NAS? It's the NAS part that has required far more expertise and maintenance than I anticipated. Storing and using LrC catalogs on one is a relatively minor addition, with significant benefits, for those who've managed to set up a NAS together with a backup strategy. I've had a good experience with accessing catalogs on a NAS from Windows PCs since 2011. My recent investigations into the underlying software (LrC on Windows 10, SQLite, SMB, SAMBA) have uncovered no technical issues, but a lot of dated and/or incorrect information in various forums. Nor have I found any actual tests/measurements of the relative risk of catalogs on NAS vs. local drives. I suspect it to be negligible.

I keep a repository of documentation and configuration files for my setup here: kgorlen/lightroom
 
Status
Not open for further replies.
Back
Top