• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom 6 questions. All the information you need is in Adobe Lightroom 6 - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. When you upgrade to subscription, there's also a Lightroom Classic version available.
  • It's Lightroom update time again! New cameras (including the Canon R5/R6), lens profiles and bug fixes, and the ability to disable built-in lens corrections for specific new cameras. Here's the usual list

Catalog on NAS?

gorlen

New Member
Joined
Jul 26, 2020
Messages
19
Location
San Francisco
Lightroom Experience
Advanced
Lightroom Version
Classic
Other factors are: (1) Lightroom now accesses the catalog in SQLite WAL (Write-Ahead Log) mode: changes are written to a .wal file and later applied to the .lrcat catalog file. ... SQLite provides a function to flush changes to storage. I don't know if Lightroom uses it, and if it actually causes the changes to be written all the way back to the server.
I used several tools (watch, inotifywait, wireshark) to observe catalog operations while having LR sync develop settings across >3K photos:

- Windows does set oplocks on the .lrcat* files, enabling client caching.​
- Data is frequently flushed to the .lrcat-wal file on the server.​
- Occasionally, SQLite checkpoint operations apply the WAL data to the .lrcat file on the server.​
- Other than the initial oplocks, no other lock requests are sent to the server. (According to the documentation, WAL mode limits database access to the (Lightroom, in this case) processes on a single PC, and these processes coordinate through shared memory.)​
- SQLite also creates a shared memory-mapped file (.lrcat-shm) in the catalog folder. According to the documentation, this file contains no persistent content and is not used for recovery.​
- SQLite creates temporary files, but none were observed in the catalog folder, so I presume that they're in a temp folder on a local drive.​
This is all good behavior. Since "one test is worth a thousand expert opinions", it would be interesting to force Windows to crash (see notmyfault) while LR is performing various catalog modifications to measure the relative risk of corruption with the catalog on a network drive vs. a local drive. I suspect it's negligible.
 
Joined
Jan 18, 2009
Messages
2,293
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Glad to see the detail; you have the energy many of us lack to dig into it.

While you are set up, I'd be curious to hear how much activity you see while lightroom is otherwise idle (but with a good sized catalog). I've always worried more about people who might just leave it sitting, especially since Cloudy came along and people want it up as a receiver for sync'd photos from mobile. Then the power goes out (or whatever).

Of course, as to testing, as they say -- it's hard to prove a negative. You can just draw some statistics around it.
 

gorlen

New Member
Joined
Jul 26, 2020
Messages
19
Location
San Francisco
Lightroom Experience
Advanced
Lightroom Version
Classic
While you are set up, I'd be curious to hear how much activity you see while lightroom is otherwise idle (but with a good sized catalog). I've always worried more about people who might just leave it sitting, especially since Cloudy came along and people want it up as a receiver for sync'd photos from mobile. Then the power goes out (or whatever).
Opening a catalog with >30K photos and then letting it idle for 30 min. resulted in 1670 SMB2 ops total on the lrcat* files on the server. No Write ops after open. LR just does a Find op on the catalog folder every 10s while it's idling. My LR isn't syncing with cloud, though.
 
Joined
Jan 18, 2009
Messages
2,293
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Opening a catalog with >30K photos and then letting it idle for 30 min. resulted in 1670 SMB2 ops total on the lrcat* files on the server. No Write ops after open. LR just does a Find op on the catalog folder every 10s while it's idling. My LR isn't syncing with cloud, though.
That's interesting. I thought it was always doing stuff in the background, but maybe only if there are things to do, like purging 1:1 previews that have expired (I realize that's not in the catalog, not sure if it updates anything there), address lookups, or such.

Thank you forchecking.
 

gorlen

New Member
Joined
Jul 26, 2020
Messages
19
Location
San Francisco
Lightroom Experience
Advanced
Lightroom Version
Classic
That's interesting. I thought it was always doing stuff in the background, but maybe only if there are things to do, like purging 1:1 previews that have expired (I realize that's not in the catalog, not sure if it updates anything there), address lookups, or such.

Thank you forchecking.
Since I thought your concern was about the catalog, I only checked ops on the three lrcat* files. Running
Code:
inotifywait -r -m --timefmt "%H:%M:%S" --format "%T %w %e" -o events.txt Master
on the server shows that LR searches the Backups, Helper, and Previews folder trees once/hour while idle.
 
Joined
Jan 18, 2009
Messages
2,293
Location
Fort Myers, FL
Lightroom Experience
Advanced
Lightroom Version
Classic
Since I thought your concern was about the catalog, I only checked ops on the three lrcat* files.
It is really, that was the only example of a known idle time thing I could think of. :whistling:
 

PhilBurton

Lightroom enthusiast (but still learning)
Premium Classic Member
Premium Cloud Member
Joined
Nov 16, 2015
Messages
2,249
Location
California, USA
Lightroom Experience
Intermediate
Lightroom Version
Classic
I would like again to make the point that @gorlen is easily in the top 10%, if not the top 1%, of technical skills in this forum. For most everyone else, you probably don't know how to cope with some Windows issue created by using a NAS, that @gorlen just fixes without even thinking about it. SQLite was not designed to work over a network, including a NAS. I personally have experienced problems with the personal finance manager Quicken, when keeping the Quicken data files on a network share. Quicken also uses SQLite.

We all know that catalog corruption very real, and sometimes even Adobe can't even recover a corrupted catalog. Get a larger HDD, get an external HDD that plugs in via USB, but please don't use a NAS. Your time and effort are more than worth the expense.
 

gorlen

New Member
Joined
Jul 26, 2020
Messages
19
Location
San Francisco
Lightroom Experience
Advanced
Lightroom Version
Classic
SQLite was not designed to work over a network, including a NAS.
From SQLite CVSTrac :

Using SQLite on a Network
SQLite database files may be shared accross a network using a network filesystem. This is never a particularly efficient method and may have problems (depending on the filesystem, or may simply not be available. These are alternative techniques for remote access to SQLite databases.
It appears that accessing a Lightroom catalog on a network filesystem has gotten a bad reputation because, years ago, catalog corruption was common due to software bugs which are now rare -- the software involved has matured.

Furthermore, a Lightroom catalog presents a less demanding situation than what SQLite's design supports. First, the catalog is locked when in use, restricting access to a single Lightroom instance, so the catalog is not being shared by two or more machines; sharing is limited to the Lightroom processes running on the same machine.

Second, catalog corruption was usually attributed to faulty network file locking software. In LrC 7.3 /7.3.1 released in April, 2018, the catalog was converted to WAL mode, which does not use network file locking.

Third, performance is no longer such a major issue. Since the catalog is not shared across machines, it can be cached in local memory, which is now plentiful. Gigbit/s Ethernet is common/inexpensive, and network equipment providing higher speeds (2.5 - 5 Gbps) over the same wiring (CAT5e) is becoming available as well (though still expensive).

My point is that using a catalog on a network file system is no longer an inherently bad idea, as are some of the other methods proposed in various forums for sharing catalogs. It would be useful to repeat the tests done by Adobe in 2007 with catalogs on both network and local drives to measure the relative risk of corruption. And as I posted previously, Adobe could maintain a compatibility list of NAS/fileserver products they've tested, as they already do for GPUs.
 
Joined
Jun 20, 2009
Messages
16,488
Location
Houston, TX USA
Lightroom Experience
Power User
Lightroom Version
Cloud Service
It appears that accessing a Lightroom catalog on a network filesystem has gotten a bad reputation because, years ago, catalog corruption was common due to software bugs which are now rare -- the software involved has matured.
I don't think this is relevant.
SQLLite is a single user database. It has no login security to prevent multiple uses from opening the same database file (a Lightroom Catalog for instance). With the file on the network, two or more users are able to make changes. These concurrent changes violate data integrity. User 1 can be changing the same records that user 2 is accessing. The result is a corrupted database. In addition to data tables, there are indexes that get rebuilt by the database engine. Then you have two database engines updating on the same index tables and you quickly trash the database.
A Client/server database like Oracle manages users and prevents different users from accessing the same table rows by locking at the row level. This is not possible in the single user database as row level locking is not even implemented
 

gorlen

New Member
Joined
Jul 26, 2020
Messages
19
Location
San Francisco
Lightroom Experience
Advanced
Lightroom Version
Classic
I don't think this is relevant.
SQLLite is a single user database.
From the SQLite documentation at Appropriate Uses For SQLite (emphasis mine) :
Situations Where A Client/Server RDBMS May Work Better
  • Client/Server Applications

    If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because of the latency associated with most network filesystems, performance will not be great. Also, file locking logic is buggy in many network filesystem implementations (on both Unix and Windows). If file locking does not work correctly, two or more clients might try to modify the same part of the same database at the same time, resulting in corruption. Because this problem results from bugs in the underlying filesystem implementation, there is nothing SQLite can do to prevent it.
It has no login security to prevent multiple uses from opening the same database file (a Lightroom Catalog for instance) ...
From Lightroom catalog cannot be opened (emphasis mine):
When Lightroom Classic is running with a catalog open, it creates a [yourcatalogname].lrcat.lock file next to the [yourcatalogname].lrcat file. This file ensures that there is no other access to the catalog in use. When you exit Lightroom Classic, the lock file is deleted automatically.
 

PhilBurton

Lightroom enthusiast (but still learning)
Premium Classic Member
Premium Cloud Member
Joined
Nov 16, 2015
Messages
2,249
Location
California, USA
Lightroom Experience
Intermediate
Lightroom Version
Classic
I don't think this is relevant.
SQLLite is a single user database. It has no login security to prevent multiple uses from opening the same database file (a Lightroom Catalog for instance). With the file on the network, two or more users are able to make changes. These concurrent changes violate data integrity. User 1 can be changing the same records that user 2 is accessing. The result is a corrupted database. In addition to data tables, there are indexes that get rebuilt by the database engine. Then you have two database engines updating on the same index tables and you quickly trash the database.
A Client/server database like Oracle manages users and prevents different users from accessing the same table rows by locking at the row level. This is not possible in the single user database as row level locking is not even implemented
To Clee's point. While the original poster may understand the risk of multiple users accessing (or trying to access) the Lightroom catalog, I daresay that some number of forum participants, as well as the vast majority of Lightroom users in the world, simply do not. Inevitably, there will be "Adobe bugs" reported when two or more family members try to do Lightroom work from different systems "to save time," or some such.

I'm enough of a tech-geek to appreciate the points in this thread. But if there is one thing I learned in my "day job" career as a software product manager, it's that you can't expect your users to be technically proficient about your product. Otherwise, a company's support team would consist of just one, very unhurried person.

Phil Burton
 

gorlen

New Member
Joined
Jul 26, 2020
Messages
19
Location
San Francisco
Lightroom Experience
Advanced
Lightroom Version
Classic
To Clee's point. While the original poster may understand the risk of multiple users accessing (or trying to access) the Lightroom catalog, I daresay that some number of forum participants, as well as the vast majority of Lightroom users in the world, simply do not. Inevitably, there will be "Adobe bugs" reported when two or more family members try to do Lightroom work from different systems "to save time," or some such.

I'm enough of a tech-geek to appreciate the points in this thread. But if there is one thing I learned in my "day job" career as a software product manager, it's that you can't expect your users to be technically proficient about your product. Otherwise, a company's support team would consist of just one, very unhurried person.

Phil Burton
I generally agree with you, but this thread is about Catalog on a NAS? It's the NAS part that has required far more expertise and maintenance than I anticipated. Storing and using LrC catalogs on one is a relatively minor addition, with significant benefits, for those who've managed to set up a NAS together with a backup strategy. I've had a good experience with accessing catalogs on a NAS from Windows PCs since 2011. My recent investigations into the underlying software (LrC on Windows 10, SQLite, SMB, SAMBA) have uncovered no technical issues, but a lot of dated and/or incorrect information in various forums. Nor have I found any actual tests/measurements of the relative risk of catalogs on NAS vs. local drives. I suspect it to be negligible.

I keep a repository of documentation and configuration files for my setup here: kgorlen/lightroom
 
Top