• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Catalog on NAS?

Status
Not open for further replies.

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
OK, this has probably been asked before, but.......

I've just upgraded my network and storage setup - I have a Windows 10 desktop PC in my office (dedicated concrete 'man-shed' in the back garden). I have Cat 6 GbE for internet and the like, plus a dedicated 20,000 Mb/s link (2 x 10Gb-SR in LACP over fibre optic) to a Synology RS3617xs with 12 x 4 TB HDDs in RAID10. NAS is in the house for security reasons and the fact that it's FAR TOO LOUD to tolerate in my office.

Rather than running my catalog on the local PC and having to back it and the backups up to the NAS, and then having the NAS sync the backup to the local storage/cloud, is there a really good reason why I couldn't/shouldn't run the catalog from the NAS on an iSCSI target volume?

My plan going forward is to keep photo folders and catalogs together in one place if possible and keep the catalog sizes relatively small.

I've read lots of arguments about network speed limitations being a reason to not have the catalog networked, but my network transfer speeds are significantly faster than even a local SATA SSD would be. Are there other good technical reasons why it's not a good idea?
 
Yeah, I tried symbolic links. Lightroom still knows the catalog is on a "network" drive.
Does it have a concept of mount points, where the remote drive becomes a folder in another drive?
 
I don't speak mac, but in windows you can mount a virtual drive in the hypervisor as well. So for example if my machine X has a VM Y, and Y has a virtual disk Z, when Y is not running I can mount Z in X and use it.

Could something like that help? A separate virtual drive that is mounted either in the VM or in the main machine, just not both?

(Note if it's different OS versions that might complicate things if the drive formats changed).

Again, I don't quite know how to spell Mac. Mc. Something like that.
Ferguson,

I rarely use VMs, but the ability to mount that virtual disk Z sounds interesting. How do you do that? I'm using MS's VM feature.

Phil
 
If it's a hyperv disk, i.e. a vhd or vhdx, just make sure no VM is using it, right click and say "mount"; it's usually the top option in the context menu.

Do NOT do this if there's a checkpoint outstanding as it will break the chain between parent and deltas and you will not be able to start the VM. But if it's just a regular drive you can mount it, even if it's the system disk for a VM.

And do NOT let two things mount it at once (e.g. you and the VM running). I think it checks and prevents it, but not sure.
 
iSCSI doesn't support locking, so the shared volume's filesystem can be corrupted if two or more client machines attempt to create or write files at the same time, unless they are all running a clustered file system specifically designed for this type of configuration (Clustered file system - Wikipedia).

I've been accessing catalogs on a NAS from Winodws PCs since 2011. See kgorlen/lightroom for details on why this works, the risks involved, and how to set it up.

I recently discovered and am now trying out a workaround on Windows to allow a catalog on a NAS with previews on a local drive to improve performance: a folder on a local SSD with symlinks to the .lrcat file and the Backups directory on the SAMBA network share.

While LrC will display the "Lightroom cannot open the catalog named ... located on network volume ..." when double-clicking or directly opening the .lrcat symlink, clicking "OK", then choosing the same symlink from the Select Catalog dialog box will succeed, and it can be set as the default startup catalog. Also, LrC locks the catalog on the network share, preventing use/corruption by other LrC instances on the same or other machines; i.e., the .lrcat.lock file LrC creates need not be on the network share in order to lock out other instances.

I'll update the details at kgorlen/lightroom in a few weeks when I'm satisfied that this works without issues.
 
Last edited:
I'm sure one can work around Adobe's restrictions. It's worth asking if it is a good idea. It's not like Adobe did that to make more money selling disk drives.
 
I'm sure one can work around Adobe's restrictions. It's worth asking if it is a good idea. It's not like Adobe did that to make more money selling disk drives.
See kgorlen/lightroom for a description of why it's technically sound and for references to background on how Adobe came to impose this restriction.
 
I've read it in the past. I've done a fair amount of SQLite programming including over networks.

But two considerations:

NAS connectivity is far more problematic than local drives. Especially since so many people use wifi. The frequency with which you may have interrupted I/O operations is likely to be far higher on NAS than a local drive. Every time you have an interruption of I/O on a stupid little SQLite database you risk corruption. It's an odds game.

It's a reason I always recommend people not put catalogs on EHD's. They are more reliable than NAS connectivity but less reliable than local, because there's always a dumb human that might hit the cable. Unfortunately they are awfully convenient for people using multiple computers.

Also, and a moot point for many perhaps: If you ever wanted Adobe to help with a corrupted catalog, telling them it is on a NAS is not motivational. o_O
 
I've read it in the past. I've done a fair amount of SQLite programming including over networks.

But two considerations:

NAS connectivity is far more problematic than local drives. Especially since so many people use wifi. The frequency with which you may have interrupted I/O operations is likely to be far higher on NAS than a local drive. Every time you have an interruption of I/O on a stupid little SQLite database you risk corruption. It's an odds game.

It's a reason I always recommend people not put catalogs on EHD's. They are more reliable than NAS connectivity but less reliable than local, because there's always a dumb human that might hit the cable. Unfortunately they are awfully convenient for people using multiple computers.

Also, and a moot point for many perhaps: If you ever wanted Adobe to help with a corrupted catalog, telling them it is on a NAS is not motivational. o_O
All true, and I'd definitely not do this over wifi.

Other considerations in favor of NAS is that it is easier and less expensive than client PCs to set up (1) with RAID to protect against single-drive failures, (2) with power from a UPS to allow graceful shutdown when power fails, (3) to be "always up" to run nightly backups, when catalogs are less likely to be in use.

Since you've worked with SQLite over networks, perhaps you know the details of how Windows handles database files. From what I've read, Windows will automatically set oplocks on network files and cache them locally. Since only a single Lightroom instance at a time can be accessing the database files, the server won't receive any requests from other clients to break the oplocks, resulting in all/most database operations happening locally until the files are closed. Is this actually what happens?
 
What I've done in the past is access SQLite on windows (e.g. Calibre as a repository) from embedded linux systems (e.g. a player piano system using windows as a library). I recall needing to disable byte range locking to get it to work with smb/cifs (norbl in options). If you didn't disable rbl, write operations would fail (I do not recall if all, or some). In my case it was very well controlled, and only for me (not a client), as I only updated on windows when adding music, so all access was over cifs, and I did not pursue further once I found a workaround. In that case caching on or off made no difference. And that was 2017-ish. Haven't gone back to it (though oddly enough that system failed last week and I'm about to rebuild it, with fresher linux as what I had is no longer updatable, so I might end up refreshing my memory).

Your access is windows to windows, I take it. Frankly I've done little of that with databases; my windows to windows was almost always MS SQL and a bit of Oracle, which isn't filesystem oriented.

Sorry. Long ramble for a short subject: we each must decide our tolerance for risk. I tend to be conservative for things hard to fix, like lightroom's catalog.
 
I recently discovered and am now trying out a workaround on Windows to allow a catalog on a NAS with previews on a local drive to improve performance: a folder on a local SSD with symlinks to the .lrcat file and the Backups directory on the SAMBA network share.

While LrC will display the "Lightroom cannot open the catalog named ... located on network volume ..." when double-clicking or directly opening the .lrcat symlink, clicking "OK", then choosing the same symlink from the Select Catalog dialog box will succeed, and it can be set as the default startup catalog. Also, LrC locks the catalog on the network share, preventing use/corruption by other LrC instances on the same or other machines; i.e., the .lrcat.lock file LrC creates need not be on the network share in order to lock out other instances.

I'll update the details at kgorlen/lightroom in a few weeks when I'm satisfied that this works without issues.
@gorlen,
In your particular use case and with your technical skills, perhaps you have a design that prevents catalog corruption. But in general, as it has been said many, many times in this forum, Adobe engineered the catalog in such a way that storing the catalog on any sort of network drive creates a risk of data corruption.

How much is your time worth? How important is it to absolutely minimize the risk of catalog corruption. Pay The Man and get an additional or larger internal or USB external drive.

And don't forget backups. On this forum, there are many tales of woe posted by people whose catalog got corrupted and didn't have a backup. Back up every single time you exit Lightroom, not just once a week or once a month.
 
Your access is windows to windows, I take it. Frankly I've done little of that with databases; my windows to windows was almost always MS SQL and a bit of Oracle, which isn't filesystem oriented.
Configuration is Windows clients to a NAS running a SAMBA server on some Linux derivative.
 
@gorlen,
In your particular use case and with your technical skills, perhaps you have a design that prevents catalog corruption. But in general, as it has been said many, many times in this forum, Adobe engineered the catalog in such a way that storing the catalog on any sort of network drive creates a risk of data corruption.

Right, I'd not recommend this for the totally technically unskilled. The main problem has been that every couple of years, either QNAP or Netgear or CrashPlan or whoever releases a software upgrade that breaks something critical, and diagnosing the problem and dealing with product tech support is a pain. I'm always looking for more reliable NAS and cloud backup vendors.

From what I've read, it doesn't appear that "Adobe engineered the catalog ...". The Lightroom catalog is an SQLite database.

How much is your time worth? How important is it to absolutely minimize the risk of catalog corruption. Pay The Man and get an additional or larger internal or USB external drive.

Space isn't the issue. I'm not convinced that simply placing a catalog on a local drive provides the best protection. In my experience, power, software, hard disk drives, and manual backup procedures fail much more often than hard-wired LAN components or NAS enclosures.

And don't forget backups. On this forum, there are many tales of woe posted by people whose catalog got corrupted and didn't have a backup. Back up every single time you exit Lightroom, not just once a week or once a month.

Good advice. Reliable backup is my primary motivation for this configuration. Accessing catalogs from multiple PCs is a bonus. Unlike a typical desktop PC, the NAS is set up with RAID to protect against single-drive failures, connected to a UPS so cached writes can be flushed to drives when power fails, and is "always up" to automatically run nightly backups -- when Lightroom catalogs are not likely in use -- to the cloud and to two local archives. The result is five automatic copies of everything, one off-site. The archives, particularly the one off-site, are important protection against ransomware.

And I regularly back up when exiting Lightroom -- to the NAS, of course.
 
Right, I'd not recommend this for the totally technically unskilled. The main problem has been that every couple of years, either QNAP or Netgear or CrashPlan or whoever releases a software upgrade that breaks something critical, and diagnosing the problem and dealing with product tech support is a pain. I'm always looking for more reliable NAS and cloud backup vendors.

From what I've read, it doesn't appear that "Adobe engineered the catalog ...". The Lightroom catalog is an SQLite database.



Space isn't the issue. I'm not convinced that simply placing a catalog on a local drive provides the best protection. In my experience, power, software, hard disk drives, and manual backup procedures fail much more often than hard-wired LAN components or NAS enclosures.



Good advice. Reliable backup is my primary motivation for this configuration. Accessing catalogs from multiple PCs is a bonus. Unlike a typical desktop PC, the NAS is set up with RAID to protect against single-drive failures, connected to a UPS so cached writes can be flushed to drives when power fails, and is "always up" to automatically run nightly backups -- when Lightroom catalogs are not likely in use -- to the cloud and to two local archives. The result is five automatic copies of everything, one off-site. The archives, particularly the one off-site, are important protection against ransomware.

And I regularly back up when exiting Lightroom -- to the NAS, of course.
@gorlen,

I'm sure you realize that you are probably in the top 10% of technical skills of the members of this forum. The "other 90%" need a total Lightroom hardware/software configuration that is "set it and forget it" with reasonable cost and excellent reliability.
 
@gorlen,

I'm sure you realize that you are probably in the top 10% of technical skills of the members of this forum. The "other 90%" need a total Lightroom hardware/software configuration that is "set it and forget it" with reasonable cost and excellent reliability.

That was exactly my goal when I decided on this configuration nearly 10 years ago. Sadly, it's not worked out as well as I'd hoped. Equipment and storage costs are reasonable, I've not lost data, performance is good, availability is high, and it normally requires only an hour or so monthly to install software updates (for security). But configuration is complicated and error-prone, and I've experienced serious outages every couple of years when software updates are defective or make unexpected changes to functionality. When this happens, my computer engineering experience is needed.

But I wonder if *any* consumer electronics product is "set and forget". My Canon cameras have come closest by far to being trouble free. Among everything else -- Windows PCs, NAS, Android phones/tablets, video/audio devices, etc. -- there's always something malfunctioning, nearly always due to software/firmware defects or incompatibilities.

I've posted because "how to share a Lightroom catalog among PCs?" is a frequently-asked question on this and many other forums. The discussions are filled with questionable responses, e.g. "can't be done", "manually switch external drives", "sync with Dropbox/Google Drive", "use iSCSI" (horrors!). So I've contributed my $.02 worth.
 
I hardly have you
That was exactly my goal when I decided on this configuration nearly 10 years ago. Sadly, it's not worked out as well as I'd hoped. Equipment and storage costs are reasonable, I've not lost data, performance is good, availability is high, and it normally requires only an hour or so monthly to install software updates (for security). But configuration is complicated and error-prone, and I've experienced serious outages every couple of years when software updates are defective or make unexpected changes to functionality. When this happens, my computer engineering experience is needed.

But I wonder if *any* consumer electronics product is "set and forget". My Canon cameras have come closest by far to being trouble free. Among everything else -- Windows PCs, NAS, Android phones/tablets, video/audio devices, etc. -- there's always something malfunctioning, nearly always due to software/firmware defects or incompatibilities.

I've posted because "how to share a Lightroom catalog among PCs?" is a frequently-asked question on this and many other forums. The discussions are filled with questionable responses, e.g. "can't be done", "manually switch external drives", "sync with Dropbox/Google Drive", "use iSCSI" (horrors!). So I've contributed my $.02 worth.
I hardly have your computer engineering experience, but I have learned a lot by some reading and a lot of trial-and-error. I am also the de facto IT support person. And I agree that there is "always something" at issue, even as Microsoft and other software companies have made big strides in reducing the need for user expertise. [Do you remember that days when you had to manually set IRQs on your PC, for the parallel and serial ports? I do, and I don't, mixing my meanings here.]

For most people, I think that the less complex their setup, the happier they will be. That probably means using an external hard drive from Western Digital (not Seagate!!!), and staying away from questionable websites.

Even my Nikon D3 is not totally problem free. When the backup batter in the D3 body fell out, or ran low, then I got weird dates for my photos. That's when I learned about EXIFTool and EXIFToolGUI.

I do also remember SCSI, since I worked at the company that produced SASI, but those days are lost in the mists of time.

Phil Burton
 
From what I've read, it doesn't appear that "Adobe engineered the catalog ...". The Lightroom catalog is an SQLite database.
Yes and no. They use SQLite, but they decided how everything inside hangs together, how normalized (or not) the data structures are. To a very real degree, how you design the schema determines how fragile the database is. But more to the current point, how you ACCESS the database also impacts that. Lightroom is very chatty -- even when you are idle (maybe especially when you are idle) it can be making massive updates to the database and doing so in multiple threads. Interruption of these idle process updates may cause corruption. Keeping the database open when not updating, depending on how carefully you manage caching and write logs can cause vulnerabilities in actual idle time.

I do not mean to imply Adobe did this badly -- or well for that matter -- but there is a LOT one can do to make a SQLite database more, or less, vulnerable to corruption. Not all designs are created equal.
 
[Do you remember that days when you had to manually set IRQs on your PC, for the parallel and serial ports? I do, and I don't, mixing my meanings here.]

I do too. That was *much* easier than programming the Peripheral Processors (PPs) on the CDC 6000 series. They didn't have interrupts -- had to poll external devices for status and data. There were 10 PPs, each with 4096 12-bit words of memory. Programmed them with punch cards in the wee hours. Only debugging tool was to force a core* dump of the entire memory, which left the location of the last instruction zeroed out as a hint. *Tiny ferromagnetic donuts strung on wires. (Apologies for the off-topic war story.)
 
I do too. That was *much* easier than programming the Peripheral Processors (PPs) on the CDC 6000 series.
" This book is dedicated to A6 & A7, without which none of the results in this book could have been saved " [Who said I don't remember anything from college]

;)

[Yes, I realize that was the main processor not the PP's; I can't recall any specifics about PP processing though I actually wrote one of my thesis projects using them.]
 
" This book is dedicated to A6 & A7, without which none of the results in this book could have been saved " [Who said I don't remember anything from college]

;)

[Yes, I realize that was the main processor not the PP's; I can't recall any specifics about PP processing though I actually wrote one of my thesis projects using them.]
Good one! Took me a few seconds to get it! :) From what book?
Recently read The Friendly Orange Glow -- fun trip for CDC fans.
 
Yes and no. They use SQLite, but they decided how everything inside hangs together, how normalized (or not) the data structures are. To a very real degree, how you design the schema determines how fragile the database is. But more to the current point, how you ACCESS the database also impacts that. Lightroom is very chatty -- even when you are idle (maybe especially when you are idle) it can be making massive updates to the database and doing so in multiple threads. Interruption of these idle process updates may cause corruption. Keeping the database open when not updating, depending on how carefully you manage caching and write logs can cause vulnerabilities in actual idle time.

I do not mean to imply Adobe did this badly -- or well for that matter -- but there is a LOT one can do to make a SQLite database more, or less, vulnerable to corruption. Not all designs are created equal.
The question is how much risk is added by accessing a catalog on a network share? Here's what makes me suspect that the considerations you mention don't matter: [Chapter 5] 5.5 Locks and Oplocks. Oplocks are enabled by default, and according to Opportunistic Locks - Win32 apps, are transparent to applications.
My understanding is that (1) Lightroom calls the SQLite library to open a catalog, (2) Windows sends the open request to the SAMBA server and sets an oplock on the database file, (3) the SAMBA server sees a bunch of read requests while the catalog is in use, but no lock or write requests, (4) On exit, Lightroom calls the SQLite library to close the database, (5) Windows sends all the modified cached data blocks back to the SAMBA server and closes the database. In this scenario, should Lightroom or the client PC crash while the catalog is open, all the modifications are lost, but the database file hasn't been modified and is left in its initial, consistent state.

Other factors are: (1) Lightroom now accesses the catalog in SQLite WAL (Write-Ahead Log) mode: changes are written to a .wal file and later applied to the .lrcat catalog file. With a catalog folder on a local drive containing a symlink to the .lrcat file on a network share, the .wal file will reside on the local drive; and (2) SQLite provides a function to flush changes to storage. I don't know if Lightroom uses it, and if it actually causes the changes to be written all the way back to the server.

While I'm curious about how this works, Lightroom has been running without catalog issues, I have backups, and photos to process.
 
Good one! Took me a few seconds to get it! :) From what book?
Recently read The Friendly Orange Glow -- fun trip for CDC fans.
It was a teal paperback on CDC architecture, though I cannot recall the book and no longer have it. Though in poking around I actually found it online: http://www.bitsavers.org/pdf/cdc/cyber/books/Grishman_CDC6000AsmLangPgmg.pdf

To your other:

In this scenario, should Lightroom or the client PC crash while the catalog is open, all the modifications are lost, but the database file hasn't been modified and is left in its initial, consistent state.
I find it unlikely that in a typical session of editing all (literally all) updates are cached.

What happens if half the data is flushed and half not at the point of a crash. The half written is unlikely to be transactionally consistent, but likely random, the least recently used perhaps.

Also,

SQLite provides a function to flush changes to storage. I don't know if Lightroom uses it, and if it actually causes the changes to be written all the way back to the server.

You can probably dig through the code to find out, but often (as you apparently hint) these kind of things are limited to calling the next layer's "flush', which may or may not connect all the way down depending on file system involved. Especially since many performance-enhancing well-meaning developers break such chains, like various disk write caches you can turn on and off.

But... I also agree with your concept -- the risk is low, you are aware of the risk.
 
It was a teal paperback on CDC architecture, though I cannot recall the book and no longer have it. Though in poking around I actually found it online: http://www.bitsavers.org/pdf/cdc/cyber/books/Grishman_CDC6000AsmLangPgmg.pdf

Cool -- thanks! It probably post-dates my CDC days since it refers to the Cyber series, which came out in the 70's, Fond memories.

To your other:

I find it unlikely that in a typical session of editing all (literally all) updates are cached.

What happens if half the data is flushed and half not at the point of a crash. The half written is unlikely to be transactionally consistent, but likely random, the least recently used perhaps.

Also,

You can probably dig through the code to find out, but often (as you apparently hint) these kind of things are limited to calling the next layer's "flush', which may or may not connect all the way down depending on file system involved. Especially since many performance-enhancing well-meaning developers break such chains, like various disk write caches you can turn on and off.

But... I also agree with your concept -- the risk is low, you are aware of the risk.

This discussion has been useful. It's gotten me to realize that separating the .lrcat and .wal files probably impairs SQLite's ability to recover databases from power outages/crashes (How To Corrupt An SQLite Database File), so I'm switching to using a symlink to the entire catalog folder on the network share, especially since I've not noticed a significant performance improvement with the previews folder on a local SSD.

OK -- back to actually using LrC to process photos. :)
 
It was a teal paperback on CDC architecture, though I cannot recall the book and no longer have it. Though in poking around I actually found it online: http://www.bitsavers.org/pdf/cdc/cyber/books/Grishman_CDC6000AsmLangPgmg.pdf

To your other:


I find it unlikely that in a typical session of editing all (literally all) updates are cached.

What happens if half the data is flushed and half not at the point of a crash. The half written is unlikely to be transactionally consistent, but likely random, the least recently used perhaps.

Also,



You can probably dig through the code to find out, but often (as you apparently hint) these kind of things are limited to calling the next layer's "flush', which may or may not connect all the way down depending on file system involved. Especially since many performance-enhancing well-meaning developers break such chains, like various disk write caches you can turn on and off.

But... I also agree with your concept -- the risk is low, you are aware of the risk.
Given all the isssues, or the potentials for corrupting the SQLite database I think it would be wise for the 90% of us, and perhaps even the top 10%, to use the database only in configutations that have been tested (adequately ???) by Adobe. If that means havinig the previews on the same drive and folder as the main catalog file, then consider that the "Adobe uncreative configuration performance penalty" and move on.

I wonder just how many forum members made it this far into this thread.
 
While LrC will display the "Lightroom cannot open the catalog named ... located on network volume ..." when double-clicking or directly opening the .lrcat symlink, clicking "OK", then choosing the same symlink from the Select Catalog dialog box will succeed,
I wish it did this on a Mac. :( Although I'm tying to load from symlinked folder but I get the same message no matter how I try to access the catalog.
 
Status
Not open for further replies.
Back
Top