Catalog on NAS?

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
#1
OK, this has probably been asked before, but.......

I've just upgraded my network and storage setup - I have a Windows 10 desktop PC in my office (dedicated concrete 'man-shed' in the back garden). I have Cat 6 GbE for internet and the like, plus a dedicated 20,000 Mb/s link (2 x 10Gb-SR in LACP over fibre optic) to a Synology RS3617xs with 12 x 4 TB HDDs in RAID10. NAS is in the house for security reasons and the fact that it's FAR TOO LOUD to tolerate in my office.

Rather than running my catalog on the local PC and having to back it and the backups up to the NAS, and then having the NAS sync the backup to the local storage/cloud, is there a really good reason why I couldn't/shouldn't run the catalog from the NAS on an iSCSI target volume?

My plan going forward is to keep photo folders and catalogs together in one place if possible and keep the catalog sizes relatively small.

I've read lots of arguments about network speed limitations being a reason to not have the catalog networked, but my network transfer speeds are significantly faster than even a local SATA SSD would be. Are there other good technical reasons why it's not a good idea?
 

Tony Jay

Senior Member
Staff member
Moderator
Lightroom Guru
Joined
Jun 8, 2012
Messages
2,418
Location
Brisbane, Australia
Lightroom Experience
Advanced
#2
The most up to date information that I have is that this is impossible.
The catalog needs to be on a drive local to the computer - a NAS does not count in this regard.

A couple of versions back I think it was possible (Lr4.x and Lr5.x).

Welcome to Lightroom Forums BTW!

Tony Jay
 
Joined
Jul 24, 2012
Messages
27
Location
Arlington, VA
Lightroom Experience
Intermediate
#3
I don't have a speed-demon network, but I have a similar setup as you. I put the current year's photos and the catalog on an SSD local to my computer, then sync to the NAS. Older photos are stored physically on the NAS (but the previews are on the local SSD). I put the catalog and the preview folder within my Dropbox (which is on the SSD). In this fashion, my "current" photos load quickly when need be (and the previews for older ones are "local"). The catalog syncs with revision(s) to dropbox (which has come in handy), but the whole arrangement uses less than 60 GB on my local SSD. Like you, my NAS syncs to the cloud so that there are numerous backups distributed geographically (and that's good as we had our house struck by lightning last year and catch fire!). My sense is that the catalog and the files you're working on (zooming in, etc) should be on the fastest SSD you can manage locally... Others may refute this, but I have little speed bottlenecks with this arrangement.
 

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
#4
The most up to date information that I have is that this is impossible.
The catalog needs to be on a drive local to the computer - a NAS does not count in this regard.

A couple of versions back I think it was possible (Lr4.x and Lr5.x).

Welcome to Lightroom Forums BTW!

Tony Jay
Thanks Tony. I know Lightroom won't allow the catalog to run on a network drive per se, however an iSCSI target NAS isn't a 'network' drive as far as the OS is concerned. It's block level storage which can be formatted to various file systems and appears to the OS as a local hard drive. Hence my question.
 

Tony Jay

Senior Member
Staff member
Moderator
Lightroom Guru
Joined
Jun 8, 2012
Messages
2,418
Location
Brisbane, Australia
Lightroom Experience
Advanced
#5
Thanks Tony. I know Lightroom won't allow the catalog to run on a network drive per se, however an iSCSI target NAS isn't a 'network' drive as far as the OS is concerned. It's block level storage which can be formatted to various file systems and appears to the OS as a local hard drive. Hence my question.
If that is the case then it might work.
Why not give it a go!
Just remember to back everything up in case of disaster.

Let us know how you go.

Tony Jay
 

clee01l

Senior Member
Staff member
Moderator
Lightroom Guru
Joined
Jun 20, 2009
Messages
13,742
Location
Bellaire, TX USA
Lightroom Experience
Power User
Lightroom Version
Classic 7
#6
...an iSCSI target NAS isn't a 'network' drive as far as the OS is concerned. It's block level storage which can be formatted to various file systems and appears to the OS as a local hard drive...
As long as it is not connected through an ethernet port it is not a network connection. But if you access it through the ethernet connection LR won't recognize the drive a a local drive.
 

Wernfried

Active Member
Joined
Jan 31, 2017
Messages
136
Lightroom Experience
Intermediate
Lightroom Version
Classic 7
#7
As long as it is not connected through an ethernet port it is not a network connection. But if you access it through the ethernet connection LR won't recognize the drive a a local drive.
I think this is a very simplified statement. Nowadays you can virtualise almost everything. You can hide the network share easily with a symbolic link.
Some weeks ago I made even a test where I put my Catalog to a virtual hard disk which exist only in system memory.

Best Regards
Wernfried
 

JohanElzenga

Lightroom Guru
Staff member
Moderator
Lightroom Guru
Joined
Jul 2, 2015
Messages
7,163
Location
Netherlands
Lightroom Experience
Power User
Lightroom Version
Classic 7
#8
I don't think that the question whether Lightroom sees the disk as a local disk is very relevant. What is relevant is whether it is safe to use such a setup. As far as I know, it is always technically possible to start
Iightroom from a catalog on a shared disk, but you run a great risk of catalog corruption. For that reason I would never trust my production catalog to such a setup, even if the system presents the disk as a virtual local disk and even if initial tests seem to indicate that Lightroom can be fooled into thinking that it is.
 

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
#9
This is exactly what I'm trying to understand. What is the risk of running the catalog on a non-local drive? Why is there a 'great risk of catalog corruption'? I understand that once upon a time network connection speed could have been an issue. In these days of 10Gb and 40Gb network connections over fibre to high performance NAS, speed is no longer a factor.
 

Roelof Moorlag

Active Member
Joined
Jan 27, 2013
Messages
881
Location
Netherlands
Lightroom Experience
Power User
Lightroom Version
Classic 7
#10
A network is the most likely cause of dropped connections (what causes corruption in the Lightroom database). Old network cables, overheating and/or overworked switches/hubs, creaky old NIC's and badly written device drivers are the most common problem areas for this. Bandwith alone is not enough.
 

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
#11
A network is the most likely cause of dropped connections (what causes corruption in the Lightroom database). Old network cables, overheating and/or overworked switches/hubs, creaky old NIC's and badly written device drivers are the most common problem areas for this. Bandwith alone is not enough.
Roelof, thanks for your reply, but I'm really looking for hard facts, rather than conjecture.

My network cables aren't cables, they're custom length, steel-wire-armoured pre-terminated, fully tested OM3 fibre-optics, made by a network company who supply to the UK military. There's no switch involved, connection is direct. NICs are new, made by Intel, running Intel's latest driver set which underwent extensive design and testing to allow use of VLANs and teaming in non-server Windows SKUs - long running saga here: I211/I217-V Windows 10 LACP teaming fails

Dual connections are bonded via LACP, which provides redundancy and seamless failover: Understanding Link Aggregation Control Protocol - Technical Documentation - Support - Juniper Networks

A flaky network is something I don't have.

If no one can really explain Why I shouldn't do it, I'll have to do as Tony suggested - Try it out and report my findings....
 

JohanElzenga

Lightroom Guru
Staff member
Moderator
Lightroom Guru
Joined
Jul 2, 2015
Messages
7,163
Location
Netherlands
Lightroom Experience
Power User
Lightroom Version
Classic 7
#12
I think it's more than that. Apparently it has something to do with the fact that the Lightroom catalog is based on SQLite, but I'm no expert in this field so I can't tell you exactly what the problem is.
 

Jim Wilde

Lightroom Guru
Staff member
Moderator
Lightroom Guru
Joined
Feb 1, 2010
Messages
10,722
Location
West Sussex, UK
Lightroom Experience
Advanced
Lightroom Version
Classic 7
#13
What I know is that the Lightroom catalog uses SQlite, and that either LR or SQlite (or both) does not allow the catalog to be placed on a network volume. Attempting to open a catalog on a volume which Lightroom detects as being a network volume will fail, and the error message will explain why.

Of course there are ways around this restriction, and over the years some users have reported success. However, one of the Adobe engineers (Dan Tull) did some experimentation in hoodwinking Lightroom into using a network-based catalog, and he reported that he consistently managed to irretrievably corrupt the catalog (and at the time he was the acknowledged expert in repairing corrupted catalogs). Since then, Adobe's position has, I believe, remained unchanged....i.e. network-based catalogs are not supported, period.

Whilst you may have success in getting it to work, we would prefer that the actual technical details are not posted in an open forum post.....not everyone has the same level of technical competence, so blindly trying to replicate what may be complex setup instructions could easily lead the less savvy user into catalog disaster, which we would rather not have happen.
 

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
#14
What I know is that the Lightroom catalog uses SQlite, and that either LR or SQlite (or both) does not allow the catalog to be placed on a network volume. Attempting to open a catalog on a volume which Lightroom detects as being a network volume will fail, and the error message will explain why.

Of course there are ways around this restriction, and over the years some users have reported success. However, one of the Adobe engineers (Dan Tull) did some experimentation in hoodwinking Lightroom into using a network-based catalog, and he reported that he consistently managed to irretrievably corrupt the catalog (and at the time he was the acknowledged expert in repairing corrupted catalogs). Since then, Adobe's position has, I believe, remained unchanged....i.e. network-based catalogs are not supported, period.

Whilst you may have success in getting it to work, we would prefer that the actual technical details are not posted in an open forum post.....not everyone has the same level of technical competence, so blindly trying to replicate what may be complex setup instructions could easily lead the less savvy user into catalog disaster, which we would rather not have happen.
Thanks Jim. I appreciate your viewpoint.

Even if I get it to work successfully, it's not something I would advocate that others attempt. I've experienced corrupted catalogs in the past but having a robust backup strategy always prevented any significant loss.

I'll experiment at my own risk and won't share my experiences openly.

If any of the more tech savvy are curious to learn about my experience, feel free to message me.

I'll leave the discussion here.

Thank you everyone for your contributions.

Graeme
 

Jim Wilde

Lightroom Guru
Staff member
Moderator
Lightroom Guru
Joined
Feb 1, 2010
Messages
10,722
Location
West Sussex, UK
Lightroom Experience
Advanced
Lightroom Version
Classic 7
#15
Graeme, no harm in posting back a "yes I did" or "no I did not" get it to work message, I'm sure some folks would like to know the outcome of your test. But thanks for agreeing not to openly post the detailed steps that you use.
 

Gnits

Senior Member
Joined
Mar 21, 2015
Messages
1,095
Location
Dublin, Ireland.
Lightroom Experience
Power User
Lightroom Version
#16
I suspect that the underlying protocols are different when retrieving data from a network than retrieving data via a local storage device. Networks, by their nature have their own means of handling errors and the standards required to react to networked errors may have different response time standards, regardless of the speed of the physical / logical infrastructure.

It is an interesting topic which I am sure will continue to evolve, but I agree, caution required.
 

clee01l

Senior Member
Staff member
Moderator
Lightroom Guru
Joined
Jun 20, 2009
Messages
13,742
Location
Bellaire, TX USA
Lightroom Experience
Power User
Lightroom Version
Classic 7
#17
I think it's more than that. Apparently it has something to do with the fact that the Lightroom catalog is based on SQLite, but I'm no expert in this field so I can't tell you exactly what the problem is.
Yes SQLite is a Single user database and this database makes up the LR catalog file. It detects the location of the catalog file. If this file is located on a network resource, protections for referential integrity are invoked. When the file is located on a network resource, there is no way in the database engine to prevent the file from being opened and edited simultaneously by multiple clients. While you can assure yourself that the file will only be opened by one instance of LR, SQLite can not and there is no database user security being enforced to lock out multiple instances of this file being opened. There is no quicker way to corrupt the catalog file that have two users update then same record at the same time. In a relational database, records in one table are related to other records in other tables and indexes. Making one change ripples out into several tables and indexes. Continuity is maintained by the data base engine. Continuity can not be maintained if several data base engines are operating on the same file.
 

gstrek

New Member
Joined
May 6, 2014
Messages
13
Location
Detroit, Michigan USA
Lightroom Experience
Intermediate
Lightroom Version
4.x
#18
Graeme and anyone else interested. I am a technology professional and agree this is NOT for everyone nor will I make this my production methodology. This is a proof of concept and intellectual, tech exercise only.

I have a Synology NAS with a single 1 GB network connection to a switch (also using VLAN but not LACP). I created an iSCSI LUN and target on the NAS, formatted and mounted it to Windows 7 as drive E:. I was able to create a new LR catalog, import 20 photos and do some edits. It was slow but does work. I should add my Synology NAS is a very low-end device.

SQLite does NOT throw an error as it does when you try to create a new catalog on a network share. iSCSI operates at a much lower level, as Graeme mentioned it presents as a raw, unformatted disk and appears to the OS as a direct-attached storage device.

The is set up for only a single machine to map so this would force a single user/machine to access the catalog.
 
Last edited:
Joined
Feb 3, 2015
Messages
61
Location
Virginia
Lightroom Experience
Power User
#19
If you want a really slow performing catalog, feel free to put it on a NAS.

Sent from my XT1650 using Tapatalk
 

Ferguson

Linwood Ferguson
Staff member
Moderator
Lightroom Guru
Joined
Jan 18, 2009
Messages
1,689
Location
Cape Coral, FL
Lightroom Experience
Intermediate
#20
I suspect that the underlying protocols are different when retrieving data from a network than retrieving data via a local storage device. Networks, by their nature have their own means of handling errors and the standards required to react to networked errors may have different response time standards, regardless of the speed of the physical / logical infrastructure.
That's true of almost all storage, the underlying protocols are different. IDE is brain dead compared to SCSI in some ways. CIFS and SMB (both for accessing NAS data) are different from each other but are treating almost identically. Almost all the protocols have varying degrees of redundancy and robustness. And "network storage" can run over numerous protocols, some with lots of error checking and redundancy, some with very little.

To me there's one big difference, and it actually applies to the (permitted) use of EHD/USB drives -- When you have an internal drive in your computer, absent hardware failure, the drive is up and available all the time the computer is available. When you have a drive outside the computer - NAS for sure, EHD also -- it is extremely easy for the operator (which any statistic you check will show is the least reliable component) to accidentally disconnect it. Unplug the card reader and... ooops, that wasn't the card reader, it was my EHD. You get the idea.

Also, "network" to many people means Wifi, subject to all sorts of issues of interference and capacity.

Also, and I am sure with exceptions, external systems are just not as reliable in most cases. When's the last time an IDE, SCSI, or SAS or SATA cable failed inside a computer, but you hear of external cables (and USB hubs) failing all the time.

Add in you will often have a UPS or battery on your computer, but maybe not on your EHD/NAS. Or a different UPS that might go down separately.

Honestly, Lightroom's technical restrictions aside (to coin a phrase) -- you are just plain better off keeping your images close, and your catalog closer (to your computer).
 

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
#21
If you want a really slow performing catalog, feel free to put it on a NAS.

Sent from my XT1650 using Tapatalk
That may be true for 1000Base-T on a low end NAS. On a high end NAS however with 20Gb network, I expect the results will be rather different.....
 
Joined
Feb 3, 2015
Messages
61
Location
Virginia
Lightroom Experience
Power User
#22
You are still adding about 120ms of latency for every database call... I run bonded 10gigE to my storage server and I still won't put my catalog on there. I even have the storage to saturate it.


Still not worth the added latency vs just keeping the catalog and caches on local nvme storage.

Sent from my XT1650 using Tapatalk
 

Ferguson

Linwood Ferguson
Staff member
Moderator
Lightroom Guru
Joined
Jan 18, 2009
Messages
1,689
Location
Cape Coral, FL
Lightroom Experience
Intermediate
#23
Just bear in mind that what you guys are discussing is pretty darn far away from what the average lightroom user is dealing with. Adobe is trying to keep guys with aging laptops from shooting themselves in the foot. It's kind of like someone with a 10" polar mount telescope saying to the guy with the point and shoot "I don't see why you can't get a good shot of that nebula, mine works fine".
 

Wernfried

Active Member
Joined
Jan 31, 2017
Messages
136
Lightroom Experience
Intermediate
Lightroom Version
Classic 7
#24
What I know is that the Lightroom catalog uses SQlite, and that either LR or SQlite (or both) does not allow the catalog to be placed on a network volume. Attempting to open a catalog on a volume which Lightroom detects as being a network volume will fail, and the error message will explain why.
In general you can create a SQLite database on a network share, see SQLite Hompage: SQLite database files may be shared accross a network using a network filesystem. This is never a particularly efficient method and may have problems (depending on the filesystem, or may simply not be available. These are alternative techniques for remote access to SQLite databases.

The issue is not the network drive as such, the problem is on a SHARED drived other processes may access the same database file at the same time as you do which can corrupt the database.
As long as you can ensure that only one process (i.e. one single Lightroom instance) access the catalog, you should not get any issue.

If someone likes to read some technical details how a SQLite database get corrupt (resp. where it does not) here you go: How to Corrupt an SQLite Database. However, it might be difficult to be understood for non-engineers.

Wernfried
 

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
#25
You are still adding about 120ms of latency for every database call... I run bonded 10gigE to my storage server and I still won't put my catalog on there. I even have the storage to saturate it.


Still not worth the added latency vs just keeping the catalog and caches on local nvme storage.

Sent from my XT1650 using Tapatalk
I'm curious about such high latency. What setup are you using that gives 120ms latency per database call?
 
Top