Catalog on NAS?

Status
Not open for further replies.
G

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
OK, this has probably been asked before, but.......

I've just upgraded my network and storage setup - I have a Windows 10 desktop PC in my office (dedicated concrete 'man-shed' in the back garden). I have Cat 6 GbE for internet and the like, plus a dedicated 20,000 Mb/s link (2 x 10Gb-SR in LACP over fibre optic) to a Synology RS3617xs with 12 x 4 TB HDDs in RAID10. NAS is in the house for security reasons and the fact that it's FAR TOO LOUD to tolerate in my office.

Rather than running my catalog on the local PC and having to back it and the backups up to the NAS, and then having the NAS sync the backup to the local storage/cloud, is there a really good reason why I couldn't/shouldn't run the catalog from the NAS on an iSCSI target volume?

My plan going forward is to keep photo folders and catalogs together in one place if possible and keep the catalog sizes relatively small.

I've read lots of arguments about network speed limitations being a reason to not have the catalog networked, but my network transfer speeds are significantly faster than even a local SATA SSD would be. Are there other good technical reasons why it's not a good idea?
 
Sort by date Sort by votes
If LR is installed locally on the NAS (actually another computer on the network running an OS that can install LR, Then every thing is local to that machine. You would need to login remotely to that machine to run LR, Your computer them would behave as a dumb terminal. I have not attempted this remote login recently, but the screen refresh on the terminal is glacial.
 
Upvote 0 Downvote
clee01l said:
I have not ! this remote login recently, but the screen refresh on the terminal is glacial.
Click to expand...

I hear you! This truly is a half baked idea. Thinking outside the box actually requires that you know where and what the box is. For sure. But I knew some computer animators working for a large movie studio you have heard of in Southern California. All their jobs got moved to canada and during and after the transition stragglers would work from terminals down south with super fast connections to their data (large amounts) from a keyboard and monitor. What I am suggesting would be local over a 10gbe connection.

I am still researching this idea. I’ll start a new thread (maybe elsewhere) if I get us too off topic but at the heart of this thread is a storage issue. Many of us are pushing Lightroom Classic’s limitations on the desktop and have workflows not yet well supported by Lightroom CC (just LightRoom now!)

To the other comment yes, the NAS stops being a NAS when you use it as I have discussed but within one’s own NAS wouldn’t I have the all the other advantages of what people normally get a NAS to do? Backup automation, expandability, etc. I mean I would have all the tools necessary to create another virtual machine in support of a local workgroup provided I have good fast local cables and switches in support of 10gbe? I confess I have limited experience with VMware. I have built machines on AWS but that’s a lot different than running VMWare on my own hardware I would imagine.

It’s just an idea. I am new to the class. Send me a private message if you want me to take this elsewhere, I get it. This is a LightRoom forum. And trust me if I had synchronous bi-directional gigabit INTERnet many of my current issues with LightRoom CC and LightRoom Web would probably not be issues. I am just trying to play to my advantages as they are now which are super fast local network (10gbe) and storage and 300/30 internet speed. Time is money and I shoot a LOT.

Thanks all.

Michael



Sent from my iPhone using Tapatalk
 
Last edited:
Upvote 0 Downvote
Michael Bateman said:
I hear you! This truly is a half baked idea. Thinking outside the box actually requires that you know where and what the box is. For sure. But I knew some computer animators working for a large movie studio you have heard of in Southern California. All their jobs got moved to canada and during and after the transition stragglers would work from terminals down south with super fast connections to their data (large amounts) from a keyboard and monitor. What I am suggesting would be local over a 10gbe connection.

I am still researching this idea. I’ll start a new thread (maybe elsewhere) if I get us too off topic but at the heart of this thread is a storage issue. Many of us are pushing Lightroom Classic’s limitations on the desktop and have workflows not yet well supported by Lightroom CC (just LightRoom now!)

To the other comment yes, the NAS stops being a NAS when you use it as I have discussed but within one’s own NAS wouldn’t I have the all the other advantages of what people normally get a NAS to do? Backup automation, expandability, etc. I mean I would have all the tools necessary to create another virtual machine in support of a local workgroup provided I have good fast local cables and switches in support of 10gbe? I confess I have limited experience with VMware. I have built machines on AWS but that’s a lot different than running VMWare on my own hardware I would imagine.

It’s just an idea. I am new to the class. Send me a private message if you want me to take this elsewhere, I get it. This is a LightRoom forum. And trust me if I had synchronous bi-directional gigabit INTERnet many of my current issues with LightRoom CC and LightRoom Web would probably not be issues. I am just trying to play to my advantages as they are now which are super fast local network (10gbe) and storage and 300/30 internet speed. Time is money and I shoot a LOT.

Thanks all.

Michael



Sent from my iPhone using Tapatalk
Click to expand...
Michael,

If you have 10 GB Ethernet and have or are willing to put up a NAS,, then you should have the funds for a reasonably fast workstation with a big enough SSD. Having the LR catalog on a NAS is a real performance boost. If you shoot a lot, then keep your actual images files on a fast (7200 rpm) HDD, also local. SSDs have dropped a lot in price in the past year.

There is also a more fundamental issue. The Lightroom catalog is based on SQLite, which is designed for local operation only, no networked operation. In Windows (not sure about MacOS), the file access commands are different between local access and network access. With network access, you risk subtle but real catalog corruption.

Phil
 
Upvote 0 Downvote
PhilBurton said:
you should have the funds for a reasonably fast workstation with a big enough SSD. Having the LR catalog on a NAS is a real performance boost. If you shoot a lot, then keep your actual images files on a fast (7200 rpm) HDD, also local. SSDs have dropped a lot in price in the past year.
Click to expand...

Huge thank you brother! The price drop in SSDs in the past year has kinda snuck up on me. You are correct I might just have a look and rethink this. I do think you meant to say that having the catalog on an SSD is a huge performance boost. Yeah.

At the end of the day my issues with Lightroom are that it’s not an enterprise tool nor a workgroup tool and that I like it too much! I am addicted to the ease in which I am able to edit massive amounts of photos while chilling out on the couch! I am such a self indulgent lazy old guy dang it! I have this love/like relationship with Lightroom mobile. I wish I could have adobe’s cloud in my own server rack. I want to have a bunch of friends over to take wildlife pictures in the woods out back and have them waiting for us back at the house ready to edit when we go back. But I digress.

Thanks for indulging my (probably futile) brainstorming. This is a great group. Thanks very much.

Michael


Sent from my iPhone using Tapatalk[/QUOTE]
 
Upvote 0 Downvote
Michael Bateman said:
Huge thank you brother! The price drop in SSDs in the past year has kinda snuck up on me. You are correct I might just have a look and rethink this. I do think you meant to say that having the catalog on an SSD is a huge performance boost. Yeah.

At the end of the day my issues with Lightroom are that it’s not an enterprise tool nor a workgroup tool and that I like it too much! I am addicted to the ease in which I am able to edit massive amounts of photos while chilling out on the couch! I am such a self indulgent lazy old guy dang it! I have this love/like relationship with Lightroom mobile. I wish I could have adobe’s cloud in my own server rack. I want to have a bunch of friends over to take wildlife pictures in the woods out back and have them waiting for us back at the house ready to edit when we go back. But I digress.

Thanks for indulging my (probably futile) brainstorming. This is a great group. Thanks very much.

Michael


Sent from my iPhone using Tapatalk
Click to expand...
[/QUOTE]
Michael,

Yes, I meant SSD when I wrote NAS. And yes, this is a truly great group of people.

I am somewhat surprised that Adobe hasn't tried to build an workgroup/enterprise version of Lightroom, because the rest of their Creative Suite products are all workgroup/enterprise ready and have corporate pricing. it very well may be that eventually, eventually (and i am completely speculating here) that the cloudy version will evolve with all of the features of the desktop version and be inherently workgroup/enterprise ready. I think I know the technical obstacles to converting the current desktop version to a workgroup tool, but it's not worth any discussion here.

If you have your own server rack, then (1) I am bright green with envy, and (2) may I suggest the following components for a killer homebuilt Lightroom workstation:

 
Upvote 0 Downvote
Since this thread appears at the top of Google results for "lightroom catalog NAS", I'd like to clarify the technical issues of why Lightroom restricts catalogs from being placed on network volumes. (Note that Storage Area Network (SAN) systems using SCSI or iSCSI over ethernet are completely different technical beasts and there is no evidence indicating that placing a catalog on a SAN volume would lead to corruption.) Here is what I wrote in the Adobe Lightroom forums:

------------------------------------------------
Re: Operating Lightroom CC Classic via network drive?

The issue with catalogs stored on network drives is that SQLite relies on the filesystem to provide correctly working file locking, and when the LR team was testing this 12 years ago, many implementations of network file systems didn't implement file locking correctly. The SQLite documentation warned about it:
"One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under Windows. Your best defense is to not use SQLite for files on a network filesystem."
Click to expand...
Senior Adobe engineer Dan Tull did some quickie experiments in 2007 [using NFS, not SMB] and found that by simply disconnecting a network cable, he could corrupt a LR catalog stored on a file server. Given that there were many potential network-attached storage (NAS) products with this problem, Adobe felt, not unreasonably, that the risk and cost of users corrupting their catalogs was too high. There used to be a hidden config.lua switch to override LR's prohibition, but that stopped working a long time ago.

See this long discussion topic starting here, including Dan Tull's contributions: Lightroom Classic and CC: Allow Catalog to be stored on a networked drive. | Photoshop Family Custom...

Twelve years later, and the landscape has changed considerably -- Windows SMB implementations are much more mature and Samba has become the standard non-Microsoft implementation of the SMB protocol. LR has switched to using the write-ahead-logging mode (WAL) of SQLite, which has different issues with network files. I'm not aware of anyone who has done recent tests with network-stored LR catalogs or written authoritatively about SQLite and newer file-server implementations.

Adobe has shown no inclination to re-examine the issue. Last year they changed LR to use the higher-performance write-ahead logging mode of SQLite and broke the longstanding ability to import from catalogs stored on network servers. Even though the fix for that is trivial, Adobe hasn't implemented it. I'm not sure the LR team has the requisite engineering skills to re-examine competently network storage of catalogs.

Finally, note that there are two somewhat distinct issues: Allowing users one-user-at-time access to catalogs stored on network servers, and allowing multiple users to access server-stored catalogs concurrently. The former is perhaps solvable with a small amount of re-engineering of how LR uses SQLite or perhaps doing a relatively low-level port to another database engine. The latter would require significant re-engineering of LR, introducing an application-level server architecture (and probably a database more amenable to that architecture, e.g. MySQL), and introducing new concepts and features into the user interface.
 
Upvote 0 Downvote
gfinlayson said:
OK, this has probably been asked before, but.......

I've just upgraded my network and storage setup - I have a Windows 10 desktop PC in my office (dedicated concrete 'man-shed' in the back garden). I have Cat 6 GbE for internet and the like, plus a dedicated 20,000 Mb/s link (2 x 10Gb-SR in LACP over fibre optic) to a Synology RS3617xs with 12 x 4 TB HDDs in RAID10. NAS is in the house for security reasons and the fact that it's FAR TOO LOUD to tolerate in my office.

Rather than running my catalog on the local PC and having to back it and the backups up to the NAS, and then having the NAS sync the backup to the local storage/cloud, is there a really good reason why I couldn't/shouldn't run the catalog from the NAS on an iSCSI target volume?

My plan going forward is to keep photo folders and catalogs together in one place if possible and keep the catalog sizes relatively small.

I've read lots of arguments about network speed limitations being a reason to not have the catalog networked, but my network transfer speeds are significantly faster than even a local SATA SSD would be. Are there other good technical reasons why it's not a good idea?
Click to expand...

Hello,

gfinlayson said:
OK, this has probably been asked before, but.......

I've just upgraded my network and storage setup - I have a Windows 10 desktop PC in my office (dedicated concrete 'man-shed' in the back garden). I have Cat 6 GbE for internet and the like, plus a dedicated 20,000 Mb/s link (2 x 10Gb-SR in LACP over fibre optic) to a Synology RS3617xs with 12 x 4 TB HDDs in RAID10. NAS is in the house for security reasons and the fact that it's FAR TOO LOUD to tolerate in my office.

Rather than running my catalog on the local PC and having to back it and the backups up to the NAS, and then having the NAS sync the backup to the local storage/cloud, is there a really good reason why I couldn't/shouldn't run the catalog from the NAS on an iSCSI target volume?

My plan going forward is to keep photo folders and catalogs together in one place if possible and keep the catalog sizes relatively small.

I've read lots of arguments about network speed limitations being a reason to not have the catalog networked, but my network transfer speeds are significantly faster than even a local SATA SSD would be. Are there other good technical reasons why it's not a good idea?
Click to expand...

Hello gfinlayson,

I was wondering what the outcome was if you attempted to run a Lightroom catalog on an iSCSI target volume or if you found some other way to do so.

Please let me know when time permits. I find myself in a similar situation, and would love to be able to have everything - the catalog and image files - on a centralized storage location.

Thank you!
 
Upvote 0 Downvote
You couldn't do it in 2012, and you still can't.
 
Upvote 0 Downvote
Hal, nevrsmer is asking about using iSCSI, which is a SAN (Storage Area Network) technology for accessing disks over the network. SAN is much different than NAS (Network Attached Storage), which is what most people want to use with LR. To the client operating system, the remote disks appear to be locally attached. The file-locking issues that LR and SQLite have with NAS server implementations don't apply to SAN volumes, since the client operating system is doing all locking exactly the same as with locally attached disks.

A quick web search shows a number of reports of people successfully placing LR catalogs on SAN volumes residing on Synonology, QNAP, and other such devices:

https://www.youtube.com/watch?v=JaCKA1YaOoM
https://forum.qnap.com/viewtopic.php?p=453677&sid=5b34dec9be3591cca1c7fc0d54b2c656https://community.spiceworks.com/topic/192537-lightroom-on-nas-san
 
Upvote 0 Downvote
John,
I stand corrected. Thanks. :)
 
Upvote 0 Downvote
iSCSI doesn't support locking, so the shared volume's filesystem can be corrupted if two or more client machines attempt to create or write files at the same time, unless they are all running a clustered file system specifically designed for this type of configuration (Clustered file system - Wikipedia).

I've been accessing catalogs on a NAS from Winodws PCs since 2011. See kgorlen/lightroom for details on why this works, the risks involved, and how to set it up.
 
Upvote 0 Downvote
Excellent detail, thanks for sharing that.

You say, "iSCSI doesn't support locking". If one client machine is accessing an iSCSI drive, and multiple processes on that single machine place OS locks on the same file, will those local locks work properly?
 
Upvote 0 Downvote
Yes, that should work. It's equivalent to accessing a directly attached (unshared) drive, except for performance.
 
Upvote 0 Downvote
That's good to know. As you've read, SQLite depends on file locking to operate correctly, and a multi-threaded program that accesses a database concurrently depends on correct locking. I believe that LR has multiple threads accessing the database concurrently, which is why Adobe engineer Dan Tull's NAS tests failed in 2007 (his environment was testing an NFS file server with broken locking).
 
Upvote 0 Downvote
Yes. The README shown at the link I gave above (kgorlen/lightroom) has references to Dan Tull's tests and relevant SQLite/SMB documentation. The situation is similar to that of GPUs: Lightroom *should* work with any GPU that *correctly* implements a supported interface (DirectX 12, Metal), but Adobe has actually tested some to assure compatibility and provides a list. Adobe could do the same for popular NAS and file server products and remove the network drive restriction.
 
Upvote 0 Downvote
The difference between GPUs and file servers, of course, is that a failing GPU is non-destructive of the catalog. While certifying popular NAS products makes eminent sense, I doubt that after all these years Adobe would do it. (I doubt whether the LR team has the requisite systems-engineering skills -- the engineers who built LR are all long gone.)
 
Upvote 0 Downvote
I have the reverse problem... sort of...

In my quest to run Lightroom 6.14 on my MacBook Pro with Catalina, I have installed a Virtual Machine running Mojave and installed Lightroom 6.14 on that. It works perfectly. So far so good.

My main catalog and all my image files are on my Mac Mini. I just want to use the MacBook Pro if I'm travelling (which won't be happening for a long time) and for being able to review some images while I'm sitting in the backyard or even in front of TV (and when I'm back travelling to work by train). For the reviewing, I would just export as catalog (with smart previews) from my main catalog and copy to the MacBook Pro. I used to do this with my old MacBook and it worked well.

I would rather copy them to local drive of the MacBook and then access them via the virtual machine so I don't have to make too large a virtual drive on the VM. The problem is that shared folders appear as network folders on the VM and so Lightroom won't let me open them.

Does anyone know of a workaround? They aren't strictly on the network and there's no way I can access them on from Lightroom on Catalina.
 
Upvote 0 Downvote
I think the best you can do is to put the catalog folder on the virtual machine's drive and put the photos on the physical drive. But that's mildly inconvenient, since after exporting the catalog from your Mini, you'll have to separate the photos from the catalog.
 
Upvote 0 Downvote
johnrellis said:
I think the best you can do is to put the catalog folder on the virtual machine's drive and put the photos on the physical drive. But that's mildly inconvenient, since after exporting the catalog from your Mini, you'll have to separate the photos from the catalog.
Click to expand...
I don't need the photo files as I'm exporting with Previews and Smart Previews. I end up with three files per export: Catalog, Smart Previews, and Previews. They all have to stay together.
 
Upvote 0 Downvote
I don't speak mac, but in windows you can mount a virtual drive in the hypervisor as well. So for example if my machine X has a VM Y, and Y has a virtual disk Z, when Y is not running I can mount Z in X and use it.

Could something like that help? A separate virtual drive that is mounted either in the VM or in the main machine, just not both?

(Note if it's different OS versions that might complicate things if the drive formats changed).

Again, I don't quite know how to spell Mac. Mc. Something like that.
 
Upvote 0 Downvote
Upvote 0 Downvote
Ferguson said:
I don't speak mac, but in windows you can mount a virtual drive in the hypervisor as well. So for example if my machine X has a VM Y, and Y has a virtual disk Z, when Y is not running I can mount Z in X and use it.

Could something like that help? A separate virtual drive that is mounted either in the VM or in the main machine, just not both?

(Note if it's different OS versions that might complicate things if the drive formats changed).

Again, I don't quite know how to spell Mac. Mc. Something like that.
Click to expand...
It is easy enough on a Mac to create a disk image that could be mounted to either the host of the guest - in fact that's exactly how Mojave was installed on the guest.

The problem with this approach is that it carves off a chunk of the host's hard drive and so really doesn't provide any benefit over just making the drive of the guest larger.
 
Upvote 0 Downvote
Status
Not open for further replies.
Back
Top