• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Catalog on NAS?

Status
Not open for further replies.

gfinlayson

New Member
Joined
Jan 17, 2012
Messages
9
Location
Maidenhead, UK
Lightroom Experience
Intermediate
Lightroom Version
OK, this has probably been asked before, but.......

I've just upgraded my network and storage setup - I have a Windows 10 desktop PC in my office (dedicated concrete 'man-shed' in the back garden). I have Cat 6 GbE for internet and the like, plus a dedicated 20,000 Mb/s link (2 x 10Gb-SR in LACP over fibre optic) to a Synology RS3617xs with 12 x 4 TB HDDs in RAID10. NAS is in the house for security reasons and the fact that it's FAR TOO LOUD to tolerate in my office.

Rather than running my catalog on the local PC and having to back it and the backups up to the NAS, and then having the NAS sync the backup to the local storage/cloud, is there a really good reason why I couldn't/shouldn't run the catalog from the NAS on an iSCSI target volume?

My plan going forward is to keep photo folders and catalogs together in one place if possible and keep the catalog sizes relatively small.

I've read lots of arguments about network speed limitations being a reason to not have the catalog networked, but my network transfer speeds are significantly faster than even a local SATA SSD would be. Are there other good technical reasons why it's not a good idea?
 
I know this thread is over a year old now but I was wondering about the NAS corruption. Does it corrupt the catalogue only if you attempt to use it? If I were to use NAS to archive catalogues but move them to local disc when I needed to access them, would they still be at high risk of corruption?
 
The SQLite database gets corrupted when the drive is disconnected while your application writes data to it which is not completed.

Some technical detail: How To Corrupt An SQLite Database File

Moving a catalog offline from one drive to another (while LR is not open) is no problem at all.

Great!! I'm having storage issues with my previews chewing through my memory and I hate working with a portable hard drive connected to my laptop. I generally only use a catalogue twice and months apart. Then I can archive the images so this solves a big issue with it eating all my laptop storage.

Thank you!
 
And be careful with external usb drives if powered by usb cable or hub. Other demands may compromise the power delivered and such drives may go to sleep, leaving incomplete database transactions.
 
And be careful with external usb drives if powered by usb cable or hub. Other demands may compromise the power delivered and such drives may go to sleep, leaving incomplete database transactions.


Noted. It is another reason I don't like working off tethered storage. I made a rookie mistake two years ago and confused solid state and HDD. Having used only solid state for so long, I didn't think twice when I accidentally knocked the cable out and lost a whole day's work because the transfer wasn't finalised.
 
I stopped using NAS some time ago because copy speed is too slow over a typical network. Faster interfaces now but beware of the slowest link in the chain and fast NAs can be expensive. I have two internal drives, one production, second is backup and 3rd backup external, which can be NAS.
 
Unfortunately, finances don't stretch to internal drives and the amount of work I produce makes it prohibitive (nearly 2TB in contracting work alone last year, then my personal and private work on top of that). And it compounds every year so 3.5 years of contracting for this company and I have nearly 6TB that they want me to store indefinitely at my cost (don't get me started on how shit that is. I don't like it. I don't agree with it. But for now, it is what it is). My desktop is too old and won't let me use Lightroom anymore since the update so I am stuck using a laptop that was only ever meant to be my secondary machine.
 
I have been grappling with this same issue and wanted to know what you think of my solution. I have not implemented this yet, I'm just planning for a 10gbps home network upgrade soon and wanted to know if anyone could see any problems with the following:

When I rebuild our home server rack I figured I'd install a beefy photo editing machine in a 4RU box. High end graphics card, CPU with lots of cores, onboard M.2, 10gbps NIC, maybe a larger SATA SSD for preview storage and not much else. The OS, Lightroom and its catalogue files will all be stored locally on the M.2 SSD and any preview files can go on the SATA drive. All DNG files would be stored on a large NAS in the same server rack and accessed over the 10gbps network connection.

I figure then I can just have a little all-in-one desktop computer in the home office and remote into the powerful computer to use lightroom over the network. I could even do this from a laptop in the living room this way. Really any computer on the home network could remote in and take advantage of the editing machine in the server room. None of the other computers would need to hold a copy of the catalogue or be very powerful at all.

Obviously two users can't use the catalogue simultaneously but I see this as a good work around to the catalogue syncing issue between computers.

What do you think?
 
@Focus that sounds like a decent approach. Some comments though: Really high end graphics cards are unlikely to help much, and will not help at all if you are accessing it remotely; so far as I know Lightroom (and I think Photoshop) cannot use a graphics card other than the one driving the display (compared to various video editors where you can have cards dedicated to them). I may be wrong - anyone?

The 10gbps is probably wasted but not expensive and can't hurt, I doubt you can saturate a 1gbs card. But maybe, depends a lot on the NAS. Obviously over wifi to a laptop (if you are not wired) depends on the wifi.

I have the same setup basically on storage, an M.2 for much, then SATA SSD for photos. I did some experimentation of moving stuff around and testing and did not see much difference with the catalog on slower vs faster disks for most operations; the previews on fast disk made a much bigger difference as did ACR cache and temp storage. So I went back to putting the catalog on Raid SSD drives just for a tiny bit of extra reliability. I found it virtually impossible to saturate an M.2 card (vs. SATA SSD which you can), by the way. Yes, there are some catalog operations that are slow, but the vast majority of the time I am waiting for the computer it's in develop mode, where the catalog is not terribly involved.

Be sure you find a good approach to backing up the result, and some mechanism for making sure you never confuse who has the master catalog if you are moving it around as you describe. I worry a lot in such cases about human error (perhaps because this human is so unreliable).

Also, for a NAS box, try to find one that has a file system aimed at detecting and correcting bit rot, e.g. one using zfs, btrfs, etc. Most don't.

Finally the one caveat I'll offer in that setup is the LAN connection. CIFS/SMB is not as robust generally as direct disk access. It's pretty robust, widely used, but if there is a weak point in your setup where corruption could creep in I would say it's that connection (assuming a quality NAS box). You might also look at iSCSI as an alternative that may be both a bit faster and more reliable, if you can get both clients and NAS that support it.
 
Be sure you find a good approach to backing up the result, and some mechanism for making sure you never confuse who has the master catalog if you are moving it around as you describe. I worry a lot in such cases about human error (perhaps because this human is so unreliable).
QUOTE]

So with what I was describing you would never need to have a master copy of a catalogue. There would only be one copy and that would stay on the server machine. The other computers would just remote in and use this copy. Then when they log out, the catalogue would stay where it was, locally stored on the server machine.
 
So with what I was describing you would never need to have a master copy of a catalogue. There would only be one copy and that would stay on the server machine. The other computers would just remote in and use this copy. Then when they log out, the catalogue would stay where it was, locally stored on the server machine.
Yes, of course, sorry, just so much in the habit of people trying to use multiple copies.
 
Yes, of course, sorry, just so much in the habit of people trying to use multiple copies.

Have you had any experience running creative cloud on Windows server OS? Does it work? Is it stable? Any downsides?
 
Have you had any experience running creative cloud on Windows server OS? Does it work? Is it stable? Any downsides?
I have not. I have found most programs though work the same, though Microsoft of course says one is "tuned" for interactive use, one for background tasks. Some programs explicitly check and won't install on desktop, or on server, if they think their home is the opposite. No idea on CC. Someone here may chime in.

There's very little any more that, on modest hardware (i.e. not-data-center type) that you can't do on Windows Pro that you can on Server. One though that is changing is rfs file systems, I think beginning soon (now?) you can't use them on regular pro, but need a special version of windows 10 or server. I think rfs is immature though, personally (at least it was when I last looked). It had the design to be a zfs-like file system, but they had not finished its support -- no obvious way to scan for problems, to fix them, it was all "set it and forget it and hope it worked because we won't tell you anything" type of microsoft fluff when I last looked. Storage pulls with ntfs, on the other hand, I use on pro and am quite happy with (though you need some powershell magic to get much use of them).
 
I have not. I have found most programs though work the same, though Microsoft of course says one is "tuned" for interactive use, one for background tasks. Some programs explicitly check and won't install on desktop, or on server, if they think their home is the opposite. No idea on CC. Someone here may chime in.

There's very little any more that, on modest hardware (i.e. not-data-center type) that you can't do on Windows Pro that you can on Server. One though that is changing is rfs file systems, I think beginning soon (now?) you can't use them on regular pro, but need a special version of windows 10 or server. I think rfs is immature though, personally (at least it was when I last looked). It had the design to be a zfs-like file system, but they had not finished its support -- no obvious way to scan for problems, to fix them, it was all "set it and forget it and hope it worked because we won't tell you anything" type of microsoft fluff when I last looked. Storage pulls with ntfs, on the other hand, I use on pro and am quite happy with (though you need some powershell magic to get much use of them).
RFS = ???

Phil Burton
 
RFS = ???

Sorry, mis-remembered, it's ReFS, for Resiliant File System.

Resilient File System (ReFS) overview

btrfs and zfs are two better known (linux-y) competitors. The object is to have a design that allows (a) interruptions (e.g. unexpected power outage) that cannot corrupt data only cause loss of what was being written, (b) provide built in checksums or similar for validating file contents is not changing and is read correctly, and (c) optionally provide redundancy for automated recovery of data if corruption is detected. It's ideal for long term storage of large amounts of data.

Rather than adopt a current standard, Microsoft had to reinvent the wheel of course.

When I built my last PC I was going to use it and experimented heavily, even posted for Microsoft's comment the observation that there were no management tools exposed (they agreed, and said "future"). Lately they announced they are pulling support from WIndows 10 Pro for it.
 
Hi,

I've tried not using a SCSI system, instead i have used a sync solution which keeps the files on the computer that runs Lightroom.
Synology NAS offers Sync capabilities, with client and server apps. It's named Synology Cloud Station Drive or Synology Drive.

Lightroom access the catalog from a local path (not network), then as soon as i change something in the catalog, it sync back to the NAS. I usually need to access previews and make slight changes in the metadata. Because the referenced files are not available, the metadata are saved in the catalog, until the referenced files are available again.

When i'm the same local network as where the NAS is located, I also access the referenced media and i can save the metadata in the file or save them in the sidecar.

The only issue with that is that you must be the only Lightroom user which access this catalog at the same time. Otherwise, things screws up and you start getting dupes and .conflicts files.

hope this helps.
 
I've got a similar setup, but my catalog file is stored locally on the particular service that I'm using. I use Resilio Sync to synchronize the catalog to my Synology NAS which in turn synchronizes the catalog on my other computer.

This works just fine and I've had no catalog issues in some 3 years of doing this way.

I tried using Google Drive to store the catalog, but it was always getting corrupted.

Resilio Sync is free to use and there is a native app for Synology NAS boxes.

Highly recommended.
 
hi, im planning to go nas on my files. ive just started researching.
so-- will lightroom work on a synology nas set up with the catalog on my laptop and the files on the nas drive?
anything we need to do? im interested in not doing iscsi, i have a mac computer.

thanks
 
Yes, that'll work miktyur. It can be a bit slower accessing the image files across a network, so you might want to build smart previews for the files you're actually working on (Library menu > Previews > Build Smart Previews) and check the Prefer smart previews instead of originals when editing checkbox in Preferences > Performance to make that a bit quicker.
 
Yes, that'll work miktyur. It can be a bit slower accessing the image files across a network, so you might want to build smart previews for the files you're actually working on (Library menu > Previews > Build Smart Previews) and check the Prefer smart previews instead of originals when editing checkbox in Preferences > Performance to make that a bit quicker.
thanks victoria! now off to save up for a nas drive :)
 
thanks victoria! now off to save up for a nas drive :)
Some pointers: A Gigabit Ethernet cable attached from router to computer and the (little) computer on the network that is the NAS will give you the best possible speeds for data transfer. I have been the NAS route and eventually abandoned the project due to performance issues moving data through a network. For your Mac a non network solution might be best. If your Mac supports Thunderbolt 2 or Thunderbolt 3, getting a TB3 external disk drive will give you transfer speeds equivalent to the internal Disk drive. I moved from a NAS to several TB3 EHDs to hold my data that does not need to fit on my internal Mac.
If you have multiple computers on the network, you can share data disks on one computer with any other computer (this is what a NAS does in essence) I have an iMac and a MBP. I can get to any shared file on my iMac from my MBP as long as both are on the same network. Of course any time you move data across an ethernet network, you are limited to the available throughput speed of the ethernet.
 
Some pointers: A Gigabit Ethernet cable attached from router to computer and the (little) computer on the network that is the NAS will give you the best possible speeds for data transfer. I have been the NAS route and eventually abandoned the project due to performance issues moving data through a network. For your Mac a non network solution might be best. If your Mac supports Thunderbolt 2 or Thunderbolt 3, getting a TB3 external disk drive will give you transfer speeds equivalent to the internal Disk drive. I moved from a NAS to several TB3 EHDs to hold my data that does not need to fit on my internal Mac.
If you have multiple computers on the network, you can share data disks on one computer with any other computer (this is what a NAS does in essence) I have an iMac and a MBP. I can get to any shared file on my iMac from my MBP as long as both are on the same network. Of course any time you move data across an ethernet network, you are limited to the available throughput speed of the ethernet.
thanks for that info. i will use it for storage only and everytime id work on a project ill move them on my computer. was after the redundancy of a drive. i have ended up with 20 or more 4tb externals lol. was looking into a synology 8 bay to add drives slowly
 
Okay. Here’s a dopey idea:

What if you had a really tricked out NAS with LOTS of RAM and cores and a fast NIC and network and built a virtual machine on it. All the storage would be “local” to the (virtual) machine running Lightroom. You could have a huge expandable volume for storing images, no?

I know this is an expensive option but so long as you had a good console to access the virtual machine you’d be set. This would NOT solve any of the multiuser issues.

Just an idea. Thanks for this thread. Lots of find information here. Including a lot of things that WON’T work. Maybe I have added to the second list!

Michael



Sent from my iPhone using Tapatalk
 
If it's a VM running "on" the NAS and local to it, then it's not a NAS at all (from the perspective of the VM). Well, unless rather than using the storage locally you access it by a CIFS/SMB/NFS/etc share, which would be a bit silly.

NAS more or less by definition means attached over the network. What it's implemented inside of it is a whole different matter. I keep a NAS system that I built from my prior desktop, and use it for backup. It didn't stop being the same desktop hardware when I turned it into a NAS. Indeed, it actually DOES have local programs running on it, including a network video recorder. The NVR accesses the drives locally, not as NAS. My nightly backup of my current desktop accesses the drives over the network, i.e. as a NAS. Same drives, it's about access, not device.

I could (and used to) run Lightroom locally on it with the catalog stored locally. I should not run lightroom on my current desktop with the same catalog in the same place accessing it over the network.
 
Status
Not open for further replies.
Back
Top