LR 5.7 Lag Free Switching Between Images In A Big Catalog - Is It Possible?

Status
Not open for further replies.

Sprocket

New Member
Joined
May 19, 2011
Messages
21
Lightroom Experience
Intermediate
Background: I have spent some time re-building a catalog for a friend. The image and video files are on a NAS (Synology 1513+ with WD Red drives configured as RAID 6) and I've been using a PC to do the transfers from a multitude of external drives. He wants to run LR from an i-Mac which has an SSD so it is pretty quick, but I'm having some problems getting LR on the Mac to see the NAS files correctly so I'm stalled at the moment. To give him access to his files while I struggle with the NAS issue, I copied all the smart previews to an external 2.5 inch HD connected via USB3 since they are too big to fit on the Mac internal disk(s). It works, but he is very disappointed with the responsiveness of this temporary setup and I now suspect that the NAS over Gigabit Ethernet is not going to be as snappy as he is hoping either.

So much for the background, here are the questions...

1) He insists on having all his files in a single catalog which is currently around 350,000 (7+ Tb) and likely to grow much bigger still with an increasing number of movies. It wouldn't have been my choice and it's a PITA for loading and backing up, but can anyone tell me what bearing catalog size has on the speed of reviewing images, assuming that LR is running on an SSD?

2) Given the amount of data and the need for redundancy, the NAS seemed like the best option. I didn't expect it to be as responsive as a fast local drive but is there something I missed?

3) How would you configure the most responsive system?

4) Finally, is LR the best tool for this job?

Hope that's not too many questions but I'd really appreciate some advice on this. Many thanks in advance!
 
OK, First you need to resolve the NAS issue. Is the NAS mounted in OS X with full read/write privileges? Are you trying to import from the NAS? using the ADD option? How fast is the GiGabit Ethernet? are there bottlenecks? Is the iMac wired to the network or is it wireless? Are your cables Cat6 and routers and switches true gigabit?

One other point, iMacs do not come with SSDs. The newer iMacs come with a Fusion Drive which is part SSD and part conventional drive. Where the data is stored on this fusion drive is transparent to the user.
And to address your questions:

  1. He insists on having all his files in a single catalog which is currently around 350,000 (7+ Tb) and likely to grow much bigger still with an increasing number of movies. It wouldn't have been my choice and it's a PITA for loading and backing up, but can anyone tell me what bearing catalog size has on the speed of reviewing images, assuming that LR is running on an SSD?
    No one has found an upper limit for images cataloged that affects performance. An system backup solution (timeMachine (preferred) or CrashPlan or (as I do) both) makes backups a painless background task. Once implemented you can forget it until you need to recover. Master original images do not change UNLESS you write metadata changes to the original file or DNG. Don't do this and the Image file will get backed up one time by timeMachine and never touched again. All of your metadata changes are made in the catalog and backups of your backup catalogs are essential for catalog stability and integrity.
  2. Given the amount of data and the need for redundancy, the NAS seemed like the best option. I didn't expect it to be as responsive as a fast local drive but is there something I missed?
    I use a combination. Current active images are stored on the local drive as they change frequently and the masters are accessed frequently. inactive master images are stored more remotely. (i.e. the NAS) Your previews folder stored with the catalog on the fastest local drive will be accessed frequently. Only when there is no current preview available will LR go first to ACR Cache and then to the master Original RAW file. In other words for the current active images everything will take place on the local drive (especially if you store current active images on the local drive.
  3. How would you configure the most responsive system?
    As mentioned in #2 Catalog, Preview Cache and ACR Cache on the local drive. Active image (!3 months worth) on the local drive. Inactive images on a remote drive since they will be accessed infrequently. Do not create smart previews unless you are using LR Mobile and need the files for an iPad or Android tablet. Converting to DNG is not necessary. The Original RAW images are fine for LR.
  4. Finally, is LR the best tool for this job?
    Lightroom is as efficient and any other single user DAM tool available.
 
Thanks for the detailed reply.

NAS & Mac: Sorry, I was mistaken about the ssd in the Mac - it's one of the other machines on the network. I'm not a Mac Person (in case you hadn't guessed) and the issue with the NAS has something to do with relative paths as far as I can tell. The shared NAS folder appears in Finder under "share" and also under "network" and I can access the files in Finder with no problem. Having migrated the catalog from a PC to the local Mac HD, I expected the files on the NAS to show as "missing" and then simply to point to the parent folder in order for LR to re-link them. LR, however, shows two instances of the NAS and attempts to tell it where the files are don't seem to work. I'd like to be able to simply type in the correct path but see no obvious way to do it. All the RAW files are converted to DNG and don't contain any significant development data that needs to be saved, so as a last resort if I can't sort it out I guess could start a new catalog and just add the files from the NAS with no copying or converting.

Network: This is not implemented fully yet as the modem/router has to be changed today (to a new Asus N-55U) but the Mac and NAS will both be linked to the router with short Cat 5e (or 6 if I can get them) cables and the other two machines on the network are via wireless. I see no potential bottlenecks in this configuration, do you? The temporary set-up I left in place was a direct USB3 link to an external drive.

Backup: I was referring to backup of the catalog as a pain, which along with integrity check and optimization seems to take about half an hour, also when starting LR for the first time (on my PC with ssd and the NAS) it takes minutes to load before the number of files in each folder appears in the left panel. The image files will be backed up in a similar way to your suggestion - as you say, most don't change very often. I used to use Crashplan when it first came out and didn't find it as trouble free as I'd hoped, but maybe it's time for another look :) . The catalog will live on the Mac, be backed up daily to the NAS which in turn will be backed up to an offsite location.

Smart Previews: I created Smart Previews because the user travels frequently and wants access to the catalog while away. Internet access is not always available so smart previews on an external drive looked like a good option.

Best Configuration: Thank you for the advice - it looks like the internal drive on the Mac will have to be upgraded. Does splitting the images across two drives complicate your backup routine? We took the decision to convert all RAW files to DNG to conserve space before the NAS was available so I wouldn't want to change that system now unless there was a really good reason. Other than the overhead when importing, what drawbacks does archiving the files as DNG have?

Sorry for the additional questions!
 
A point of clarification. The Catalog must be located on a Locally attached drive. It can not be located on the NAS although the master images can be. You have not clearly stated that the catalog is on a local drive but it sounds like it might be so.

The catalog, the Previews folder and the Smart Previews folder need to be on a Locally attached drive. That locally attached drive can be connected via Thunderbolt, USB 3 or USB 2 (in decreasing order of performance). An iMac is a desktop machine and not usually considered "portable" or "mobile". So I am confused here. So, is it an iMac with a fusion drive or a MacBookPro with perhaps an internal SSD?

Because of referential integrity checks and database optimization will take a while to chug through 350K cataloged images. I do my catalog backups once a week. Daily does not make for an efficient use of time. Let LR run its catalog backup once a week and you should be ok. If you still want daily copies of the catalog file then you can shut down LR and using the OS tools simply make a copy of the catalog file and skip the referential integrity and database optimization. This can be scripted using the Automator app which might be beyond your level of Mac expertise.

TimeMachine will only work on locally attached drives and not on a remote NAS drive . The NAS should be running it's own system backup however. So as long as the BNAS ia being backed up it matters little if the OS X computer does it (using CrashPlan) or if the OS on the NAS initiates a system backup. A RAID system is not a backup system. It only offers data redundancy. RAID filesystems are usually proprietary and not easily recovered when the RAID controller fails. (Ask me how I know this.) RAID Redundancy does not offer protection against accidental deletions or other versioning errors LR is not likely to update the original image file like Word might replace an earlier word document with a badly revised later edition. For this reason you need to run a system backup that will allow you to recover the same file in the state that it was in 6 months ago. As long as you are not Updating the metadata in the DNGs then the DNG file will get written to the NAS once and never updated. That you have DNGs is not a problem, however, the original RAW file might be more important in the future. If you replaced the original Proprietary RAW file with the DNG to save space, you will never be able to take advantage of any future technological processing advances developed by the camera manufacturer if you deleted the proprietary RAW file. Converting to DNG to save space is not a reason to convert to DNG. You need to keep a copy of the original file that came out of the camera. However, It sounds like you have already burned that bridge.

I have my master image files stored on three different drives. One is the internal HDD and the other two are 1TB Firewire800 attached HDDs. Time Machine backs up all of my critical user data onto a differ 3TB Firewire800 HDD. CrashPlan replicate the backup of my critical user data to another 3TB (USB2) HDD and to the CrashPlan Cloud. According to CrashPlan, I have about 750GB of critical user data. It is backed up in three places - Time Machine backup, CrashPlan local backup & CrashPlan Cloud backup.

If you have a catalog that shows missing folders in the Folder panel, all that you should need to do is right click on the top most level folder and choose {Update Folder location...} from the context menu. Is this not working for you? Perhaps you need to provide a screen shot of the system dialog that opens when you click on {Update Folder location...}

I hope I haven't missed any additional questions. Re Ask if there is still confusion. However, you need Cat6 ethernet and patch cables if you want Gigabit throughput. Run some network speed tests when you have the network fully configured. Don't be surprised if the I/O is less that 1/2 Gigabit. Lots of tuning may be evolved to get the network speed up to a decent number.
 
Mac/NAS/Speed

The machine is an 27inch iMac (which I'm now sitting at) - here's a copy of the Hardware Overview:

Model Name: iMac
Model Identifier: iMac11,3
Processor Name: Intel Core i5
Processor Speed: 2.8 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 8 MB
Memory: 8 GB
Processor Interconnect Speed: 4.8 GT/s
Boot ROM Version: IM112.0057.B00
SMC Version (system): 1.59f2
Serial Number (system): W80298QZDB5
Hardware UUID: 9C6F30C2-6CAA-53B5-B6E5-C37543BADAB2

It's running OS X Yosemite 10.10.1. No Fusion drive on this one, as you can see. I understand that the catalog, previews etc have to be on the local drive which is 1Tb with less than 200Gb available, hence using an external unit for the temp setup with the smart previews.

I didn't make it clear, but when he goes away he takes a MacBook Pro with a 1Tb ssd. The iMac stays home!

I have the backup catalog preference set to every time I exit LR but skip it unless there is a lot of changes just so the regular user won't forget. A week is too long and may be several thousand images.

Thanks, for the paranoia inducing RAID Controller scenario - I don't think I'm ready to hear about your experience at this time! R6 at least allows for a double failure (unless it's during a rebuild?) but, just in case the original RAW files still reside on about 25 external drives and none will be recycled until the NAS backup is in place (and tested). The arguments for and against DNG are many and varied but having jumped that way I think we'll stay with them although I've not really considered that a new Cam Raw might suddenly change the game dramatically (I don't convert my own files so I'm not a true DNG believer myself).

I'll put in the new router now and try to get you screenshot to illustrate the problem that LR is having re-linking to the files.
 
Last edited:
If the mobile computer is a MBP and the catalog is on an iMac, you still don't necessarily need Smart previews on the iMac. What catalog is being used when the user is mobile on the MBP?

OSX, LR and other apps running will eat up that free space with swapfiles and working storage. So I would try to keep that 200 GB free. If LR is needing to create intermediate files in working storage and there is no free space available to do it, there will be more induced lag when switching images.

My iMac is the same except that I have 16GB RAM instead of the users 8GB. I currently have about 400GB of my 1TB HDD free although I would not hesitate to take that down to 150GB if I needed to use the space. Every month (when I remember) I move the master images that are 4 months old over to an EHD. I do this in LR in the Folder panel to keep LR in sync. I have about 14TB of 3.5" EHDs attached to my iMac via Firewire800 or USB2. The two NAS in my network are reserved for other uses. One is a TimeCapsule that backs up my rMBP, the other is a Netgear Stora configured as JBOD not RAID1

I keep my master catalog, previews (no Smart Previews) and my active image files on the internal HDD of my iMac. When I need to be mobile, I take a rMBP and an empty catalog. If I have active images in my master catalog that I want to continue working on while away from my iMac, I create a catalog subset using the "Export as Catalog" function and copy this package to the rMBP. and use that exported catalog on the rMBP as my mobile working catalog. When I return, I import this catalog back into the master catalog.

There are reasons that DNG might be desired, it is just that throwing away the proprietary RAW to save space is not one of them. Being able to validate the DNG file with a check sum is probably the best reason. If you still have the original RAW, have you saved any space?
 
The smart previews on the iMac were simply to provide access while I sort out the NAS issue. It just occurred to me that if he wants to take them away on an external drive while traveling we're either going to have to generate them specially each time he leaves or keep the preview files along with the lrcat on the local disk. Since they are already 340Gb, that's not an overhead that the imac HD can support. Is there a way to store them somewhere else without manually copying them, I wonder?

Today I installed the new router and Cat 6 cables and cleared almost 500Gb from the iMac HD. I'm pleased to report that the performance of Lightroom when reviewing files from the NAS improved hugely and looks like it will be acceptable. I'm still struggling with the path issue but did save enough details to post here tomorrow in the hope someone can set me straight. I made a new small test catalog to check the NAS and network speed and LR behaved fine - it's just the catalog I brought over from the PC that is resiting my efforts.

Your workflow keeping newer images locally and then moving them makes a lot of sense but I doubt if he has the time or discipline to get round to it regularly! I agree that on the road a temporary catalog and then importing it is the way to go.

Did we save space by keeping the RAW files? As far as the NAS goes, I'm confident the saving is worthwhile because after the RAID there is "only" 17Tb to play with and we're nearly up to 10 as it stands. Also, the cost of the backup media will be less.

Many thanks for your help. The main performance worry has been reduced and if I can't crack the path issue in the morning I'll post the Final Question which someone here will probably find blindingly obvious!
 
You can create a Symbolic link (Alias) to the Previews folder and place this along side the catalog file. This will permit you to move the Previews to an alternate drive and with the symbolic link will be named the same name as the previews folder ([CatalogName] Previews.lrdata) LR will be happy. I suppose you could also do the same with the Smart Previews folder, but my inclination would be to just eliminate it if LR Mobile is not needed.

For the path issue, I think it will be easily solvable. Screen shots of the Folder panel (All volumes showing, top level (parent) folders showing and sub folders collapsed) and the Finder window that shows the mounted volumes should help. I'm looking for subtle spelling differences in the Volume name or parent folders. At the worst case ,I might ask to look at a copy of the catalog database file using a SQLite manager to view the related table.

As for recent (active) images kept locally, I agree that it is a user discipline issue, And I am just as guilty of being poorly disciplined as anyone. With 400GB free I can afford to be somewhat lax.
 
Hi Sprocket,

The Shared and Network paths are confusing but they are supposed to accomplish the same thing which is to mount external sharepoint as a filesystem in /Volumes on the Mac computer. I have a Synology DS1511+ and have been playing around on my system and have determined a couple of things.

First off there seems to be a problem when trying to access Synology sharepoints using the Network path it tends to leave multiple mounts. This may be what you are describing. You can easily check this out by opening the Terminal app and cd into the /Volumes directory. You will probably find the pictures folder on the NAS listed more than once. It is easy to clear. Make sure you have closed any Finder windows that have that folder open and log out and back in. You should only see a symlink to the "/" for the local system hard drive and any locally attached external drives. Once you have /Volumes straightened out, go the the Server in the Shared tab and remount the sharepoint for your image files.

From here I am not certain how to best clean up your Folders tab in the existing LR catalog. You should be able to reconnect the existing to the new mountpoint. You should see this at the top of the hierarchy next to your system drive the same as you would see an external drive.

I hope that helps.

-louie

PS Cletus. Synology does have an option to confiture a sharepoint as a TimeMachine disk. This works well and I have been using it for a couple of years.
 
PS Cletus. Synology does have an option to confiture a sharepoint as a TimeMachine disk. This works well and I have been using it for a couple of years.
Yes, there are many NAS servers that can be used as a TimeMachine backup destination. Apple's Time Capsule is the one promoted by Apple. However the point that I was making about TimeMachine is that you can not back up a NAS volume using the local OS X TimeMachine app. IOW, if your Pictures are on the NAS, LR can use the image files, but TimeMachine cannot back them up
 
Louie, That does sound like the problem but it took me an age to get to the root in Terminal (not DOS as I remembered it)! Now I've finally go there, the folder on the NAS shows only once along with the Mac HD and an external WD Passport I have plugged into a USB port.

What do you mean by "go the the Server in the Shared tab and remount the sharepoint for your image files."? I assume that you mean in finder but I don't see "server" there.
 
In a Finder window, there is a left panel with a section labeled "Shared". Your network file Server will be listed there.
 
Clicking on the server listed under the Shared label you should get a list of all the sharepoints provided by that server. One of them should be the folder where all the images are stored. Double clicking on that sharepoint will cause it to be mounted on the local system and look just like an external disk but it comes with a special icon that identifies it as a network attached storage. This is the folder you want to use in Lightroom when you redirect the the catalog.

When you use the "Find Missing Folder" navigate to the folder on the NAS by starting at the fileserver listed under the "Shared" heading. This will mount the sharepoint if necessary and reconnect the catalog to the images at the new location on the NAS.

One last step is that you may want to have this sharepoint always mounted for the user. To do this you can add it to the users "Login Items" in the "Users & Groups" in the System preferences. Just click the "+" at the bottom and then navigate to the shared folder and click the Add button. Every time the user logs in it will be there.

-louie
 
Still struggling with LR seeing the path to the NAS. Since the last post the computer hosting the LR Catalogue has been replaced with a new iMac and a new network set up. I was hoping (in vain as it turns out) that this would result in Lightroom seeing the NAS properly or at least so I can find the missing parent folder but no luck.

Screen Shot 2015-02-10 at 1.44.36 PM.jpg

The DiskStation appears under the "Shared" node in Finder and I can access all the files on the NAS quickly and with no problems. In Lightroom the folders appear as in the screenshot - the online NAS is seeing some "missing" folders that I found in the greyed out \\Diskstation.... section via the right click option, the problem is that after doing it with one folder the "shared" node disappears from the "Find Missing Folder" dialog box and workarounds like going via recent places give an Internal Error message "A source is required to change a folder location". If I try to right click on one of the greyed out years to show the parent folder, I get this message:

Lightroom Internal Error:
Database "/Users/michaxxxx 1/Documents/Pictures/Lightroom/Michxxxx NAS Archive.lrcat": column absolutePath is not unique Statement: INSERT INTO AgLibraryRootFolder( id_local, id_global, absolutePath, name, relativePathFromCatalog ) VALUES( ?, ?, ?, ?, ? );

The database is correctly shown in it's new folder so I'm assuming the error is to do with the path of the image files on the NAS. There are a few references to this message in LR2.2 and I tried one of the posted suggestions which was to delete the preferences file but that didn't do the trick.

Any other suggestions?
 
If those two Diskstation "drives" in the folders Panel represent the same physical drive, i.e. all the greyed out "missing" folders on the lower drive (which Lightroom can no longer "see") are all on the same physical drive as the upper one (which Lightroom does "see") then forget about trying to add the parent folder.....just relink the 15 "missing" folders. If there's a problem with that, can we have a screenshot of the finder window that opens after you select "Find Missing Folder", as per the attached"
attachment.php
 

Attachments

  • Find_Missing_Folder.jpg
    Find_Missing_Folder.jpg
    95.5 KB · Views: 371
Last edited by a moderator:
Thanks, Jim. I think the problem is solved but only by way of re-building the catalog. When I wrote the above post today we were halfway through copying a test I did on the old computer to see how miserable it would be to start a new catalog and import everything on the NAS by adding (as the files were already converted an in place). Someone started the copy going before I could check to see if it was behaving itself so I had to wait about three hours to see for myself. The new catalog looks as if it is fine and, although the import runs at about 250 a minute, I'd rather leave it to finish up (the new iMac should be faster, I guess. A new Asus gigabit router has improved the response time too.

Just to follow up on your answer, Finder looked just like your picture except that all trace of the Diskstation vanished after I linked one folder and as I mentioned sub folders didn't link automatically either. I must have wasted a couple of days on this, so I very much hope it's consigned to history now.

I'll report back to let you all know how a catalog this size runs.

PS: Good choice of cameras, btw!
 
Status
Not open for further replies.
Back
Top