Slow LR 6.14 Catalog Backup Problem

Status
Not open for further replies.
M

marinephoto

Active Member
Joined
Dec 4, 2014
Messages
159
Lightroom Experience
Intermediate
Lightroom Version Number
6.14
Operating System
  1. macOS 10.13 High Sierra
Awhile ago I began to experience painfully slow Catalog backups so I posted here in search of advice. By painfully slow I mean that when exiting Lightroom after clicking the checkboxes for Optimize Catalog and Backup Catalog it would take several hours for the process to complete. My Library contains around 75,000 images and the Catalog at the time was 13.7 GB in size. People responding said that was a very large catalog size for 75,000 images so I asked if an extensive Develop History could account for the large size and slow backup speed, Folks said it could so eventually I made a duplicate catalog for experimenting with and deleted the Develop History from all 75,000 photos, This reduced the catalog size from 13.7 GB to 2.34 GB. I decided that I wanted to retain the Develop History for my 4 ✩ or better images, so I did a sort for all images less than 4 ✩ and deleted the Develop History from all but 2,500 of the 4 ✩ or better images. This resulted in a catalog size of 5.79 GB. Hoping that this would result in a much faster catalog backup time I Optimized and Backed Up the modified, now much smaller, 5.79 GB catalog, but to my dismay it still takes more than an hour to complete. I don't know precisely how long because I've never had the time to sit there and wait. Neither do I know just how long it should take to complete a catalog backup.

Some had mentioned that this slow backup could indicate a problem with catalog corruption but I have never gotten any error code, nor have I seen any other problem that might indicate catalog corruption. It did occur to me today, however that the ESET Cyber Security software I am running might be playing a role in this??? Does anyone know if this might be the case? If not, any idea what the problem might be causing such slow catalog backups? I am using a 2017 21.5" Retina Screen iMac 3 Ghz Intel Core i5 Processor with 16 GB of memory, running High Sierra 10.13.6 My equipped with a 1 TB Fusion Drive with 375 GB available and I am saving the catalog to that internal drive

Thanks!
 
Sort by date Sort by votes
Hi

I have heard of Anti-virus causing issues in the past. Can you exclude files or folders from it? As the catalog is entirely local to you, one may question the need for it to be monitored.
 
Upvote 0 Downvote
Paul McFarlane said:
Hi

I have heard of Anti-virus causing issues in the past. Can you exclude files or folders from it? As the catalog is entirely local to you, one may question the need for it to be monitored.
Click to expand...
Paul last night at 11:30 pm I deactivated both my ESET Cyber Security App and Time machine, exited Lightroom clicking Check Catalog Integrity and Optimize Catalog and clicked OK. Forty minutes later the progress bar had barely moved an eighth of an inch. When I checked this morning 8.5 hrs later the screen said "Compressing Catalog." A minute or two later it said "Copying Catalog." And then when the progress bar was nearly at the end it said "Optimizing Catalog." The inner workings of Lightroom shall forever remain a mystery to me so although I am generally good at diagnosing problems this one eludes me. But clearly something is wrong here.

As I said earlier I have never gotten any catalog corruption errors, or anything indicating catalog trouble. I do however recall that a year or so ago I ran into a problem that may be playing a role in this? I began shooting with two different cameras and started getting duplicate file names. To resolve this I used Lightroom's sorting power to sort shots by camera name/model and I added that prefix to the image names: As in ShawD700_YRMODY_01.dng, ShawD500_, etc. After doing this I did see some odd images names show up from time to time with screwy names like
“ShawD71_190514_710019.DNG”. I was never able to figure out what caused this and wonder now if this may be causing catalog problems now?

I guess the question is where to I go from here? The future of 11 years work and 75,000 images depends upon me making the right decesion. I know in the past I've seen notes on this forum that Victoria knows someone at Adobe who is expert at dealing with catalog problems and I am wondering if she might be willing to forward my catalog to him to see if he can figure out what the problem is and how to fix it?
 
Upvote 0 Downvote
I don't see a reason to send this catalog file to Victoria just yet. Even a 5.79GB catalog seems huge for an inventory of ~75K images.
I would like you to make a different kind of test. Select ALL of the images in the 5.79GB master catalog and use the function " Export as a Catalog..." to create a new catalog This will make a copy of the metadata about all of your ~75K images in a new catalog. it will leave behind any detritus and orphaned records that might be causing problems. Once you never this Exported catalog , report its size (Hopefully it will be under 1GB) and open it in LR 6.14 like you would your master catalog . The exit Lightroom normally doing the usual catalog backup/integrity check. You can then report any observed difference in backup behavior.
 
Upvote 0 Downvote
clee01l said:
I don't see a reason to send this catalog file to Victoria just yet. Even a 5.79GB catalog seems huge for an inventory of ~75K images.
I would like you to make a different kind of test. Select ALL of the images in the 5.79GB master catalog and use the function " Export as a Catalog..." to create a new catalog This will make a copy of the metadata about all of your ~75K images in a new catalog. it will leave behind any detritus and orphaned records that might be causing problems. Once you never this Exported catalog , report its size (Hopefully it will be under 1GB) and open it in LR 6.14 like you would your master catalog . The exit Lightroom normally doing the usual catalog backup/integrity check. You can then report any observed difference in backup behavior.
Click to expand...
Cletus

Thanks! Sounds like good news to me and much appreciated. I will follow your instructions soon as I get a moment and report back what I discover. Hopefully the news wiil be good!
 
Upvote 0 Downvote
Why do you run Eset Cyber Security at all? Why not dump it completely as it does not do anything that the mac does not do on itself !. Setting it off does not help, only uninstalling helps, and then a restart. Another remark I want to make is this, did you set Filevault on your mac? If so do you really know why?
 
Upvote 0 Downvote
LexS said:
Why do you run Eset Cyber Security at all? Why not dump it completely as it does not do anything that the mac does not do on itself !. Setting it off does not help, only uninstalling helps, and then a restart. Another remark I want to make is this, did you set Filevault on your mac? If so do you really know why?
Click to expand...
To answer your question ESET was loaded on my computer by the repair shop I paid to transfer all my stuff over from my old iMac. They gavve it a strong recomendation at the time. I had my doubts but gave them the OK to load it. When the 1 year subscription ran out I did not renew. I'm still not clear on the vulnerability of the Mac OS to threats. Some sites say you need protection, others say you do not. I recently happened upon the very impressive MacMost site on YouTube and like you Gary says that the Mac OS already features the ability to get rid of threats and that nothing else is required. Soon as get a moment I will get rid of it. As to Filefault...I watch James Bond movies, but I have no desire to star in one of them.
 
Upvote 0 Downvote
Look in the Apple Forum. Mac OS & System Software - Apple Community. Choose your OS, although it is the same in all OS: numerous discussions on Antivirus and other "saviour apps": always recommended NOT to install these.
BTW in Windows OS10 the built in safety is better than any outside app too, Look in the windows security from the menu.
 
Upvote 0 Downvote
LexS said:
Look in the Apple Forum. Mac OS & System Software - Apple Community. Choose your OS, although it is the same in all OS: numerous discussions on Antivirus and other "saviour apps": always recommended NOT to install these.
BTW in Windows OS10 the built in safety is better than any outside app too, Look in the windows security from the menu.
Click to expand...
I checked out your link and it does indeed appear that ESET is not the best thing since sliced bread. I cheched out ESET's instructions for uninstalling the app and will do so. My question is just how important is it to track down and delete the ESET files left behind after following their uninstall directions? Gary at MacMost says it is simply not worth the trouble.
 
Upvote 0 Downvote
clee01l said:
I don't see a reason to send this catalog file to Victoria just yet. Even a 5.79GB catalog seems huge for an inventory of ~75K images.
I would like you to make a different kind of test. Select ALL of the images in the 5.79GB master catalog and use the function " Export as a Catalog..." to create a new catalog This will make a copy of the metadata about all of your ~75K images in a new catalog. it will leave behind any detritus and orphaned records that might be causing problems. Once you never this Exported catalog , report its size (Hopefully it will be under 1GB) and open it in LR 6.14 like you would your master catalog . The exit Lightroom normally doing the usual catalog backup/integrity check. You can then report any observed difference in backup behavior.
Click to expand...
Cleetus,

I finally have time to try out your catalog export idea to see if it fixes my problem. When I clicked on Export As Catalog there are 3 check boxes. I don't think I need to check "Export Negative Files" as they already exist on my hard drive but I'm unsure whether I should check off "Include Available Previews" or not.

My current preview file is 15.73 GB in size. Should I include it in the Catalog options?

Thanks!
 

Attachments

  • Screen Shot 2020-02-24 at 3.13.44 PM.png
    Screen Shot 2020-02-24 at 3.13.44 PM.png
    429.4 KB · Views: 216
Upvote 0 Downvote
You don’t need to export the previews or negative files. The new Lightroom catalog file will map to the existing “digital negatives” the previews will automatically regenerate as needed.


Sent from my iPad using Tapatalk
 
Upvote 0 Downvote
clee01l said:
You don’t need to export the previews or negative files. The new Lightroom catalog file will map to the existing “digital negatives” the previews will automatically regenerate as needed.


Sent from my iPad using Tapatalk
Click to expand...

Cletus,

I followed your instructions and exported my current catalog to the Desktop. The operation took a few minutes to complete but sadly the exported catalog remained the same size 6.11 GB. I'm not sure where that leaves us. Any thoughts?

Thanks!
 
Upvote 0 Downvote
marinephoto said:
Cletus,

I followed your instructions and exported my current catalog to the Desktop. The operation took a few minutes to complete but sadly the exported catalog remained the same size 6.11 GB. I'm not sure where that leaves us. Any thoughts?

Thanks!
Click to expand...

Is it still slow to backup? If it is not any faster that the old one, zip up the OLD one and send it to Victoria. There iOS a wetransfer email address that she uses. I don’t know it off the top of my head and I’ll have to get off TapaTalk to look it up


Sent from my iPad using Tapatalk
 
Upvote 0 Downvote
@marinephoto:
what I forgot to mention earlier: whenever you deactivate/uninstall an Antivirus or any other "protector" software, you should restart after the uninstall, before going further !! as long as you do not have restarted, there are still some parts that interfere in your system.
 
Upvote 0 Downvote
clee01l said:
Is it still slow to backup? If it is not any faster that the old one, zip up the OLD one and send it to Victoria. There iOS a wetransfer email address that she uses. I don’t know it off the top of my head and I’ll have to get off TapaTalk to look it up


Sent from my iPad using Tapatalk
Click to expand...

Cletus,

I started a Catalog backup 1 hour ago and the "Checking catalog integrity" progress bar has barely moved so the Export as Catalog trick hasn't fixed the problem. You said you'd recommend I zip the original Catalog file and send it to Victoria. Did you ever find the email address for me to use?

Thanks!
 
Upvote 0 Downvote
Just wanted to let folks know that Victoria and her contact at Adobe were able to get my catalog fixed up. It's still 6 GB in size (probably due to develop history size) but it now backs up in under 10 minutez whereas it took upwards of an hour before. Great big Thank You! to Victoria and her friend at Adobe and thanks to all here for your help as well.
 
Upvote 0 Downvote
Whoops! It turns out I celebrated victory a bit too soon...

When I got the catalog back from Victoria I made a few small edits to one image and then closed Lightroom with Check Integrity and Optimize Catalog checked and the backup took a little less than 10 minutes to complete. So it appeared that all was well.

Yesterday after uploading 400 new images and editing a dozen of them I closed Lightroom with Check integrity and Optimize Catalog selected and the backup took more than an hour! I have no idea why simply adding 400 images to the catalog would have increased catalog backup time so much.

At this point it occurred to me that I couldn't recall how long backups were taking a year ago when I was still using Lightroom 3 on my previous iMac. So I fired up my old computer, launched LR 3, uploaded several hundred images and then closed Lightroom with Check Integrity and Optimize catalog checked. The 11.74 GB catalog took 1 hr and 35 minutes to back up.
So these slow catalog backups in LR 6 were apparently inherited from my previous LR 3 catalog which I imported into LR 6.

What role the Develop History might be playing in all this has been a question needing a definitive answer for some time. So I decided today to make another backup copy of my catalog and then do a total deletion of the Develop History on all 80,000+ images. Previous to this I had deleted the Develop History on all but 2,500 4 star images and the catalog was reduced to slightly less than 6 GB from more 13 GB.

Today after deleting the Develop History on all 82,000 images the catalog shrunk to 1.29 GB!

To test backup time of the much reduced catalog I imported an sd card of 400 images, edited a dozen or so of them then closed Lightroom with Check integrity and Optimize catalog selected. Backup time= 1' 11"!

Apparently the villain in all of this was a very large Develop History after all! Hard to believe that the Develop History of only 2,500 images was nearly 4 GB in size but that was apparently the case here! Although my total collection of 80,000 images doesn't seem that large the majority of those images represent a final selection fromhundreds of thousands of images I've shot over the past decade and therefore many of them have undergone post processing and had fairly extensive develop histories. So from now on I am going to periodically trim back my develop histories.
 
Upvote 0 Downvote
Status
Not open for further replies.
Back
Top