• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom Classic questions. All the information you need is in Adobe Lightroom Classic - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. And better still, the eBooks are updated for every release, so it's always up to date.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Slow backup and diagnosis

Status
Not open for further replies.

debi.lightcraft

Member
Premium Classic Member
Joined
Aug 21, 2021
Messages
76
Lightroom Version Number
Lightroom Classic version: 10.4
Operating System
  1. macOS 10.14 Mojave
I decided to begin a new post because my previous post was mixed in with another one. I am new and learning the rules. Thank you for your patience! Below is Jim Wilde's comment I am responding to:

"A couple of hours" is simply not right for a backup of a 1.7GB catalog. Can you tell from the backup process if any of the stages seems to take up most of that time?

Where does the catalog reside? If it's on the same internal drive that you are backing it up to, then that's something you should consider changing (a system drive crash means you've lost the catalog and the catalog backups).

Unplugging external hard drives shouldn't have any effect on the speed of the catalog backup."

Thank you Jim. So I believe my catalog and backup both exist on the imac hard drive. Most of my photos are on external drives which I import and work on in LR. There are some still on the computer in various places, a hold over from many years ago when I was starting out. Also, I store select photos in a special collection area in LR. I believe the first checking of integrity takes the longest. However, the computer also freezes up from time to time during the backup which is unsettling! Actually, the whole computer is running slow. I pulled out my old imac (I purchased this one a few years ago) and the old one boots up in a snap unlike my newer one! Also, When I purchased the new imac, the transfer was done by the company. I wonder if something happened in the transfer? Looking at my old imac I notice that the backup file structure is a little different. Would that jam things up? Should I change the location of the backup to match the old one? However, I might relocate the backup to an external drive. Do you think this might make things run faster? Also, when my files were transferred some photos went missing. In fact, I had many photos in the old iphoto catalog which linked to LR and and none of those were transferred over. There are also other missing photos. Could this be jamming up the works? Should my next step be to try to get the photos all linked up as a first step in figuring out what is going on? What do you think might be a first step? Thank you in advance!
 
Solution
Glad that this has a successful conclusion.

This is an example which demonstrates the amount of orphaned data which can reside in a catalog. While it is possible that some of this may have been left, at some stage, by errors within Lr, especially in the early versions, Lr has been running so long by so many users on so many platforms that Adobe will have optimised the stability of their database processing in so far as what is practically possible.

We tend to forget that Lr can be asked to complete complex database updates on lots of metadata fields across very large number of image records. These processes may be running in the background even as you are using Lr for some other task. It s not surprising that these tasks can be...
I am in the process of finding missing photos in order to clean up the mess that may be contributing to the slow backup. I have two external hard drives with the same name. The first is pretty old so I copied most of the photos to a newer drive which I keep plugged into my computer so I can use it while I am working. The older one pretty much sits on my desk unused except now I am using it because it has photos that turn up missing when not plugged into LR. I tell them apart on the computer by the color of the icon, not by the name which is unfortunately the same. I seem to run into snags when they are both plugged in at the same time. Is this a coincidence or is it a bad idea to plug them in at the same time while tracking missing photos. I'm very uneasy about renaming them. Thank you so much in advance for your thoughts and advice!

Lightroom Classic will have set the catalog path to the old name. Presumably you have images that are also cataloged to a path on the new volume with the same name. It won’t hurt to rename the old volume because missing images in LrC that are on it will still be missing and can be found using the “Find missing folder/find missing file” process.

By renaming the old volume and mounting it alongside the new volume, you will create less confusion in the file system and probably make it easier to relocate the missing images.


Sent from my iPad using Tapatalk
 
Lightroom Classic will have set the catalog path to the old name. Presumably you have images that are also cataloged to a path on the new volume with the same name. It won’t hurt to rename the old volume because missing images in LrC that are on it will still be missing and can be found using the “Find missing folder/find missing file” process.

By renaming the old volume and mounting it alongside the new volume, you will create less confusion in the file system and probably make it easier to relocate the missing images.


Sent from my iPad using Tapatalk
Thank you Clee. On second thought would it be fine to buy a new external hard drive and transfer all the lightroom photos from the external old hard drive to the new one under a new name? That way I don't risk messing up the old hard drive which is sluggish anyway and has other non-lightroom documents on it anyway. ?
 
Last edited:
Ok. I have ordered a 1 terabyte external hard drive (G Drive) due here on Wednesday. I plan to transfer my lightroom photos from a very old clunky external hard drive to the brand new one but under a different name. This will distinguish it from my other 6 month old external hard drive (the two had the same name which was confusing-- see above). Hopefully this will make searching for missing photos simpler because I can have them both plugged in to my imac at the same time without confusing LR and myself! (Thank you for the clarification clee) So the identification of missing photos comes first. Next I will see if the backup is faster. If not I will tackle the suggestion by Clee and Gnits to use the import from another catalog to a new empty catalog which should bring over the good records leaving the detritus behind. However, it was also suggested earlier that I should move my backup so it doesn't exist on my computer next to the catalog which could be disastrous in the event of a hard drive failure. To address this I am thinking of purchasing yet another hard drive, this time a one terabyte Sandisk (which is supposed to be really fast) as a designated place to store my catalog backup. Is this a sensible plan? Thanks!
 
A few thoughts.

Every 18 -24 months I create a new Catalog from my existing Catalog. It gives the database a chance to restart from a fresh start. Who knows how many times a Lightroom process has aborted mid way (power cut, system crash) leaving an accumulation of broken indices within the catalog. This may also reduce the size of the database. I am sometimes prompted to do this by the sense that Lr or Lr backups are running slower than usual. The internal housekeeping within the Lr Catalog has improved over the years, but a fresh catalog, with internal indices rebuilt from scratch is worth considering.

I second the suggestion to check the amount of free space on your drive.

Finally, as you have loads of external drives, consider backing up your catalog to an external drive. The obvious benefit is that you do not use your live and backup catalog if/ when your main drive fails. Use your most recently purchased external drive on the basis that it might be the fastest and also least likely to fail or maybe it is time to get a new fast external ssd drive for backup (faster and more resistant to various, but not all, forms of failure). Check how much space your old catalogs are consuming and decide if it is time to delete your oldest backups ( are you really going to go back to a backup taken months or years ago).

A few thoughts.

Every 18 -24 months I create a new Catalog from my existing Catalog. It gives the database a chance to restart from a fresh start. Who knows how many times a Lightroom process has aborted mid way (power cut, system crash) leaving an accumulation of broken indices within the catalog. This may also reduce the size of the database. I am sometimes prompted to do this by the sense that Lr or Lr backups are running slower than usual. The internal housekeeping within the Lr Catalog has improved over the years, but a fresh catalog, with internal indices rebuilt from scratch is worth considering.

I second the suggestion to check the amount of free space on your drive.

Finally, as you have loads of external drives, consider backing up your catalog to an external drive. The obvious benefit is that you do not use your live and backup catalog if/ when your main drive fails. Use your most recently purchased external drive on the basis that it might be the fastest and also least likely to fail or maybe it is time to get a new fast external ssd drive for backup (faster and more resistant to various, but not all, forms of failure). Check how much space your old catalogs are consuming and decide if it is time to delete your oldest backups ( are you really going to go back to a backup taken months or years ago).
Thank you Gnits--it has taken me so long to reply because I am dealing with multiple issues in LR including tracking down missing photos. I have spent days at this and simply can't find them all (about 40 are left) and originate in the Photos library on my iMac. But I have concerns about my hard drive as the whole system is slow--not just LR. So I really want to backup my catalog now to an external drive for the reasons you suggest. I will follow your suggestion in your earlier post! Also, you suggest creating a new catalog from my existing catalog to clean up the broken indices. Would you be able to explain how to do that? Or point me to the right place? (I have the missing facts and questions ebook).

Also, do you think I should purchase a fast external drive -- I'm thinking of a sandisk with up to 1050 MB/s Read Speed or is my g drive with up to 130 MB/s Data Transfer Speed perfectly adequate? I apologize for such basic questions and for any lack of clarity, but while I have learned a lot about the basic editing tools in LR, I jumped in years ago without understanding the catalog and am paying the price now. Thanks!
 
Last edited:
Busy right now. I will revert later today or tomorrow latest.

Re Drive. Choice of drive is always a compromise between price, performance and capacity.
Decide how much storage you need will help define the capacity question and then check price v performance to see if you can find your specific sweet spot.

Regarding external storage……. Check the max speed of the port you plan to use. If it is usb2 then a fast drive will not deliver immediate speed benefits, but will help future proof possible future computer upgrades. Thunderbolt 3 /4 is the fastest port type at the moment, with speed diff which is orders of magnitude compared to ,say, usb2.
 
Thanks a million for getting back so quickly Gnits! From what I can tell, my 2017 iMac has 4 USB 3 ports (compatible with USB 2) and 2 thunderbolt 3 (USB-C) ports with support for thunderbolt (up to 40 Gbps) and USB 3.1 Gen 2 (up to 10 Gbps). I will check with B&H tomorrow to see what they have available (while considering cost) to make use of this. I'm hoping a fast external hd will help speed up my LR backup and also protect my catalog in the unhappy event of a hd failure! Meanwhile I'll wait for your explanation re how to create a new catalog from my existing one. Thanks so much again!
 
I have created some notes to assist with creating a new version of a catalog, leaving the images in their current location.

I also take this opportunity to get Lr to rebuild previews as required (ie I do NOT copy the existing previews), as there may be an accumulation of orphan preview files from previous Lr crashes, etc.

I am happy for anyone to critique the attached pdf or to add comments or suggestions. Please note there are 2 pages.
 

Attachments

  • MoB_LrExportAsCatalog.pdf
    192.6 KB · Views: 185
Wow! Thank you so very much for taking the time and effort to create this excellent document Gnits! I have read it quickly and will go back after deciding on a hard drive (possibly a new one--have to call b&H) and then read your instructions very carefully and summon up the courage to move forward.
 
You can always create a new test catalog, with say 5 images in a folder on your desktop, use the instructions to create a new test catalog in a location of your choice.
You can then double click in this new test catalog and see that the images have remained in their existing desktop location, even though your test catalog has been rebuilt from scratch in a new location.

Top tip. Give the new catalalog a short, sensible, recognisable filename.
Remember, you can also use the File Open Recent Catalogs to go back to open your main catalog. Write down the name and location of your current main catalog, so you do not confuse it with your test or previous c atalogs.
 
Last edited:
You can always create a new test catalog, with say 5 images in a folder on your desktop, use the instructions to create a new test catalog in a location of your choice.
You can then double click in this new test catalog and see that the images have remained in their existing desktop location, even though your test catalog has been rebuilt from scratch in a new location.

Top tip. Give the new catalalog a short, sensible, recognisable filename.
Remember, you can also use the File Open Recent Catalogs to go back to open your main catalog. Write down the name and location of your current main catalog, so you do not confuse it with your test or previous c atalogs.
Thank you Gnits. I'm really glad you suggested a way to test the process before working with the big catalog and really appreciate the thought and time you have put into this question! Meanwhile, I ordered a new fast(er) external hd and my first step will be to try my catalog backup to the new drive. After seeing what happens with that I will take on the task of importing the old catalog into a new one and understand the steps and details you have so carefully described.
 
You can always create a new test catalog, with say 5 images in a folder on your desktop, use the instructions to create a new test catalog in a location of your choice.
You can then double click in this new test catalog and see that the images have remained in their existing desktop location, even though your test catalog has been rebuilt from scratch in a new location.

Top tip. Give the new catalalog a short, sensible, recognisable filename.
Remember, you can also use the File Open Recent Catalogs to go back to open your main catalog. Write down the name and location of your current main catalog, so you do not confuse it with your test or previous c atalogs.
Hi Gnits--I have tested the process for creating a new catalog out of the old using a test catalog (from about a week ago) according to your excellent pdf. The process was smooth and a new catalog was created. Before I move ahead with my main catalog (over 30,000 photos) I'm wondering about unchecking all the boxes (step 3) and if this might lead to losing some of the edits or ratings etc? Thank you in advance--I don't want to make any mistakes!
 
Check boxes.
1. Export Selected Photos Only (you want all photos selected .to be sure....to be sure) ... Leave unticked.
2 Export Negative Files. You do not want to move 30000 image files. Leave unticked.
3. Build / Include Smart Previews . Again, I want to start from scratch and not have lots of old /orphaned smart previews. Leave unticked.
4. Include Available Previews. The objective is to start with as fresh a setup as possible. Leaving unticked means no orphaned / corrupt previews are migrated.

Sometimes I saw 3 options and sometimes 4. I always go into All Photographs and Cmd/Ctrl A to insure all photos are selected. You should then see the number of images and virtual images will be created in the newly built catalog.
 
Check boxes.
1. Export Selected Photos Only (you want all photos selected .to be sure....to be sure) ... Leave unticked.
2 Export Negative Files. You do not want to move 30000 image files. Leave unticked.
3. Build / Include Smart Previews . Again, I want to start from scratch and not have lots of old /orphaned smart previews. Leave unticked.
4. Include Available Previews. The objective is to start with as fresh a setup as possible. Leaving unticked means no orphaned / corrupt previews are migrated.

Sometimes I saw 3 options and sometimes 4. I always go into All Photographs and Cmd/Ctrl A to insure all photos are selected. You should then see the number of images and virtual images will be created in the newly built catalog.
Thank you for the explanation! Question--Re, "start from scratch" -- where does that mean LR is starting from? What happens to edits and edit history, ratings, etc?
 
Thank you for the explanation! Question--Re, "start from scratch" -- where does that mean LR is starting from? What happens to edits and edit history, ratings, etc?

Gnits might be on the other side of midnight. So, I just clarify what I think he meant. “Start from Scratch”. Refers only to the Preview images in the preview folder(s) By leaving that box unticked, you will cause Lightroom to build a new set of previews in the preview folders. It will use the edits stored in the catalog to be applied to the original images to create the preview images.


Sent from my iPad using Tapatalk
 
Gnits might be on the other side of midnight. So, I just clarify what I think he meant. “Start from Scratch”. Refers only to the Preview images in the preview folder(s) By leaving that box unticked, you will cause Lightroom to build a new set of previews in the preview folders. It will use the edits stored in the catalog to be applied to the original images to create the preview images.


Sent from my iPad using Tapatalk
Thank you clee! Ok, I will assume this also means that the edits, ratings, etc., will all be represented in the new catalog--so I'll give it a go and see what happens!
 
OMG! It has worked!! At least I think it has worked! This is amazing! I was able to import the old master catalog into a new one according to Gnits' crystal clear instructions. The new catalog, on my hard drive opens up with the same number of photos as the old master catalog and as far as I have been able to tell, all the edits, ratings, presets I created are still intact. It moves more rapidly than the old one. I haven't really tested the editing yet, but I have an idea it will be faster too. Then I backed it up to my new external ssd sandisk. This took a total of 6 minutes (as opposed to an hour and a half on my old catalog, what with the computer freezing up etc.). The majority of time backing up the new catalog was in checking the integrity and then optimization, with back up itself maybe seconds. The new master catalog (1.23 GB) is about half the size of the old master catalog ( 2.58GB). Similarly, The new backup zip file is also much smaller (708.8mb) as opposed to the old backup (1.74 GB). Question: Is this reduction in size ok? What causes it? I have other questions such as what to do with the old master catalog, but for now, assuming the reduction in catalog size is fine, I am delighted, as this problem has been with me for over a year. Thanks so much Clee and Gnits for guiding me through missing photos, new hard drives and what to call them, step by step instructions on how to create a new catalog all the way to a backup and where to put it. I feel I am making progress.
 
…The new master catalog (1.23 GB) is about half the size of the old master catalog ( 2.58GB). Similarly, The new backup zip file is also much smaller (708.8mb) as opposed to the old backup (1.74 GB). Question: Is this reduction in size ok? What causes it?

“That is why it has been recommended by me and others to use the import from another catalog (your slow master) to a new empty catalog. this will only bring over the good records and leave the detritus behind. “

You had a lot of detritus in the old catalog. Zip the old catalog file up and save it in your backups folder. Hopefully you will never nee to recover from a backup as far back as this bloated catalog.


Sent from my iPad using Tapatalk
 
Glad that this has a successful conclusion.

This is an example which demonstrates the amount of orphaned data which can reside in a catalog. While it is possible that some of this may have been left, at some stage, by errors within Lr, especially in the early versions, Lr has been running so long by so many users on so many platforms that Adobe will have optimised the stability of their database processing in so far as what is practically possible.

We tend to forget that Lr can be asked to complete complex database updates on lots of metadata fields across very large number of image records. These processes may be running in the background even as you are using Lr for some other task. It s not surprising that these tasks can be interupted by factors outside of Adobe’s control, such as power outage, memory issues, disk i/o problems, forcing Lr to shutdown, etc.

As part of this exercise, I completed the steps documented in the pdf above, so I could create the screen grabs, etc. In my case, there was absolutely no difference in size between my old and rebuilt catalog. It is approx 18 months since I last completed such a rebuild, which indicates that nothing out of the ordinary has happened to my catalog in that period of time. This is reassuring for me.

However, there must be many many Lr users who have catalogs which may be accumulating orphaned data over many many years. At some stage, thus orphaned data will trip up other normal Lr processes. Not surprisingly, backing up a Catalog is probably one activity with a high likelihood of this happening.

I will continue to create a fresh version of my catalog every 18-24 months, as documented above, , just as a routine maintenance task.
 
Solution
“That is why it has been recommended by me and others to use the import from another catalog (your slow master) to a new empty catalog. this will only bring over the good records and leave the detritus behind. “

You had a lot of detritus in the old catalog. Zip the old catalog file up and save it in your backups folder. Hopefully you will never nee to recover from a backup as far back as this bloated catalog.


Sent from my iPad using Tapatal

Glad that this has a successful conclusion.

This is an example which demonstrates the amount of orphaned data which can reside in a catalog. While it is possible that some of this may have been left, at some stage, by errors within Lr, especially in the early versions, Lr has been running so long by so many users on so many platforms that Adobe will have optimised the stability of their database processing in so far as what is practically possible.

We tend to forget that Lr can be asked to complete complex database updates on lots of metadata fields across very large number of image records. These processes may be running in the background even as you are using Lr for some other task. It s not surprising that these tasks can be interupted by factors outside of Adobe’s control, such as power outage, memory issues, disk i/o problems, forcing Lr to shutdown, etc.

As part of this exercise, I completed the steps documented in the pdf above, so I could create the screen grabs, etc. In my case, there was absolutely no difference in size between my old and rebuilt catalog. It is approx 18 months since I last completed such a rebuild, which indicates that nothing out of the ordinary has happened to my catalog in that period of time. This is reassuring for me.

However, there must be many many Lr users who have catalogs which may be accumulating orphaned data over many many years. At some stage, thus orphaned data will trip up other normal Lr processes. Not surprisingly, backing up a Catalog is probably one activity with a high likelihood of this happening.

I will continue to create a fresh version of my catalog every 18-24 months, as documented above, , just as a routine maintenance task.
I thank you Gnits for the time and effort you have put into helping diagnose and resolve the slow backup of my LR catalog and also for your patience and explanations as to the origins of these frustrating issues. Once again, you have anticipated my questions. I have long wondered about the cause of the the slow backup to my catalog--(eg., was it not enough ram, something missing in the data transfer to a new computer, too many missing photos?). My understanding now is that LR's ability to process data can be affected by many factors and "orphaned" data may be compounded over the years (probably originating way back in my LR3 catalog). Finally here is a way to import the master into a new catalog that leaves the broken data behind and captures the solid working data. I've been dealing with this for a long time (and probably before I was even aware of it) and have had no luck in resolving 'til now! I thought about starting a new catalog, but am glad I held off as this appears to have solved the problem. I'm eager to test it all out with some new photos.

I will follow your lead in repeating the process in 18 months or sooner if need be. But do you have any suggestions as to how to prevent these errors in the first place? I hope to get to the point where I can complete the steps in the pdf and say as you do there is no difference in the size of the two catalogs.

Thank you !!!

Debi
 
“That is why it has been recommended by me and others to use the import from another catalog (your slow master) to a new empty catalog. this will only bring over the good records and leave the detritus behind. “

You had a lot of detritus in the old catalog. Zip the old catalog file up and save it in your backups folder. Hopefully you will never nee to recover from a backup as far back as this bloated catalog.


Sent from my iPad using Tapatalk
Thank you! I will put it aside and hope I never have to recover from it (the old catalog)!
 
I thank you Gnits for the time and effort you have put into helping diagnose and resolve the slow backup of my LR catalog and also for your patience and explanations as to the origins of these frustrating issues. Once again, you have anticipated my questions. I have long wondered about the cause of the the slow backup to my catalog--(eg., was it not enough ram, something missing in the data transfer to a new computer, too many missing photos?). My understanding now is that LR's ability to process data can be affected by many factors and "orphaned" data may be compounded over the years (probably originating way back in my LR3 catalog). Finally here is a way to import the master into a new catalog that leaves the broken data behind and captures the solid working data. I've been dealing with this for a long time (and probably before I was even aware of it) and have had no luck in resolving 'til now! I thought about starting a new catalog, but am glad I held off as this appears to have solved the problem. I'm eager to test it all out with some new photos.

I will follow your lead in repeating the process in 18 months or sooner if need be. But do you have any suggestions as to how to prevent these errors in the first place? I hope to get to the point where I can complete the steps in the pdf and say as you do there is no difference in the size of the two catalogs.

Thank you !!!

Debi
“I thought about starting a new catalog,”.

Imeaning a separate catalog with brand new photos
 
But do you have any suggestions as to how to prevent these errors in the first place? I
Some root causes can be completely outside your control, such as a power failure while Lr is updating the catalog, a bug in Adobe or 3rd party software, a disk drive which may be failing, faulty memory and lots of other events outside of your control.

That is why regular backups are so important.

You can positively work to minimise such events by making sure your Pc/Mac has a quality power supply, is operating within the recommended temperatures, components are not drawing more power than the power supply can deliver and other normal computer hygiene actions. You cannot eliminate all such factors.

Overclocking Pcs may damage or cause internal component failure (cpu, memory, motherboard). Installing graphics cards with more power demands than the Pc motherboard can supply, incorrectly configured memory, etc., can be elements which are self inflicted.
 
Some root causes can be completely outside your control, such as a power failure while Lr is updating the catalog, a bug in Adobe or 3rd party software, a disk drive which may be failing, faulty memory and lots of other events outside of your control.

That is why regular backups are so important.

You can positively work to minimise such events by making sure your Pc/Mac has a quality power supply, is operating within the recommended temperatures, components are not drawing more power than the power supply can deliver and other normal computer hygiene actions. You cannot eliminate all such factors.

Overclocking Pcs may damage or cause internal component failure (cpu, memory, motherboard). Installing graphics cards with more power demands than the Pc motherboard can supply, incorrectly configured memory, etc., can be elements which are self inflicted.
Thank you for your tips — i will try to stay on top of it. It’s a wonder anything ever works at all.
 
I have created some notes to assist with creating a new version of a catalog, leaving the images in their current location.

I also take this opportunity to get Lr to rebuild previews as required (ie I do NOT copy the existing previews), as there may be an accumulation of orphan preview files from previous Lr crashes, etc.

I am happy for anyone to critique the attached pdf or to add comments or suggestions. Please note there are 2 pages.
Hi Gnits, just wanted to say many thanks for creating this pdf. Today I was getting exasperated with the slowness of my pc when closing Lightroom and I realised that it was primarily due to my never having updated my catalog in far too many years. This was just what was needed and I've taken the liberty of downloading and saving this for next time.
Paul
 
Status
Not open for further replies.
Back
Top