Sync Sync issues causing repeated LrC crash during Rebuild Sync Data

[mathewlynn]

Lightroom Version Number

13.1

Operating System

1. macOS 14 Sonoma
2. iOS

Dear Forum,
I've been using Lightroom Classic since 2016 but my knowledge is still pretty basic, it always just worked! Over the past 6 months I've had some major syncing issues so I've had to learn the hard way how to fix them, mostly with threads I’ve found here, thank you! The previous issue last October was similar to this one, and eventually I went back to a previous catalog backup which seemed to solve it, and I just re-did my work, no problem. This time that doesn't seem to work and I'm not really sure what to do next!

The current issue started when I synced some freshly made albums (to my iPad/iPhone), but several of those albums remained empty. I left it for a few days but it didn't resolve. I decided on Rebuild Sync Data, which filled the albums on my devices, but now I'm stuck in a continuous loop of rebuilding sync data and system crashing, running out of application memory. Working from previous catalogs doesn't seem to resolve the issue this time. I've tested the integrity of my current catalog many times, the first time I did this it did say there was a corruption, and I think Lightroom made a Corrupt Catalogs folder in my Lightroom files location, all my Lightroom files live in the original location on my desktop, Pictures/Lightroom. There's probably some tidying up I need to do there.

The sync activity panel during Rebuild Sync Data seems focussed 100% on metadata for all images, but often it dosen’t show anything. Currently it’s missing one image in Uploads, compared to All Synced Photographs. My external hard drive is connected every time I try the rebuild.

I use Lightroom ecosystem syncing in a very particular way, everything only goes in one direction, from LrC on desktop out to my devices in my art studio for sketching and painting.
There are two locations for images:
1. An external hard drive with photos from my cameras, these are images I take for my studio work.
2. A native folder 'Collected Images' in Pictures, containing a massive folder structure of found research images, this structure is synchronised into LrC, and set up as albums in the same folder structure synced to my devices. There are a very large number of albums because often there's very specific areas of research and focus, 135 subject areas with sometimes 100 or more albums in each, I haven't counted them all!, but I don't think I'm near the limit?, even considering when I add the album structure replicated from my external hard drive work.

There was a time that I was adding iPad screenshots to specific Lr Mobile albums on the fly without really understanding what I was doing. I've stopped that, but those images are still floating around and obviously in Cloud Storage. I can get rid of those if necessary.

There are currently 130230 synced images (130229 in Uploads, preferences panel), which I can easily cull once I resolve this, but the research images I like to have with me all the time wherever I am, so I’m always going to want to have a lot of synced images! Images from my external hard drive are more client and project based and can more easily be removed from syncing when a job is completed.

I'm trying to avoid Delete All Synced Data, because re-syncing and then moving that huge number of albums into my device folder structure will be incredibly tedious and time consuming, moving one album takes up to a minute, but I realise that might be the next thing! Victoria mentioned somewhere that sometimes this is the best way to go, from a clean start and rebuild.

Best and thanks,
Mathew

LrC 13.1
Sonoma 14.3.1
Lightroom iPad and iPhone

 

[clee01l]

Are you checking the preferences section for sync errors?
Periodically I find that sync hangs for no apparent reason. If it is just one or two images, remove them temporarily from syncing collections and All sync photos to clear the sync errors. Also never sync videos. They are always a problem.

I’ve never needed to resort to rebuilding sync. But what I do more often than I would like is log out of the Adobe Application app, exit that app and LrC and reboot the computer


Sent from my iPhone using Tapatalk

 

[mathewlynn]

Are you checking the preferences section for sync errors?
Periodically I find that sync hangs for no apparent reason. If it is just one or two images, remove them temporarily from syncing collections and All sync photos to clear the sync errors. Also never sync videos. They are always a problem.

I’ve never needed to resort to rebuilding sync. But what I do more often than I would like is log out of the Adobe Application app, exit that app and LrC and reboot the computer


Sent from my iPhone using Tapatalk

Ahh, interesting thank you Cletus!, I'll try the logging out option and get back!

When I can see it, the preferences panel just has metadata in that Sync Details/Errors column, there was a very large number of images, both from my external hard drive and my native research folder mentioned above. At the moment I can't get it to show anything as it just heads pretty quickly to crashing as soon as I launch. This constant crashing is pretty concerning, and there's no normalcy with this constant attempt to sync everything with big numbers going up and down.

I never sync videos, though I have noticed over the years there have been problems with iPhone Live photos. I avoid them like the plague now, but there's still a few of those floating around in the source locations.

 

[mathewlynn]

Ahh, interesting thank you Cletus!, I'll try the logging out option and get back!

When I can see it, the preferences panel just has metadata in that Sync Details/Errors column, there was a very large number of images, both from my external hard drive and my native research folder mentioned above. At the moment I can't get it to show anything as it just heads pretty quickly to crashing as soon as I launch. This constant crashing is pretty concerning, and there's no normalcy with this constant attempt to sync everything with big numbers going up and down.

I never sync videos, though I have noticed over the years there have been problems with iPhone Live photos. I avoid them like the plague now, but there's still a few of those floating around in the source locations.

Sorry for the delay Cletus, I'm sharing a new workspace and it's not always available!

I tried the logging out method, and for about a minute or so I thought it had worked!, with the sync green tick smiling at me, then it went back into its routine. I took a bunch of screenshots of various things hoping it might help:

This morning that was just locked on that figure, no spinning wheel.

And this morning I synchronised some new images to test syncing on devices while all this is going on, by making a new synced album of three images, that's why we're at 130233. I didn't think anything would show up on the iPad/iPhone because it's doing this massive resolve, and it didn't.

Uploads here is not at the right number, and no sync activity was showing up.

My catalog location looks like a real mess, and I'm sure I should be cleaning that up! I assume I'd be fine to remove anything that hasn't got today's date on it, except for the Mobile Downloads.lrdata file - I found out the hard way with the previous problem that I need to keep that one.




Since all these above I've updated to 13.2 just now, and decided to try another Rebuild Sync Data. It seems to be working through it all Ok, but I've been here before! All the previous attempts started to break down with the numbers going up and the system crashing. Just checking again, it's worked through all the Uploads in the Sync Pref panel, but still only at 320,229, I'm not sure if that's meaningful?

The numbers in the little sync panel are now bouncing around up and down (which I believe is normal) in the vicinity of 25,000.

Best and thanks,
Mathew

 

[clee01l]

A few things come to light with your screen shots. I missed before that you are trying to sync ALL of your 192,000 images. Lightroom Classic is trying to make Smarts preview for sending to the Adobe Cloud. You are probably running out of FreeSpace that LrC needs in Working Storage for temporary files Make sure that you have at least 100GB of freespace before you start to sync AND sync small batches ( perhaps less than 10K images) Create a Collection that syncs an add to the collection in small batches syncing one batch at a time until the all of the batches have been synced. You will never get 192.000 file to sync in one continuous sync. Rebuilding Sync Data ia just a waste of tiny if all you are going to di is try to sen 192.000 Smart Preview DNGs to the Adobe Cloud.

Your Master Catalog is "Lightroom Catalog-2-2-v13.lrcat". All of the files and folders with "Lightroom Catalog-2-2-v13..." in the name need to be kept. Older catalogs and Temporary files I would recommend moving to a safe location until you can determine whether they can be deleted.

The Backuos folder should be located on a different volume where it will be safe when this volume fails and you lose your master catalog .

The Default folder for Downloaded Sync Files is "Mobile Downloads.lrdata" because you have not specified a folder scheme to collect the image files synced from the Adobe Cloud. You should change this in preferences.

 

[mathewlynn]

Thanks Cletus,

I've just come back in and have had to Force Quit Lightroom in the Rebuild Sync Data process - just before it froze, it was hovering around 75000 images but nothing listed in the Sync Pref panel. The Uploads in this panel have caught up to the full 130233, the same number as All Synced Photographs in the Catalog pane.

Just going through your points:

- The 192336 is 'All Photographs', all the ones I've ever loaded in right?, the ones that are synced or trying to sync is the 130233 I thought?
- The original problem was that a pretty small sync session didn't complete, so I decided to Rebuild Sync Data. I was under the impression that this is what happens when you do that?, so there's no control over the sync numbers, the process just has to resolve?
- Free Space and Working Storage I don't know anything about, and not sure where to find them, I'll have a poke around.
- Because I'm in stuck this self-defeating loop of Rebuild Sync Data and crashing, I'm not sure how you would make smaller batches of syncing in this context? I'm not trying to sync them all at once on purpose is also what I mean.
- I think all the Smart Previews are already in the Cloud, because everything is on my iPad and iPhone? It's saying 130229 on my iPhone and iPad in All Photos.

Thanks for catalog files help!, I'll clean all that up.

And thanks around the backups - I left the Backups folder there because I back the entire desktop up every 1-2 weeks, and I was assuming that would be Ok if I had to retrieve anything? I also do a double each time, and one backup lives offsite. And as an extra precaution I periodically copy all the most recent backups onto my Pictures external RAID drive (images location 1, mentioned in my first post), that gets double backed up too because of the photos, so I was thinking I'd have enough options?

And yes, I wasn't really sure about that selection for the Mobile Data in the Preferences panel, I think I may have turned that off way back in 2019 with that issue with the updated Mobile apps, when they started to upload everything from their respective camera rolls without asking! That's solved now I think and I can reestablish that link for the Mobile Downloads.lrdata thank you.

 

[mathewlynn]

Thanks Cletus,

I've just come back in and have had to Force Quit Lightroom in the Rebuild Sync Data process - just before it froze, it was hovering around 75000 images but nothing listed in the Sync Pref panel. The Uploads in this panel have caught up to the full 130233, the same number as All Synced Photographs in the Catalog pane.

Just going through your points:

- The 192336 is 'All Photographs', all the ones I've ever loaded in right?, the ones that are synced or trying to sync is the 130233 I thought?
- The original problem was that a pretty small sync session didn't complete, so I decided to Rebuild Sync Data. I was under the impression that this is what happens when you do that?, so there's no control over the sync numbers, the process just has to resolve?
- Free Space and Working Storage I don't know anything about, and not sure where to find them, I'll have a poke around.
- Because I'm in stuck this self-defeating loop of Rebuild Sync Data and crashing, I'm not sure how you would make smaller batches of syncing in this context? I'm not trying to sync them all at once on purpose is also what I mean.
- I think all the Smart Previews are already in the Cloud, because everything is on my iPad and iPhone? It's saying 130229 on my iPhone and iPad in All Photos.

Thanks for catalog files help!, I'll clean all that up.

And thanks around the backups - I left the Backups folder there because I back the entire desktop up every 1-2 weeks, and I was assuming that would be Ok if I had to retrieve anything? I also do a double each time, and one backup lives offsite. And as an extra precaution I periodically copy all the most recent backups onto my Pictures external RAID drive (images location 1, mentioned in my first post), that gets double backed up too because of the photos, so I was thinking I'd have enough options?

And yes, I wasn't really sure about that selection for the Mobile Data in the Preferences panel, I think I may have turned that off way back in 2019 with that issue with the updated Mobile apps, when they started to upload everything from their respective camera rolls without asking! That's solved now I think and I can reestablish that link for the Mobile Downloads.lrdata thank you.

I'm not sure, is this it?

 

[clee01l]

That 1.06 TB is your free space. It may not be empty if there are temporary files left behind from active processes. When you re boot, you should get all of that 1.06TB available for working storage for Lightroom and other running apps.

You are wanting to sync all 192K image to the Adobe cloud right?

Even though you have a lot of free space, I’d still recommend syncing in Small Batches. The number in All Sync’ed Photographs special collection is the number of images from your master catalog that are in the Adobe Cloud.
I recommend doing the following:
1. Create a collection and give it a name. Set it to sync with the Adobe Cloud.
2. Assign a few hundred image to this collection and watch the sync progress. When all have sync’s completely, add 1 -2 thousand more images to this collection and repeat the process.
3. Each time the sync process completes add more images until the new added images is around 25,000. If you can sync 25,000 images at a time, then keep adding batches of 25,000 images to that collection until you have added all of the master catalog to the Adobe Cloud. Be careful that you do not try to sync videos as that will almost always cause a problem.


Sent from my iPad using Tapatalk

 

[karl.nyhus]

I left the Backups folder there because I back the entire desktop up every 1-2 weeks, and I was assuming that would be Ok if I had to retrieve anything?

Every 1-2 weeks? That would not be nearly good enough in my opinion. (1) I have Apple Time Machine set to update its backup disk every hour. (2) Once a day I update an external backup disk via Carbon Copy Cloner software. I have a set of these external disks that I rotate weekly and store in a fire safe when not in use. (3) And each night, a scheduled backup is performed to an offsite service. I use Arq Premium but many people rely on BackBlaze.

 

[mathewlynn]

That 1.06 TB is your free space. It may not be empty if there are temporary files left behind from active processes. When you re boot, you should get all of that 1.06TB available for working storage for Lightroom and other running apps.

You are wanting to sync all 192K image to the Adobe cloud right?

Even though you have a lot of free space, I’d still recommend syncing in Small Batches. The number in All Sync’ed Photographs special collection is the number of images from your master catalog that are in the Adobe Cloud.
I recommend doing the following:
1. Create a collection and give it a name. Set it to sync with the Adobe Cloud.
2. Assign a few hundred image to this collection and watch the sync progress. When all have sync’s completely, add 1 -2 thousand more images to this collection and repeat the process.
3. Each time the sync process completes add more images until the new added images is around 25,000. If you can sync 25,000 images at a time, then keep adding batches of 25,000 images to that collection until you have added all of the master catalog to the Adobe Cloud. Be careful that you do not try to sync videos as that will almost always cause a problem.


Sent from my iPad using Tapatalk

Thanks again Cletus, and for the clarity on storage.

No, the 192K images are the total number I've brought into Lightroom over the years, I don't want all of them synced. What I currently have synced is the 130233 which are my supposedly currently useful images, but some of them need to be un-synced because they're old projects or not necessary right now for me to have in the studio for the art works I have going and in development. The research portion of my synced images (mentioned first post) contains a very large number of Collections, and I want that with me all the time.

- The thing I'm finding a bit confusing in relation to your instructions is that images are only trying to re-sync right now in the context of me initiating a Rebuild Sync Data, with crashing on every attempt, so the process is never resolved. I'm not really sure what actually happens in this process, I sort of assumed it was a refreshing of all the linkages to each image already in Adobe Cloud.
- I did try adding to the current situation (mentioned above) with a sync of a new bunch of 3 images in a new research album/Collection (Artist - George Bellows), but nothing has shown up on devices, and therefore I wouldn't expect them or album to be in Lightroom Web, and furthermore I'd sort of expect them to be at the back of the queue with whatever craziness is going on!
- So every time I launch LrC again I'm assuming it's still in the middle of this rebuild process? - and whatever I do, the system is only going to crash, and that includes following your recommendations.
- I'm not sure how to stop this cycle of chaos so I could even try your suggestions! Or are you saying just do it anyway by adding my already synced images in Adobe Cloud (because they're all up there already and on my devices) into a separate Collection? Or by implication are you saying un-sync everything then take those steps?
Anyway, I think you're referring to bringing the rest (all) of my images into being synced, which is not what I'm trying to do. It's solving this chaos, because I can't function in my studio without sync working at all times.

How do I break out of this cycle to some sort of normalcy so that I could even follow your instructions?, that's what I'm wondering, sorry if I'm missing something really obvious!

The one thing I'm trying to avoid is having to un-sync everything, or doing a Delete All Synced Data - as I mentioned moving freshly re-synced albums (there's lots of them) into the extensive folder system I set up on the devices would be very time consuming, takes about a minute to move just one album for me.

Best and thanks

 

[mathewlynn]

Every 1-2 weeks? That would not be nearly good enough in my opinion. (1) I have Apple Time Machine set to update its backup disk every hour. (2) Once a day I update an external backup disk via Carbon Copy Cloner software. I have a set of these external disks that I rotate weekly and store in a fire safe when not in use. (3) And each night, a scheduled backup is performed to an offsite service. I use Arq Premium but many people rely on BackBlaze.

Thanks Karl, definitely food for thought, particularly now that my offsite option needs another solution, I may need a fire safe and I'll look at all those other things.

For me the balance is still about right, because I'm not taking photos (or collecting them) as a photographer, I'm making art works with them. After I do a fresh bunch of work in Develop, I'll generally do all those backups, or when there's a big influx of images. I don't do this as much, and re-doing work is not as much of a hassle, it's not about the photograph as an end in itself. As long as my key images are reasonably safe I'm Ok. And for my contemporary work, all my crazy presets that I made to help with some processes.

The most important thing to me is solving this sync chaos!, I need it to be working every day in the studio and it's been nearly two months. It's sending research images in a pretty constant stream (usually daily), and images that I take for painting commissions or to solve pictures. Sometimes I need to go bakc to the desktop in the middle of the day and make a bunch of copies with edits or presets, and then send them out to the studio on the iPad etc. It's pretty constant.

TY

 

[clee01l]

- The thing I'm finding a bit confusing in relation to your instructions is that images are only trying to re-sync right now in the context of me initiating a Rebuild Sync Data, with crashing on every attempt, so the process is never resolved. I'm not really sure what actually happens in this process, I sort of assumed it was a refreshing of all the linkages to each image already in Adobe Cloud.

I am assuming that on the Preferences {Lightroom Sync} tab, you are resyncing by clicking on the {Delete all Synced Data} button. If not what do you mean by ReSync?
Clicking on the {Delete all Synced Data} button deletes all of the Synced Smart Preview files from the cloud and starts over. Everything in the All Synced Photographs special collection and and Collection set to sync will be sent again.

If there are images that you do not want to sync, you need to remove these from the "All Synced Photographs" Special collection AND from any Collections the have the sync flag turned on or if necessary, remove whole collections from the sync process. FWIW, I have about 100,000 images in my catalog. Of these I only have about 15,000 in the Adobe Cloud. The Adobe Cloud is reserved for only my very best images or images that I have imported through Lightroom Mobile.

 

[prbimages]

Just clutching at straws, but ... a new version of Lightroom Classic was just released, Version 13.2, which includes some bug fixes, so it might be worthwhile upgrading and see if anything improves.

 

[mathewlynn]

I am assuming that on the Preferences {Lightroom Sync} tab, you are resyncing by clicking on the {Delete all Synced Data} button. If not what do you mean by ReSync?
Clicking on the {Delete all Synced Data} button deletes all of the Synced Smart Preview files from the cloud and starts over. Everything in the All Synced Photographs special collection and and Collection set to sync will be sent again.

If there are images that you do not want to sync, you need to remove these from the "All Synced Photographs" Special collection AND from any Collections the have the sync flag turned on or if necessary, remove whole collections from the sync process. FWIW, I have about 100,000 images in my catalog. Of these I only have about 15,000 in the Adobe Cloud. The Adobe Cloud is reserved for only my very best images or images that I have imported through Lightroom Mobile.

Thanks again Cletus - no what I'm doing and stuck on is Rebuild Sync Data, that option you get when you hold down the Option key in the Sync Pref panel, Lightroom is continually crashing at this stage. I went for this option trying to sort out the original problem with fresh Albums/Collections syncing Ok on devices but not filling up with the images.

I'm assuming Delete All Synced Data is the totally nuclear option and I'm kind of desperately trying to avoid that one! The reason being I have a very large number of synced Albums/Collections spread out in an extensive folder structure on my Devices/Cloud, I haven't counted them, but there could 5000-6000 of them. That's mainly due to my research process 'Collected Images' mentioned in the first post, it's mostly comprised of that, and it's something I like to have with me all the time. Deleting and re-syncing all these would take an eternity to move back into their Device/Cloud folders, sometimes up to a minute just to move one album when it's newly synced. Essentially, Delete All Synced Data will surely empty out what's in my device folder structure.

I realise this might be where I'm heading, and then I'd definitely only do it in small batches!, it would be the only way to keep track of what lives in the 135 main subject areas for Collected Images etc.

But, I've never done a Delete All Synced Data, so I'm not exactly sure what the behaviour is, I imagine it will just try and go through every Collection in LrC with a sync check and sync it? It's at this point I'd probably un-sync everything so I've got control with what's appearing on the devices and I'd move it around in careful batches re-populating the folders.

And yes, I see how you use it. I plan to do A LOT of gardening when I get it back to normal!

I may try an older backup again today to see what happens, I tried a bunch of them working backwards, none of them seemed to give me 'normal LrC'. I backup my catalog very regularly, but some of the behaviour around starting up from these has been a bit strange and unusual too TBH.

Thanks so much for hanging in there!

 

[mathewlynn]

Just clutching at straws, but ... a new version of Lightroom Classic was just released, Version 13.2, which includes some bug fixes, so it might be worthwhile upgrading and see if anything improves.

Thanks PRB!, yes I've done that with no improvement, I think it was by my Thursday post

 

[mathewlynn]

Thanks PRB!, yes I've done that with no improvement, I think it was by my Thursday post

*Peter

 

[mathewlynn]

View attachment 22781

Thanks again Cletus - no what I'm doing and stuck on is Rebuild Sync Data, that option you get when you hold down the Option key in the Sync Pref panel, Lightroom is continually crashing at this stage. I went for this option trying to sort out the original problem with fresh Albums/Collections syncing Ok on devices but not filling up with the images.

I'm assuming Delete All Synced Data is the totally nuclear option and I'm kind of desperately trying to avoid that one! The reason being I have a very large number of synced Albums/Collections spread out in an extensive folder structure on my Devices/Cloud, I haven't counted them, but there could 5000-6000 of them. That's mainly due to my research process 'Collected Images' mentioned in the first post, it's mostly comprised of that, and it's something I like to have with me all the time. Deleting and re-syncing all these would take an eternity to move back into their Device/Cloud folders, sometimes up to a minute just to move one album when it's newly synced. Essentially, Delete All Synced Data will surely empty out what's in my device folder structure.

I realise this might be where I'm heading, and then I'd definitely only do it in small batches!, it would be the only way to keep track of what lives in the 135 main subject areas for Collected Images etc.

But, I've never done a Delete All Synced Data, so I'm not exactly sure what the behaviour is, I imagine it will just try and go through every Collection in LrC with a sync check and sync it? It's at this point I'd probably un-sync everything so I've got control with what's appearing on the devices and I'd move it around in careful batches re-populating the folders.

And yes, I see how you use it. I plan to do A LOT of gardening when I get it back to normal!

I may try an older backup again today to see what happens, I tried a bunch of them working backwards, none of them seemed to give me 'normal LrC'. I backup my catalog very regularly, but some of the behaviour around starting up from these has been a bit strange and unusual too TBH.

Thanks so much for hanging in there!

And actually, there's another bit of strange behaviour that crept in several LrC upgrades ago:

Sometimes, but not always, LrC doesn't show the sync check boxes when launched, this is what it looks like at the moment (and apparently none of the problems mentioned above!). The only way to get them to show when it looks like this is to make a new Collection, close LrC, then re-launch, and only then will the sync check box be next to the Collection and ready to check.

And in relation to all of the above, if I make a new Collection now to prompt this, I'm 99% sure it will head back to the Rebuild Sync Data loop and crash, I'll try it of course!

 

[mathewlynn]

And actually, there's another bit of strange behaviour that crept in several LrC upgrades ago:

Sometimes, but not always, LrC doesn't show the sync check boxes when launched, this is what it looks like at the moment (and apparently none of the problems mentioned above!). The only way to get them to show when it looks like this is to make a new Collection, close LrC, then re-launch, and only then will the sync check box be next to the Collection and ready to check.

And in relation to all of the above, if I make a new Collection now to prompt this, I'm 99% sure it will head back to the Rebuild Sync Data loop and crash, I'll try it of course!


View attachment 22782

View attachment 22783

Now it's back to this, it's stopped working at least once, and everything slows down in the lead up, running out of application memory needing a Force Quit etc.

I'm not really sure if it's still trying to Rebuild Sync Data, but as I've said that was the last thing I did.

I cleaned out the catalog file location and set up a file for the Mobile Downloads in Sync Prefs like this: