Sync Sync, Memory, CPU, and Network (14.0.1)

[neilhunt]

Lightroom Version Number

14.0.1

Operating System

1. macOS 15 Sequoia

After getting impossibly stuck, I decided to start over and sync from LrC (14.0.1/Mac-M1Pro) to a brand-new empty LR in the cloud.
300k images, 200k in collections potentially to sync.

I created a new account and verified a clean empty cloud database. I connected my LrC catalog to that, enabled sync for a handful of folders, and watched 10k photos update pretty quickly. I enabled sync for a bunch more folders, and got stuck, with about 100k pictures to sync, 12k arrived in the cloud, and basically nothing happening.

Starting LrC now requires 5GB of memory, but it quickly climbs to 20G, 40G, 80G, and sometime overnight gets to 200G, and then blocks because "Running out of system memory".
Meantime, the CPU is churning to put it at the top of the list: 200-300% of CPUs (2-3 threads running full time).
But the network usage is pitiful: it's sent 137kB in half an hour. It's up to 72GB of memory.
It's been three days, and the synced number changes by 10-100 a day. At that rate, 100k is going to take 3 years to sync. And I'm going to have to restart LrC several times a day to keep it from getting blocked!

Any hints how to get it going again?
What to look for to see why it's stuck?

----

I looked on the web at lightroom.adobe.com and hit the "Sync issues" folder: it was showing 900 photos "missing" from the cloud. With some sleuthing, I found this was due to two collections that I had obviously enabled for sync, and (since it's so sluggish to respond) must have clicked again and disabled for sync. Not synced, and all the sync issues were from those two collections. I re-enabled sync for those two collections, and the number of sync issues has fallen to 600 in two hours. A little faster than 100 a day, but not much.

Interesting - the total number of synced photos reported on lightroom.adobe.com has stayed absolutely steady at 12971 all morning - I guess that must count the sync issues as part of the count.

 

[clee01l]

Help me to understand.
You are syncing Smart Previews for LrC. None of the collections contain movies and none of the images are reported missing in any collection. Is that correct? How much free space do you have on your primary volume to hold the vast quantity of temporary files that the Sync process will generate in working storage? When is the last time you rebooted the computer to free up working storage?

How many CPU cores and GPU cores does the M1Pro have? How much unified meamory?

When you display Sync Preferences, What is showing in Sync Activity?
If you display Sync Preferences and press the {Option} key, Then the {Sync Status on Web} button, What does the Sync Issues web page show? Can you post a screen shot?
Next, if you are not able to determine the problem, display Sync Preferences and press the {Option} key, Then the {Sync Status on Web} button. Generate a diagnostic report.

 

[neilhunt]

Yes.
Zero files missing.
Handful of movies.
1TB free on 8TB of disc.
Last rebooted yesterday afternoon.
64G of memory
CPU: 10 cores (8+2)
GPU: m1 metal 32 cores

I have generated a diagnostic report - didn't mean much to me - is there somewhere I should send it?
The HTML file is too large to view in Chrome, but before it crashes, looking at the lists of files to sync etc. nothing seems broken or incorrect.
I can send you a GoogleDrive link if you want to take a look...

To be clear, the issues I want to resolve are:
* performance: extrapolating current performance suggests 3 years to sync, and
* secondarily memory usage: every few hours it maxes out at 200+ GB working memory and needs to be restarted. (It's up to 130G right now, after running for 1h30)

Sync Preferences: Shows 11851 photos syncing (hasn't changed materially since this morning)
The dialog lists thousands of files all (as far as I can see): SyncType = "Uploading", Pathname = "<valid>", Sync Details/Errors: "Image"
I counted over 1000; the scrollbar is about 10% down, so credibly it lists all 11851 photos trying to sync.

Sync Issues on the web shows (now) 492 "issues" (down from 550 when I wrote this morning) - all of which are images in collections in LrC which are marked to sync, and are valid images, which appear all to be from two collections:

 

[clee01l]

How do you get 8TB of disk in a MBP? I was not aware that Apple ever shipped a MBP with more than 1TB or perhaps 2TB of internal disk.

Movies won't syne and may be the cause of your sync hanging. Remove all movie format files from the Collections being synced and restart LrC and sync.


Zip and send the generated Diagnostic report to me at [email protected]

 

[neilhunt]

It feels like I'm getting throttled.

I've discovered that if I shut down LrC and restart it, it immediately syncs 10, or sometimes 20 pictures within the first 30 seconds. Then freezes, and uploads nothing more for the next couple of hours (it just keeps growing larger and larger in memory).

Another shutdown, another 10 pictures. I can shut it down and restart it every 30 seconds, and get 200 pictures in ten minutes. Very predictable. Very unsatisfactory.

One other thing: if I shut it down after only 30 seconds, shutdown is nearly instant. If I wait an hour, shutdown might take 3-5 minutes before the app finally goes away.

 

[clee01l]

The behavior that you describe sounds very much like a machine that has run out of freespace in the primary drive and has over filled working storage, I still do not understand how you can get an 8TB disk as your primary disc (MacIntosh HD)

Your M1 Pro has 10 CPU cores and 32 GPU cores. Once again Adobe has removed the Neural Engine for the process due to bugs.
What is your bandwidth of the inaternet connection on the computer? Are you using WiFi or do you have a wired Gigabit ethernet connection.

When you run a speed test like https://www.speedtest.net/. What upload speeds are reported? Does your local internet provider (ISP) throttle uploads?

 

[neilhunt]

I cleaned up a bit, and got to 2TB free. Also upgraded to 15.1 and got a clean boot with little else running. I also cleaned up the sync issues (figured out you have to remove from all collections, then remove from cloud for each, and wait a long time). Overnight it synced exactly zero photos. Today, it's accelerated to about 100 an hour. That means a full sync of 200k photographs will take 2k hours, or about 3 months. Better than the 3 years I had a couple of days ago. Still pretty poor. Not ready for prime time, I feel...

Internet is Comcast 10-50Mbps upstream, load shared with Starlink, 7Mbps upstream. Wifi is from a Unifi WAP about 6 feet away serving just my MBP and phone. It's 2x2 MIMO Wifi6. Fast.com (from my MBP) reveals 300/39Mbps right now, (on top of the Lightroom sync).

And yes, I really do have 8T (2T free):

 

[neilhunt]

Another observation: I have a plugin script that I have written that iterates over selected photos, matching them up with Strava tracks and iNaturalist observations, and updates titles, GPS coordinates, and keywords. This script is in two parts - one creates a table of updates, and the second loads the updates into the LrC catalog. The first part runs now as always at 50-100 photos per second. The second part usually runs at a rate of 10-20 pictures a second; today while LrC is actively syncing, it is practically moribund, taking approximately 300-600 seconds per photograph. I'm guessing that it is hanging up at the "withWriteAccessDo" call to open a context to update each photograph. Perhaps this gives some hint why the sync process is running so slowly too.

 

[clee01l]

Fast.com (from my MBP) reveals 300/39Mbps right now, (on top of the Lightroom sync).

I think that your 39Mbps upload speed is what is killing your sync efforts. My AT&T fiber is 1Gbit to my router and I have Gbit wiring in my house. The Mac Studio is wired about 8 feet from my wiring closet and Router. I see 919 Mbps Down and 929Mbps up speeds. So, IMO your sync bottleneck is your WiFi and Comcast. I don't believe that LrC v14.0.1 and MacOS are even challenged because of your limited bandwidth.

 

[clee01l]

200k photographs

Another way to look at this is to estimate the total size of these files in MB As an example, I looked a a folder containing 22,000 files. It reports 512 GB (511,898,667,107 bytes or 4.095E12 bits) If I divide that by your upload speed 39mbits I eventually can determine how long it would take to upload those files. you can do the same math on your 200,000 files.

 

[Jim Wilde]

A couple of observations:

1. I don't see how 300/39Mbps speed can be regarded as "limited bandwidth". Prior to my recent house move was network speed was only 40Mbps down and 6Mbps up, and during that time I uploaded around 20k images (500GB) directly to the cloud back in 2017 and it took around a week IIRC. However, I was careful to do the upload in smallish chunks, such as a year at a time, and I did nothing much else while that upload was going on.

2. If the OP is uploading from LrC then of course only Smart Previews are being generated and uploaded, and they will average out at between 1 and 2 MB per SP. So 200k smart previews would be between 200 and 400GB, and with a consistent upload speed of 39Mbps that should take around 22 hours (though it would likely be somewhat longer due to the next point).

3. The speed of uploading to the cloud is obviously constrained to the slowest part of the chain, no matter how fast the speed between the user and their ISP. It was always noticeable that the actual speed I was getting would change throughout the day (best time for upload from the UK was usually morning when most of the US was asleep and so not bothering the servers in CA).

 

[clee01l]

I uploaded around 20k images (500GB) directly to the cloud back in 2017 and it took around a week IIRC. However, I was careful to do the upload in smallish chunks,

The OP is trying to upload 200,000 Smart Previews not 20,000. And the OP is not trying to sync in smaller chunks.

So 200k smart previews would be between 200 and 400GB, and with a consistent upload speed of 39Mbps that should take around 22 hours (though it would likely be somewhat longer due to the next point).

3. The speed of uploading to the cloud is obviously constrained to the slowest part of the chain, no matter how fast the speed between the user and their ISP. It was always noticeable that the actual speed I was getting would change throughout the day (best time for upload from the UK was usually morning when most of the US was asleep and so not bothering the servers in CA).

The OPs ISP is Comcast (a cable TV provider) that is notorious for limiting the bandwidth of their customers to accommodate the nature of cable transmissions and to provide more download bandwidth for TV and Movies.

 

[Jim Wilde]

The OP is trying to upload 200,000 Smart Previews not 20,000. And the OP is not trying to sync in smaller chunks.

My point was that the combined size of the OP's 200,000 smart previews is SMALLER than the combined size of my 20,000 originals. I managed that upload in about a week using a network connection theoretically around 1/6th of the speed of the OP's. IOW, a 39Mbps upload speed should be more than adequate for the task if that connection isn't being throttled to the point of unworkability.

 

[clee01l]

My point was that the combined size of the OP's 200,000 smart previews is SMALLER than the combined size of my 20,000 originals. I managed that upload in about a week using a network connection theoretically around 1/6th of the speed of the OP's. IOW, a 39Mbps upload speed should be more than adequate for the task if that connection isn't being throttled to the point of unworkability.

On this we agree. I think the connection is being throttled.

Over the weekend I was in a rural location with only a cellular connection on my iPadPro. My down speeds were reported at 16-25Mbps and up speeds were 0.3 -0.64. Making it impossible to sync my iPad with the Adobe cloud either direction.


Sent from my iPad using Tapatalk

 

[neilhunt]

Another way to look at this is to estimate the total size of these files in MB As an example, I looked a a folder containing 22,000 files. It reports 512 GB (511,898,667,107 bytes or 4.095E12 bits) If I divide that by your upload speed 39mbits I eventually can determine how long it would take to upload those files. you can do the same math on your 200,000 files.

Over a 24 hour period Nov 4-5, in a period of sustained better performance, I managed to sync 2,400 photos. The largest originals are 50MB., and if I were uploading the full originals at maximum size, that would be 2400 x 50 x 8 = 960,000 Mbits. (Most photos were more like 10MB, and one assumes that the smart preview is considerably smaller.). A better estimate would peg a smart preview at 1MB (my cache is around 300GB for 300k pictures), which would make the upstream demand a mere 19,200 Mbits for 2400 pictures.

Meanwhile, my up channel for those 24 hours had a capacity of 39Mbps x 3600 x 24 = 3,369,600 Mbits - at least 4x bigger than the complete upper bound of the channel, and 200x bigger than a more realistic estimate. This makes me think that 39Mbps uplink speed is nowhere near the limiting factor.

On another occasion, I have touched 1TB of files triggering a cloud backup refresh to Google Drive that completed in 3 days, which is well over 30Mbps - so I have considerable confidence that Comcast is not limiting my traffic anywhere near a limiting factor for sync.

The math you propose is simple - The 200k originals occupy 3TB of space on my disc; the originals would upload in 9 days at a rate that I have previously seen, and 7 days at 39Mbps. The rate I am getting (on a good day) will finish syncing in 89 days - about 10x worse that fully uploading full sized originals, and probably 500x worse than uploading smart previews.

 

[neilhunt]

On this we agree. I think the connection is being throttled.

Over the weekend I was in a rural location with only a cellular connection on my iPadPro. My down speeds were reported at 16-25Mbps and up speeds were 0.3 -0.64. Making it impossible to sync my iPad with the Adobe cloud either direction.


Sent from my iPad using Tapatalk

But I'm not in a rural location with poor connectivity or on a cellphone! I'm in the heart of Silicon Valley with a premium connection and high performance network gear. There is no way that upchannel performance (or Comcast throttling) is limiting this.

Also, before my cloud copy got mangled, I routinely synced thousands of pictures an hour, using the same equipment and connection.

[By way of credentials, I was head of engineering at Netflix until 2017, where we pushed better than 30% of last mile traffic. I know a little about network performance!]

 

[neilhunt]

Why am I posting here? I've tried similar questions on the Adobe forum and had little response.

Adobe has just completely redone syncing, and supposedly it works much better. I have a test case that demonstrates that it does not, at least in one particular situation, that may push some boundaries of size and complexity. I was hoping that perhaps I'd get the attention of an insider who could maybe use my case to debug some of the challenges, since I am somewhat technically capable and can perhaps help provide a specific problem to investigate. I will stop posting more on this here; you know where to find me if you are such an insider and this is of interest.

I do thank Clee and Jim for their helpful suggestions. I know it takes time to unravel what is being asked, and suggestion are always welcome.

 

[clee01l]

I'm in the heart of Silicon Valley with a premium connection and high performance network gear.

Why do you not have at least GiGaBit service to your router? I don't consider Comcast to be a Premium connection. I would think Silicon Valley would be overflowing with GiGaBit ethernet.