Sync Changed LR catalogs, now have 1000+ smart previews disconnected

[mrskayleehorne]

Lightroom Version Number

14.0.1

Operating System

1. Windows 10

Hello! I’ve been struggling with Lightroom for about a week. I have started a new catalog and decided to sync this with the adobe cloud for iPad editing instead of the old catalog I was running off of for the past two years (yikes, I know!) — Once this new catalog started to sync, it went through the process of re-“downloading” all of the previous catalog (wasn’t expecting or wanting that, but I let it do its thing) and I did my best to clean everything up, get rid of the collections I didn’t need, and reconnected folders where they were missing once it was done.

Well, it never seems to be done!! I’m still sitting on over 25,000 unmatched “downloaded-smart-previews” and an additional 1,700+ still “syncing” from the cloud, but that number and sync status just ping pongs up and down by 5-10 images every few seconds. All of my folders are matched up, despite some of the remaining smart previews definitely belonging to them and the amounts are not matching the file amount in the root folders… Is there any way to batch select and find missing photos to finish this up? I can’t sit and do it individually for all of them but I also would be really sad to loose my edits.

All of this cloud work is adding up to a huge waste of time in the effort of doing something that Adobe had laid out as a simple catalog switch. I spend a lot of time editing away from our main desktop hub where all the files are imported and live, so using an iPad integration is kind of crucial for my workflow right now.

Appreciate any insight you can give.

 

[clee01l]

You can sync only one LrC catalog at a time to the Adobe Cloud. Consider merging this new catalog into the old catalog and the sync should settle out.

Having tried to sync the new catalog will cause the old synced images to be removed from the Adobe Cloud or downloaded to the new catalog.

 

[Califdan]

Since August, I've been running tests and documenting the intracacies of what happens when you change which catalog is syncing with LR/Cloud and it's not pretty. I'm dong this in anticipation of writing a blog on the topic. So far I have 5 (maybe 6) unique test cases and the blog, even before adding screen shots, is 26 pages long in Word. I thought it would take me less than a week to determine what happens and write the blog but it turned out tp be so complicated, with very strange and unexpected results, that I had to run the same tests multiple times to assure myself that I was seeing what I was seeing.

Your post seems to coorespond with what my tests are showing. When you sync a new unrelated catalog, everything that's in Lr/Cloud comes down to it with the images in whatever form they are in in the cloud. If they originated in LrC, then that form is an SP (Smart Preview) with no physical image. If they had originated in Lr/Cloud, the full size original comes down to LrC. However, if the newly synced catalog already has the same images in it things different things happen depending on a whole raft of factors. Depending on these factors, sometimes the Lr/cloud image becomes a VC (Virtual Copy) hooked to the one already in LrC. Sometimes it gets added as a new full image and sometimes just an SP gets added.

In the case of the SP's if you use your OS to place the original image into the folder shown in LrC as the folder where the SP is, it will no longer be considered as missing. But there are lots of nuances and it would help to understand in minute detail exactly what steps you performed. For example how did this new catalog come into existence? Did you start with an empty catalog and re-import everything? Did you use an "export as catalog"? Did you start with a new empty catalog and just make it the sync catalog?

In addition, again depending on a whole set of factors, sometimes the original image in LrC will no longer will sync even though it is marked as syncing and is present in the "all synced photographs" special colleciton. you make a change to the image and the sync status goes up by 1, then nothing happens in Lr/Cloud, the sync status goes down by 1 and sometimes the image in LrC is automatically removed from all synced collections, and from the All Synced photographs specail colleciton and looses the synced icon. This all dependfs on things like where the image entered the Lightroom ecosystem (cloud or classic) how the new catalog got there and attained images before making it the sync catalog (eg re-import to new catalog, revert to a backup catalog, export/import as catalog, etc.) and what, if anything, was done to images along the way (for example after the backup you reverted to was taken but before you made it the sync catalog) That's why we need to know the exact steps in squence that got you where you are.

 

[Gnits]

@Califdan I admire your skillset and tenacity to progress this issue.

What I do not understand is that the Adobe architect of this synch ecosystem does not provide a detailed description of what happens (or should happen) when a user changes the catalog linked to the cloud ecosystem. They obviously coded this to behave in a certain way. I think there is a high degree of reckless disregard by Adobe for the integrity of users datasets, both within the cloud and within an LrC users catalogs.

Imagine the uproar if banks approached credit card or atm transactions in a similar manner.

 

[clee01l]

They obviously coded this to behave in a certain way.

I think the intention was for there to be one and only one catalog to sync ever. As Dan is finding the sync process was not designed to work with multiple catalogs and it looks like this might have been shortsighted. The revised sync process I think is an attempt to rectify some of the short comings of the sync process but still does not address the attempts with multiple catalogs issue.

 

[mrskayleehorne]

For example how did this new catalog come into existence? Did you start with an empty catalog and re-import everything? Did you use an "export as catalog"? Did you start with a new empty catalog and just make it the sync catalog?

I grabbed four collections with a few sessions I hadn’t fully begun working on yet and added them to a collection set, then exported that as a new catalog.

I didn’t do anything to the files from there, just gave it a once over - so nice and tidy! Then hit sync and it all starts to hit the fan. The Catalog started pulling all the cloud data in, placing it under the old collection folders & titles in a set called “From Lightroom” — that’s fine, I’ve had it do that before. I didn’t have a ton on my iPad “cloud catalog” so I was expecting those to come through and then I’d just need to manually un-sync them from the LrC end. I let that run for several hours and then realized it was pulling things from the cloud that I had stopped syncing from LrC years before. At this point I may have made a mistake? I went into the Lr cloud online, and I started deleting files from 2 years ago that definitely didn’t need to be reaccessed or synced. These are still hanging out in the “deleted” area of cloud, ticking away until the 60 day mark to finalize that deletion or I can restore them.

I always have imported raw files to our HD and backup drive, pulled those into Lightroom from the hard drive, synced them into a collection - done the work, export - then un-sync that collection when I’m through. My cloud “footprint” is remarkably small and is all about utilizing SP, it feels like it should be a very simple and clear pathway to and from cloud for these files... so, why would 2 year old, definitely finished and un-synced files be coming through or in the cloud at all? Once they got pulled in, they weren’t in the “From Lightroom” collections, but in a separate downloaded-smart-previews folder within the C drive photos folder.

(As an aside, I do not have “Specify location for Lightroom’s Synced images” checked within Preferences. Should I do this, and if so, when? I let LrC run overnight and it still is ping-ponging the sync number around 1,100 files.)

 

[Califdan]

I think the new architecture was an attempt to keep there from being so many images stuck in mid sync which it may have helped but certainly did not completely resolve. I presume they decided that the tech support resources being spent with users' stuck images was larger than the redesign effort.

The more I play with it the more it seems that the base case (i.e. don't swap catalogs or only swap to a new catalog that is a direct backup of the current catalog with no intervening changes done in either Lr or Lrc since the backup was taken) works pretty well. The problem as I see it is that there was no clear goal with other scenarios (e.g. should recovering from a backup catalog put everything back to the state at the time of the backup or should it put everything back to the state of the BU catalog EXCEPT things that had synced to the cloud in which case the Cloud version wins (they seem to have choosen the latter). However, I don't believe they throught through or tested all the permuations that could reasonably be expected and as such there are a lot of un-intended consequences and inconsistent behavior.

Unfortunatly folks like @mrskayleehorne who started this particular thread are "stepping into it" as they say and are suffering the effects of a poorly designed and/or poorly implemented or at least poorly tested process.

I grabbed four collections with a few sessions I hadn’t fully begun working on yet and added them to a collection set, then exported that as a new catalog.

Okay, so this tells me that the new catalog is not a blood relative of the old catalog. By that I mean that internal id numbers on the images in the new catalog are not the same as they were in the old catalog which in my testing seems to be relevent.

I didn’t do anything to the files from there, just gave it a once over - so nice and tidy! Then hit sync and it all starts to hit the fan. The Catalog started pulling all the cloud data in, placing it under the old collection folders & titles in a set called “From Lightroom” — that’s fine, I’ve had it do that before. I didn’t have a ton on my iPad “cloud catalog” so I was expecting those to come through and then I’d just need to manually un-sync them from the LrC end.

This is consistent with my tests.

What is not clear to me is what result you are looking for? Do you want all the Lr/Cloud images in the new catalog or only the subset of images in the export and import as catalog broutght over? And, what do you want in the Lr/Cloud? Everything that was there before or just the images you brought over from the old catalog with the export/import?

I let that run for several hours and then realized it was pulling things from the cloud that I had stopped syncing from LrC years before.

This is probably not a true statement. Most people believe that if they remove an image from their synced collections that stops the image from syncing. This is not the case. In order to stop an image from syncing you need to removed it from the "All Synced Photographs" special colleciton in the Catalog Panel (which also removes it from all other synced collecitons). Most folks don't know to do that.

At this point I may have made a mistake? I went into the Lr cloud online, and I started deleting files from 2 years ago that definitely didn’t need to be reaccessed or synced. These are still hanging out in the “deleted” area of cloud, ticking away until the 60 day mark to finalize that deletion or I can restore them.

That's not a problem. Removing them from Lr/Cloud has should un-sync them in LrC (assuming they had gotten there) but otherwise leave them in LrC. Having them in the deleted list in Lr/Cloud is also not a problem but if you wish you can right click on the "deleted" album and select to empty that album or open the deleted albums, select as many images as you like and then right click on one of them and select to delete it. (BTW, my commets are using the LR/Cloud desktop appl - it may be a bit different is using one of the other Lr/Cloud apps

I always have imported raw files to our HD and backup drive, pulled those into Lightroom from the hard drive, synced them into a collection - done the work, export - then un-sync that collection when I’m through.

In your first sentance of this section do you mean Lightroom Cloud or Lightroom Classic? Sounds like you mean Lightroom Classic as cloud doesn't have collecitons. As said before, unsyncing the collection does not unsync the photos in it so they are all still in the cloud and syncing with LrC

My cloud “footprint” is remarkably small and is all about utilizing SP,

This validates my assumption that you imported into LrC. Hiowever, unless you'd also been deleting from the All Synced Photographs the could set of images is probably way biggetr than you thought. In fact you mentioned that after swapping catalogs it was downloading things you un-synced years ago which validates my point.

it feels like it should be a very simple and clear pathway to and from cloud for these files... so, why would 2 year old, definitely finished and un-synced files be coming through or in the cloud at all?

Already answered

Once they got pulled in, they weren’t in the “From Lightroom” collections, but in a separate downloaded-smart-previews folder within the C drive photos folder.

Correct. You may also notice that they are SP's, not full size images and if you take a peek, the "downloaded-smart-previews" folder on disk is empty and in LrC those images are marked as SP's only with the original files miss

(As an aside, I do not have “Specify location for Lightroom’s Synced images” checked within Preferences. Should I do this, and if so, when? I let LrC run overnight and it still is ping-ponging the sync number around 1,100 files

I reccomend changing this folder but not if there are images be sent back and forth at the moment.



Here's what I suggest IF your goal is to have a new catalog with ONLY the images in the few collections involved in the Export/Import as catalog AND you only want those same images in the Adobe Cloud. However I don't think you need this complicated a process at all going forward. if the goal is to have a big master catalog but only have a few collections synced at a time then ignore my steps below, let me know and I'll send a new set of steps.

1. In LrC, Paus syncing
2. Delete the new catalog
3. in Lr/Cloud select "All Photos" and delete them all (I'm counting on your statement that there are only SP's synced from LrC in your cloud account - no oirignal images) and that the full originals are still safely on your local hard drives and are in your original LrC catalog.
4. if desired emtpy the deleted album in Lr/cloud
5. Delete all user albums in Lr/Cloud
6. In LrC, Re-o the export as Catalog
7. In the new (small) LrC catalog, unsync the collections you brought over If they are marked as syncing
8. Verify that the "All Synced Photosgraps" special collection is empty
9. In the new LrC catalog set the sync folder to be some real empty folder on your system
10. In new LrC catalog make it the sync catalog. Do not have it copy the sync data from the old catalog
11. Nothing should sync at this point. Verify that Lr/Cloud is still empty and that the All synced photographs is also empty
12. Sync one of the collection you brought over. Verify that it arrived OK in Lr/cloud and that an album with the same name was created. No SP's should come down to LrC from Lr/cloud as you deleted them all in Lr/Cloud
13. If that was OK, then syunc the remaining collections

This new catalog will not be properly synced with the cloud. Your Old (big) catalog will still be fine but not synced.

 

[mrskayleehorne]

Okay, perhaps I am getting marginally closer. I’ve been working on this for the majority of today…

It’s sitting at around 670 pending sync files.

How you explained All Synced Photographs, YES, that’s exactly the mistake I’ve been making for 2 years. Untangling that web of smart previews is such a pill but I believe I’m closing in on it…

What is not clear to me is what result you are looking for? Do you want all the Lr/Cloud images in the new catalog or only the subset of images in the export and import as catalog broutght over? And, what do you want in the Lr/Cloud? Everything that was there before or just the images you brought over from the old catalog with the export/import?

Yes, so to answer this question: I definitely was thinking it would only carry over or “link” back to the photos I exported into the new catalog. Total rookie move - I just don’t deal with catalogs often enough to remember that it doesn’t work this way. I wanted to downsize what was on the cloud since those files don’t need to be interacted with anymore. I’d rather keep what’s been taken in the past 6-8 months max, since anything past that has been edited and exported and delivered. In all likelihood, I won’t return to it.

I still have 26,000+ “Missing Photographs” without folders to batch-match them to, so I’m not sure what to do there other than wait and give the computer and servers time to catch up on syncing before trying to connect them again. (The “Connect nearby photos” checkbox doesn’t seem to work when I reconnect files right now.)

I really appreciate your detailed breakdown and instructions. I’m tempted to start a new catalog from scratch, reimport my unedited work, and empty the entire backlog from Cloud (which is the same as All Synced Photographs?) to behave as an entirely empty and new cloud… is this possible? Can you tell me if that would wipe the edits or anything relevant from LrC as well? I feel like the answer is “no,” because they’re smart previews and the edits are actually stored in the .xmp file on the desktop end once synced (?), buuut everything I assume I know with Adobe for the past decade proves to be wrong somewhere. I appreciate your time - you are a knowledgeable resource here, it’s a relief to just speak with someone who knows how whack it is.

(For what it’s worth, as I’ve written the above info, sync jumped to 1,074 photographs being synced.)

 

[Califdan]

No need to "start over with new catalog" In fact I suggest that YOU DO NOT DO THAT.

You also don't need a 2nd catalog. What you need to do is go back to your master catalog, Really un-sync everything from LrC, and clear Lr/Cloud. Then just synce the images you want. Here are the steps

1. In LrC, with the currently syncing catalog open, Pause syncing
2. Close LrC
3. Delete the that new catalog
4. in Lr/Cloud select "All Photos" and delete them all (I'm counting on your statement that there are only SP's synced from LrC in your cloud account - no oirignal images) and that the full originals are still safely on your local hard drives and are in your original LrC catalog.
5. if desired emtpy the deleted album in Lr/cloud
6. Delete all user albums in Lr/Cloud
7. In LrC open your master catalog (the one you exported FROM)
8. Assure that sync is still paused
9. (you may want to deal with any missing images at this point but it is not requred)
10. Un-sync all collections that may be marked to sync
11. Remove all images from the "All Synced Photosgraps" special collection
12. In preferences, set the sync folder to be some real empty folder on your system
13. Un-pause sync. It will ask if you want to make this the synced catalog - Yes. It will also ask to copy the synced images - No
14. Nothing should sync at this point. Verify that Lr/Cloud is still empty and that the All synced photographs is also empty
15. Sync one of the collections you brought over. Verify that it arrived OK in Lr/cloud and that an album with the same name was created. No SP's should come down to LrC from Lr/cloud as you deleted them all in Lr/Cloud
16. If that was OK, then syunc the remaining collections

This whole thing should take less than 30 minutes

Going forward, when you want to remove a collections worth of images from Lr/Cloud do this

1. open the collection in LrC
2. Unsync the collection
3. Select all the images in that now un-synced collection
4. With those images still selected, click on "All Synced Photographs" special colleciton. Those same images will still be selected
5. Hit the delete key on your keyboard to remove them from the "All Synced Photograpsh' special collection. This will remove them from Lr/Cloud (I assume those images are not in any other synced collections but if they are they will also be removed from them)
6. If desired you can also delete the collection or leave it as you see fit.

 

[mrskayleehorne]

Okay! Super helpful! Haven’t touched it yet, I’ll take a look in the morning. Thank you!