Thoughts on backing up Lightroom cloud?

[Laura Smith]

Lightroom Version Number

Lightroom 3.3 X64 [ 20200611-1039-7df19f1 ] (Jun 11 2020)

Operating System

1. Windows 10

I'm finding myself using Lightroom cloud much more than Classic, so I'm looking for workarounds for the things I still use Classic for, to see if I can stop using it completely. One thing I still use Classic for is a local backup. It'd be great to hear your thoughts on the pros and cons of different approaches to backing up Lightroom cloud without Classic.

Option 1: "Store a copy of all originals at the specified location"

Seems like a no-brainer: just do this and then backup that directory using my normal backup system. But I have a small SSD in my laptop, so the originals would have to be stored on a USB disk that isn't connected all the time (that's what I do with Classic at the moment, and just have the current month's photos on the SSD). Has anyone tried that on a removable disk? How does Lightroom cope if you try to use it without that disk? Does it just happily use its cache and pull originals on demand as usual? Or does it get confused?

Massive downside of this idea: I don't have a local copy of edits.

Option 2: "Export Original + Settings"

This will give me a local backup of edits, as well as let me put it on a removable disk without any hassle. Periodically I would go to All Photos, sort them by Import Date, select all and export to the USB disk (which itself is backed up locally on another disk).

Downsides: It's a manual process that I have to remember to do and has the room for human error.

If you've got anything to share on these two options or if you've got an Option 3 of your own, that would be great to hear!

 

[clee01l]

Lightroom (cloudy) stores local data in the "Lightroom Library.lrlibrary". Using your system back up app for that file is sufficient along with a backup of any original image files stored at the specified location. What your goal should be is to recover from any catastrophic disk failure. Adobe has adequate protections in place to preserve any user data in the cloud.

 

[Johan Elzenga]

Lightroom (cloudy) stores local data in the "Lightroom Library.lrlibrary". Using your system back up app for that file is sufficient along with a backup of any original image files stored at the specified location.

I respectfully disagree. In theory this may be correct, but only in theory. The XMP metadata will be very hard to find if you would ever need to retrieve the edits from a backup of this library.

 

[Jim Wilde]

Lightroom (cloudy) stores local data in the "Lightroom Library.lrlibrary". Using your system back up app for that file is sufficient along with a backup of any original image files stored at the specified location.

Backing up the local library appears to be a waste of time and disk space. The catalog inside the local library is subservient to the catalog in the cloud, so if you make a big user error (e.g. changing metadata inadvertently on multiple selected images) you cannot restore the local library from a backup taken before the error in order to revert to the pre-error state. As soon as you replace the local library with the backup, then launch the desktop app, the local catalog is immediately updated to the same state as the (erroneous) cloud catalog. As soon as I found that out I removed the local library from my backup procedures.

Backing up images is easy, backing up edits is less so.....it's either a manual export with settings, as Laura is doing, or periodic use of the Downloader app (which is really buggy as at the last time I tested it earlier this year). Or jump through some hoops to keep originals in Classic and Cloud in sync.

 

[clee01l]

I respectfully disagree. In theory this may be correct, but only in theory. The XMP metadata will be very hard to find if you would ever need to retrieve the edits from a backup of this library.

The complete picture is this comment that you ignored. “Adobe has adequate protections in place to preserve any user data in the cloud.”


Sent from my iPad using Tapatalk

 

[clee01l]

Backing up the local library appears to be a waste of time and disk space. The catalog inside the local library is subservient to the catalog in the cloud, so if you make a big user error...

I was not considering the lack of version control with Adobe’s cloud service. But for preserving your correct image data the Cloud is adequate. And that was the basis of my comment.


Sent from my iPad using Tapatalk

 

[Laura Smith]

Thank you all for sharing your thoughts - invaluable as ever! It's got me thinking to clarify what I actually want the backup for. As you say Cletus, Adobe has the data protected more carefully than I could. So for a disk failure I would happily rely on it being there and just pull everything down from the cloud again. I think there are probably four reasons I think I want a full local backup:

1. To be able to access my images if I have a local internet failure. On reflection, I could probably cope: I'm not doing this for a living.
2. Because I've always gone by the idea of owning my own data, in open or at least commonly interchangeable formats. Because it just seems like a good idea, because you never know what might happen. Perhaps this is just a principle I could learn to live without - does it actually serve any practical purpose in this instance? Adobe could disappear overnight. But even in 2020, the year of the unexpected, that seems pretty unlikely. Doesn't it?!
3. To ensure my family can easily access sentimental photos if I'm not, well, here. I think is actually important, especially given that my edited photos in this category live on Lightroom web galleries. So there is no copy of output files elsewhere. Would unedited originals serve this purpose? Probably.
4. To account for user error on my part. Perhaps deleting a load of images accidentally and not realising until a year down the line when I go looking for them, by which time they aren't in the cloud bin. This seems pretty important to me. And probably the most likely use case for a local backup, at least in the immediate future!

I'm a reboot away from finding out what happens if Lightroom wakes up without the external hard disk where it has originals stored (I can't for the life of me unmount the darn thing without rebooting, but that's another story). I'll post back shortly...

 

[Laura Smith]

Well that seems to fail pretty gracefully:

There's a question to ponder about how well it manages the cache without the originals directory present. But assuming it does that well, then that's definitely an option.

Of course, that doesn't give a backup of edits, just originals. But it has automation going for it where exporting Original + Settings doesn't. Decisions, decisions... Thoughts?

 

[Laura Smith]

[...] or periodic use of the Downloader app (which is really buggy as at the last time I tested it earlier this year). [...]

I hadn't come across this. Just downloading it. What were the bugs?

 

[Jim Wilde]

1. To be able to access my images if I have a local internet failure. On reflection, I could probably cope: I'm not doing this for a living.
2. Because I've always gone by the idea of owning my own data, in open or at least commonly interchangeable formats. Because it just seems like a good idea, because you never know what might happen. Perhaps this is just a principle I could learn to live without - does it actually serve any practical purpose in this instance? Adobe could disappear overnight. But even in 2020, the year of the unexpected, that seems pretty unlikely. Doesn't it?!
3. To ensure my family can easily access sentimental photos if I'm not, well, here. I think is actually important, especially given that my edited photos in this category live on Lightroom web galleries. So there is no copy of output files elsewhere. Would unedited originals serve this purpose? Probably.
4. To account for user error on my part. Perhaps deleting a load of images accidentally and not realising until a year down the line when I go looking for them, by which time they aren't in the cloud bin. This seems pretty important to me. And probably the most likely use case for a local backup, at least in the immediate future!

1. I use the option to download smart previews (I don't store originals locally because I use a fully integrated "hybrid" Classic & Cloud workflow). They'd probably be perfectly adequate to use if you are offline.
2. It depends on how much faith you have in Adobe, and more importantly whether you'd be confident of being able to get to your cloud data in the event of a major issue. As I've said earlier, backing up images is easy, backing up edits/metadata is not so much.
3. That's really what the Downloader is designed to do, i.e. provide your family with up to 12 months to download your content should anything happen to you. Or for you to download it should you decide to switch away from Adobe. The issue for me with that first case is wondering what my family would make of a bunch of Raw+XMP sidecar files, many DNGs, and some Jpegs. Because I have Classic and Cloud in lock-step, I can "publish" all originals to jpeg, and in fact I did start that process last year but with other things happening that project is currently stalled.
4. It would depend on how effective your local backup solution is. Bear in mind that if you accidentally delete a load of images from the cloud, those deletions are automatically applied to the folder containing the local copy of the originals. So you'd have to have some effective version control for your backups that would allow you to access images deleted from the original source over a year ago.

 

[Jim Wilde]

I hadn't come across this. Just downloading it. What were the bugs?

There were a few niggly things which hopefully will get corrected. The main issue, however, is that metadata changes applied in Cloudy are (were) not included in the downloaded files....edits are, metadata isn't. That may be a bug, or may be by design. I've communicated my findings directly to Adobe, but their Downloader "team" aren't terribly communicative....so until I retry following the next update I don't know if it has been fixed/changed.

I don't know if you intend to run it, but you might get some surprising results. One of them is that for Jpegs that have been edited a separate "edited" version of those files is also downloaded, which has the edits baked in.....which makes a bit of sense, but no exif data is included, which doesn't make sense. And why just edited Jpegs, why not generate baked-in Jpegs for all downloaded files?

Another is that if you have imported Raw+XMP into Cloudy (not via catalog migration), the downloader will download the Raw+XMP.....but only if you have applied subsequent edits to those raw files in the cloud. Thus if you have imported previously edited raw files with an associated XMP containing those edits into Cloud, obviously in the cloud apps you will see the resulting edited image....but unless you subsequently apply another edit in the cloud, the Downloader will only download the original raw file without an XMP sidecar.

Hopefully these issues will be fixed in due course, but until they are you really need to consider the Downloader as the lifeboat (leaky) that it is intended to be, and not think of it as a means of backing up your edits and metadata. If you are really concerned about having that complete backup, exporting Original + Settings is the only really viable solution as things stand today.

 

[Laura Smith]

1. I use the option to download smart previews (I don't store originals locally because I use a fully integrated "hybrid" Classic & Cloud workflow). They'd probably be perfectly adequate to use if you are offline.

That's a definite possibility for me - would certainly be more than adequate for offline use. My SSD is only 224 GB. Smart previews for my catalogue so far would be 20 GB, for 17,000 images, and I have perhaps 10,000 more to import. That's heading towards a decent sized chunk of my internal disk, which is probably okay for now, but further down the line (without the option to choose a date range for storing smart previews) could become a problem.

3. That's really what the Downloader is designed to do [...] The issue for me with that first case is wondering what my family would make of a bunch of Raw+XMP sidecar files, many DNGs, and some Jpegs.

That's about what I'd arrived at today - I think you've helped me to realise that I probably need two stores of photos: a backup for me, and an outputted selection of edited JPEGs for my family. Do my family really want 100 similar unedited images, or just the one pick that's been edited and saved as a JPEG?!

4. It would depend on how effective your local backup solution is. [...] So you'd have to have some effective version control for your backups [...]

Current backup system is rubbish, but part of this process is fixing that - so yes, that would be the aspiration.

There were a few niggly things which hopefully will get corrected. The main issue, however, is that metadata changes applied in Cloudy are (were) not included in the downloaded files....edits are, metadata isn't.

Metadata! Thank you for saying that - I'd been preoccupied thinking about edits and hadn't considered metadata. I've made the decision to do all metadata input on cloud. So even my current local backup done through Classic isn't collecting metadata any more. That's quite a biggie. I'm spending quite a bit of time during lockdown going through the best part of 20 years of photos adding metadata. Any backup that doesn't have that metadata in it is hardly a backup, come to think of it. So that's probably just ruled out everything except manually exporting Originals + Settings, hasn't it? Backup through Classic: no metadata. Backup through Downloader: no metadata. Backup through getting cloud to save originals locally: no metadata. I think we have an answer!

I don't know if you intend to run [Downloader], but you might get some surprising results.

Oh dear that all sounds like it really is a Leaky Lifeboat of Last Resort! I installed it and had a look at it. Beyond the bugs, the other issue for me is that you can only download everything at once. I don't really want to have to download everything every time I make a backup. I'm on 195 GB already. So as you say, it's a tool that isn't designed for backup, but that isn't perfect at doing what it's designed to do either.

So I think my solution is to:

• Periodically (when I get around to it?! Eeek) export Originals + Settings to an external disk, keeping a series of versions as I see fit. Once I get all my old photos done with metadata and do a full export, I think I can probably happily live with just downloading the full current year (or current quarter, or something) every time. I'm unlikely to make big edits to anything old, and if I do, I can live with redoing them. After all, I'm doing this for fun, I'm not a pro spending hours retouching and earning a living from it.
• Periodically export edited selects at full size to a separate directory on the external disk, organised in a way that would make sense to anyone finding them.

This gives me a full, versioned backup for myself and a useful collection of photos for my family. All on one external disk. I think there's probably little value in me keeping a backup of the backup: because at the end of the day this is all just a backup of the Adobe cloud. So the chances of the two failing at once are so slim it's not worth worrying about. Although I might take one copy of the 20-year archive on another disk once I've finished organising it.

Not perfect. Not automated. But actually appears to be the only solution that backs up originals + edits + metadata. Thank you all for your help to understand what I need!

 

[jcsnyc]

Not to be negative, but I think a Backup of a Backup does make sense. As an example, Garmin has been dealing with a Ransomware issue for the past week or so. Were our friends at Adobe to be a victim AND you had a hard drive failure, you could lose a lot. I use a separate backup service to back up my Mac & External Hard drives so I have more control. (I am pretty happy with BackBlaze - though, with any backup service, that first backup takes a LOOONG time.)

I also have a TimeMachine backup going. While probably overkill, I thought I lost about 15 years of photos a few years back and that taught me a lesson...

And thanks for such a thorough and thoughtful thread. I plan to dig into all you learned as I improve my strategy for backups.

 

[akrabat]

So I think my solution is to:

• Periodically (when I get around to it?! Eeek) export Originals + Settings to an external disk, keeping a series of versions as I see fit. Once I get all my old photos done with metadata and do a full export, I think I can probably happily live with just downloading the full current year (or current quarter, or something) every time. I'm unlikely to make big edits to anything old, and if I do, I can live with redoing them. After all, I'm doing this for fun, I'm not a pro spending hours retouching and earning a living from it.
• Periodically export edited selects at full size to a separate directory on the external disk, organised in a way that would make sense to anyone finding them.

FWIW, I'm a huge fan of setting monthly reminders for manual repetitive tasks like this. I also agree with jcsnyc that while Adobe shouldn't ever have a problem, if they do lose all your photos no matter how much they say sorry you've still lost all your photos. Controlling your own disconnected backup is a good idea if you consider them irreplaceable.

Also, I really like the idea of a curated set of edited JPGs in a separate directory for others and am going to steal it! Mind you, I'm also coming to the conclusion that the only photos that will survive me long term are the ones in printed photo books…

 

[akrabat]

Somewhat related, this is the message on image.canon at the moment:

On the 30th of July, we identified an issue within the 10GB long term storage on image.canon. Some of the original photo and video data files have been lost.

We apologize for any inconvenience.

ouch!

 

[Chris Wimlett]

A really interesting thread. I’ve finally decided to go fully cloudy, as it all works so well with latest versions of PhotoShop, and there’s a lot of useful information here. Especially the curated folder idea, which I’ll also steal

My desktop will be my main system and I’ll store originals there, but I’m unclear about what to do about local copies of smart previews. Is it best to just to do this for my laptop and iPad for when I’m offline?

 

[hrboyce]

@Jim Wilde ... is there a link somewhere to this "fully integrated hybrid classic and cloud" workflow that you use? I have had nothing but issues trying to use them both together. Its gotten so bad I have given up on classic. Because of that know my backup strategy isn't perfect but I try to cover myself as best I can by having the cloudy version put a copy of my raw images on and external disk, exporting my finished edits as jpgs, and using the DOWNLOAD to capture raw + XMP files to a separate hard disk. I have found the LR Downloader to work well as I don't keyword or add much meta data beyond what the camera puts in the raw file. Running it periodically allows it to restart where it left of and minimizes the time required to have the second copy on my PC. I am curious about how having the XMP files would help. Finding software that can use them to recreate my edits accurately is not easy if I decided to no longer play Adobe's subscription game, what software would you recommend to import the raw and XMP files?

 

[clee01l]

@Jim Wilde ... is there a link somewhere to this "fully integrated hybrid classic and cloud" workflow that you use?

I don't know of any and AFAIK, Adobe does not endorse a hybrid workflow.
My hybrid workflow was developed after conversations with Jim.
It is as follows:

• Important points:
  1. Original files synced UP to the cloud do not count against the storage plan limits
  2. LrC syncs only SmartDNGs which are proxy files of the original that resides on the local disk.
  3. These Smart DNGs can substitute for the real thing in the cloud environment (Lr Mobile) and can be edited just like the original.
  4. Imported images that original in Lightroom are full sized images and are charged against your plane storage limits
  5. These full sized cloud originated images sync DOWN to Classic as full size originals
• It is not recommended that you run both LrC and Lr on the same PC. The redundancy is not helpful. If you should choose to, then there is no need to save a Lr original import locally since LrC will have and duplicate that same file,
• The Keyword structure in Lr is flat and hierarchal keyboarding is not supported. If you use hierarchal Keywords in LrC, then be prepared for the incompatibilities.
• I use Lightroom on my iPad Pro as a replacement for LrC on a laptop.
• I use my iPadPro as a front end to import images in the field (or down stairs from my upstairs iMac) When I have a good internet connection in the field, my images are almost always imported into LrC by the time I return.
• I chose to sync only my most important LrC images to the cloud (about 7000) instead of my whole LrC inventory.

 

[Jim Wilde]

@Jim Wilde ... is there a link somewhere to this "fully integrated hybrid classic and cloud" workflow that you use?

I'm afraid not....it's an undocumented and unsupported way of working. The basics of getting original images into the cloud instead of smart previews from Classic are documented in the Sync section of your Classic book. Conventional wisdom says either import only into Classic and sync smart previews, or import only into Lightroom and allow them to sync down to populate the Classic catalog. The latter method is OK if you particularly want originals in the cloud, but it's a major issue if you do add a lot of metadata to the images, especially the things that don't sync. I have always added keywords and location data to every image I keep, and that's a problem as neither of those things sync between Lightroom and Classic.

I did want originals in the cloud, but I'm not yet ready to abandon Classic, and I didn't want to be applying essentially the same metadata in both Classic and Lightroom, so the only solution that I could come up with requires me to import only into Classic and optionally sync smart previews if I want to edit and/or apply some of the metadata that does sync (Titles, Captions, etc.) on either platform. All non-syncing metadata is added only in Classic, using my hierarchical keyword structure, When I've completed a batch of images (and this workflow now makes me fully complete processing of a batch, rather than have bits of the workflow hanging around for months), I convert the raw files to DNG, save metadata to the files, then remove them (but obviously not delete from disk) from Classic. When the removal has then synced to the cloud, I import them into Lightroom, which picks up all the metadata (with flattened keywords, but that's not an issue for me) and edits, and those opriginal files are then automatically synced back down into Classic. Because the metadata has been embedded in the DNG, Classic reads it and adds the keywords back into the keyword list in the correct place in the hierarchy. Same with Location data, that's also populated in the Metadata Panel from the embedded XMP. Even colour labels, if they were applied to the file at the time that the XMP was embedded, are restored in Classic.

It works reasonably well for my needs, though I would never expect most people to want to follow it. It's also a bit of a pain when you need to change something in the keyword/location data fields afterwards. It's doable in various ways, but can be a time sink so a busy pro would never countenance such a workflow. I'm a hobbyist, so I can afford the time to occasionally rework metadata changes to both sides of the cloud, but I don't always enjoy it.

 

[Johan Elzenga]

Original files synced UP to the cloud do not count against the storage plan limits

I don’t think that is true. Smart previews do not count, but originals do count.

 

[Linwood Ferguson]

Adobe has adequate protections in place to preserve any user data in the cloud.

Why do you feel that to be true?

Can I suggest you try an experiment: Produce and edit an image in the cloud. A week later delete it. Later (however long the recycle bin lasts, if there is one) ask Adobe to restore the file for you. Will they?

Also, Adobe has been hacked before.

And let's not forget this from a few months ago, with Adobe basically saying "we lost them and cannot get them back":

Lightroom App Update Wipes Users' Photos and Presets, Adobe Says they are 'Not Recoverable'

Personally I would feel a lot more reassured if Adobe would publish details of what they do to protect your photos, but from what I have read mostly they just say "trust us".

 

[clee01l]

I don’t think that is true. Smart previews do not count, but originals do count.

True, because originals from LrC are sync’d only as Smart DNGs


Sent from my iPad using Tapatalk

 

[clee01l]

Why do you feel that to be true?

Can I suggest you try an experiment: Produce and edit an image in the cloud. A week later delete it. Later (however long the recycle bin lasts, if there is one) ask Adobe to restore the file for you. Will they?

Also, Adobe has been hacked before.

And let's not forget this from a few months ago, with Adobe basically saying "we lost them and cannot get them back":

Lightroom App Update Wipes Users' Photos and Presets, Adobe Says they are 'Not Recoverable'

Personally I would feel a lot more reassured if Adobe would publish details of what they do to protect your photos, but from what I have read mostly they just say "trust us".

I see your line of reasoning, and perhaps I should amend with some caveats. What I meant was that Adobe have sufficient redundancy in place to preserve your files. The problem is stupid user mistakes. You are only covered for 60 days against user deletions. The problem with Lightroom wiping user data was not cloud related as the users having the issue were not syncing in the cloud. and the data was wiped locally because there was not sync in the cloud.


Sent from my iPad using Tapatalk

 

[Linwood Ferguson]

The problem with Lightroom wiping user data was not cloud related as the users having the issue were not syncing in the cloud. and the data was wiped locally because there was not sync in the cloud.

True, though the loss was also caused by Adobe programming not user error per se. Users who lose unsync'd images because of unexpected hardware failure or loss is one thing, but to have them erased by Adobe programming before they sync clouds the ability to call it user error (pun intended).

Also, while not Adobe, bear in mind that Canon users shortly before lost their data despite being sync'd to the cloud:

Canon says its cloud service can't restore users' lost videos or full-size images

Personally I think users have generally failed in their responsibility to hold cloud vendors accountable. We (and by that I mean massively large companies and governments also) just dump our data there with no real accountability, and cross our fingers because they say "trust us". I remain astounded, but largely have given up playing chicken little (well, except occasionally, like this morning).

 

[hrboyce]

I know this is a Lightroom forum but I am interested in learning where anyone would go with their backed up RAW+XMP files if they decide Adobe was no longer the way for them? Part of any backup strategy has to be how will I use the backups if/when needed. I went on that quest before signing up for the Photography plan and I found it disturbing how difficult it would be, if it was possible at all, to import my hours of work into another application.

I resigned myself to assuming I would need to export them as JPGs with as much detail as possible or start over if the result wasn't what I wanted at the time. I actually keep a ★★★★★ folder with albums of duplicate images I want to make sure I preserve as they are today without having to cull thru them all. Unfortunately it doesn't help with the LR Downloader as it pays no attention to your organization and drops all the downloads into a date based folder structure. And yes I realize that I am "wasting" expensive cloud storage but I've decided to kick that can down the road until I get a lot closer to needing more space and I am hoping that over time as alternatives become available Adobe will be forced to properly price their cloud services.