• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Practical Backup Solutions

Status
Not open for further replies.

Michael Naylor

Active Member
Joined
Nov 21, 2016
Messages
145
Location
Spain
Lightroom Experience
Intermediate
Lightroom Version
Classic
Given my ISP doesn’t yet offer fibre in my area and having accepted I can’t afford to buy several fault-tolerant server farms connected by private high speed data links to several different locations, what are my options?

How about three directly connected 10TB drives in separate enclosures or maybe a few raid 6 NAS servers in different parts of the house? Whatever it is, it could be quite expensive, considering the number of photos and videos I’ve produced over the years.

I read many comments on this forum advocating the use of a good backup system, but little of what exactly this might be. I’m interested to hear what these advocates are using, preferably before I accidentally erase any more disks.
 
I doubt that you require 3 10TB EHDs and certainly, any RAID is overkill. Here's what I have in a "belt & suspenders" type of setup.
  • TimeMachine — Backs up to alternately to a AirPort TimeCapsule (NAS) and a locally attached EHD.
  • CrashPlan — Simultaneous backup to a locally attached EHD and Code42's Cloud Service.
Both the TimeMachine and CrashPlan apps are free. The EHDs are 8TB & 3TB respectively. A 3TB TimeCapsule is ~$400USD CrashPlan Cloud Backup is ~$60USD/yr. My internet is throttled by my service plan and it took me about a month to initially back everything (3 computers) up CrashPlan's Cloud but there is no noticeable drawn down on bandwidth for the incremental backups every 30 min. Because I back up two Macs to the Time Capsule, I have a additional EHD attached to the Airport Time capsule. Internally, everything backsup via WiFi to my router. So, fibre is not a requirement.
 
Last edited:
There are a few aspects to backup.
First is to protect against loss of the building, this is what offsite backup is for. So you will see people talk about backing up to the cloud. The old school solution is to backup to some media and ship it offsite. How far and how often you ship it offsite is a risk management call. In your case, taking the backup media to a friends house or a barn located a mile away would likely cover the vast majority of disaster situations.
Second is how the backup process works. With Apple, you have Time Machine built in which is a rather good solution. So if you have enough attached storage space, time machine takes care of versioning and all tracking all the changes over time and avoids the need for many other strategies.
 
Hi Mike,

This is a good question. I use a number of backup strategies.

A local backup provides a frequent and quick backup. As a Mac user simply use TimeMachine. May target backup disk is a Synology NAS with 5 2GB drives. This gives me 5.9TB backup volume with a hot spare. I have about 2TB of system and user data including all my photos. The TimeMachine backup is currently taking about 4TB.

A remote backup gives you disaster recovery for catastrophic problems such as fire, floods etc.. For this I use CrashPlan. I only backup my user data including photos and not any system or application data as that can be reconstructed faster and easier from other sources.

Additionally I keep at least one bootable clone of my system hard drive. I will usually update this just before any major system software upgrades. And then about once a week. For this I use Carbon Copy Cloner. It is configured to run any time I turn on my backup system drive and to remind me once a week.

Finally as protection from a HD failure I have my User data on a pair of 3TB mirrored internal drives in my MacPro.

In practice this is pretty effortless to maintain. TimeMachine and CrashPlan are fully automated and I just have to pay attention to make sure they are running. Both are good at notifying you of any problems.

About a year ago my system SSD failed completely. By simply booting from my cloned HD and running TimeMachine restore I was back up and running in about 1 hour. While you could also do a full restore from TimeMachine to a new hard drive that would take quite a bit longer.

I have also configured my system so that my home directory is on a separate drive from my system data. This makes it much easier to boot from a backup system and keep running. If your system has space for an additional internal hard drive I would recommend implementing considering this. It can be done using the Advanced Options for your user in the Users & Groups panel. Proceed carefully if you go this route.

-louie
 
Mike from your previous thread am I correct in thinking that your NAS is your primary storage; i.e. you have no images on your laptop/desktop? In that case RAID may be useful for performance (depending on RAID level) and disc redundancy on the NAS. For back-ups of the NAS itself then RAID is, pardon the pun, redundant.
 
Mike from your previous thread am I correct in thinking that your NAS is your primary storage; i.e. you have no images on your laptop/desktop? In that case RAID may be useful for performance (depending on RAID level) and disc redundancy on the NAS. For back-ups of the NAS itself then RAID is, pardon the pun, redundant.
Until I embarked on changing things and screwed up badly, my configuration was as follows…

1) An iMac with internal 3TB Fusion drive, for current work inc. all photos.

2) A 3TB Time Capsule backing up the iMac.

3) A Carbon Copy Clone of the iMac to an external 4TB G-Tech.

4) An old USB2 8TB Raid 5 box for archiving (mainly videos) that were too large to have on the iMac.

Obviously, the weakest link was the 8TB Raid with no backup at all, although it could survive a single disk failure. So, I began to to develop a half baked plan that was to have 2 external raid 5 boxes (OWC Thunderbolt 4), one to move the existing 2TB drives to and the other to house 4 new 4TB drives. Being cautious (or so I thought), I bought the first OWC, popped in an old 1TB to test it, and accidentally erased the 8TB Raid instead of the old 1TB drive. I have the EaseUS Recovery Wizard software running as I type!

This unforeseen circumstance has given time to reconsider how I might utilise a second OWC, what size drives it should contain, and how the drives should be configured. N.B. these OWC enclosures can be configured with SoftRaid to whatever one can dream of. Lastly, I figured have 2 identical enclosures would offer the ability of swapping the drives over in the event one enclosure fails.

Overall, this would have ended up like this…

1) The iMac with internal 3TB Fusion drive, but with no critical user data.

2) The 3TB Time Capsule backing up the iMac.

3) An OWC 8TB Raid 5 with at least 2 partitions, one being for photos.

4) An OWC 16TB in either raid 5 with multiple partitions, 2 raid 1 sets, or 4 single disks. Each of these being used for different Carbon Copy Cloner backups of the iMac and the 8TB raid partitions.

5) The 4TB G-Tech would now be free to be used away from home.

That was the plan, but now I’m open for other ideas.
 
RAID is not backup. Raid is for data redundancy and 24X7 data availability. You do not need either. You need version control which is something that can only be had by a good Backup app like TimeMachine.

Do you have any single disk large enough to backup up all of your critical data? Note that your critical data is not ALL of your data. Your critical data consists of your user settings, documents and photos.
For your purposes, RAID is wasted disk space unless you have a duplicate RAID to back up the first. It can't survive a RAID controller failure. JBOD and RAID-0 are poor excuses for back up media. Although they can be useful for storing non critical data ordata that is backed up to another device.

With a disk that has enough capacity, TimeMachine can backup multiple volumes. If you have user data on one volume and photos on another, you should be taking advantage of this.
I have a 3TB TimeCapsule that has a 5TB EHD attached. The 5TB EHD is used by Time Machine to backup my iMac primary volume and one iMac EHD. The 3TB drive is used by my MBP for TimeMachine backup. So I back up all of my critical user data from two computers to the TimeCapsule (NAS). My other iMac backup is an 8TB EHD that receives the same multi volume critical data that is backed up to the 5TB drive.

There is no purpose served by splitting disks into multiple partitions. Modern Filesystems can manage single partitions at least as large as 10TB (Actually for HFS+, it is just under 8 exabytes (EB)). While you can span partitions across multiple drives, the result is only good until one of the drives fails. So your backup volume limit is only as large as your larges single volume. Judging from the information provided, this for you is 2 or 3 TBs If you have total critical data that is larger than 3TBs, you need larger single volumes than what you currently have for adequate backup.
 
RAID is not backup. Raid is for data redundancy and 24X7 data availability. You do not need either. You need version control which is something that can only be had by a good Backup app like TimeMachine.
TimeMachine is great when its working, but twice now its bombed out with an Invalid Data message telling me to reformat and start again. I believe this happens when it starts pruning.
 
A couple of observations (and just my humble two penn'th) on your planned set-up, some of which just reiterates things people have already said.

The general 'to RAID or not' argument. Since you have a RAID box and the disks to go with it I would continue to use it for disk redundancy since it contributes to your system's resilience. With RAID 5 your read performance will be fast but write performance somewhat slower. But I would not duplicate the RAID for back-up. If either the primary RAID fails completely, or the back-up disk fails, you still have all of your data. But if you chose to ditch the RAID and redistribute your existing disks that's fine. If you were starting from scratch I would not say go and buy a RAID 5. That decision is far less critical than getting that primary storage backed up in the first place.

Your point 2). As I understand things this is adequate protection for your system. But there should ideally be a copy off-site too.
Your point 3). Since you already have this box it will maintain disk redundancy for what will be your primary data store.
Your point 4). I would not bother with RAID for your back-up copies. It over-complicates things. Keep a disk or two. If your back-up disk fails just switch it out; no need to rebuild a back-up array.

As people have said, having off-site back-up is important in a comprehensive plan. If your internet connection isn't that great that can be an issue. It depends how and what you back up across t'net. If you're trying to regularly back up an entire partition to the cloud it's not going to be fun. If you're uploading just your data files incrementally, like Cletus and Louie, then after the initial overhead it can get easier; but it really depends on both your connection speed and the amount of data that that you change on a daily basis.

Or an off-site back-up might just be an an outbuilding. In which case, depending on your data and your physical security you might want to consider an encrypted back-up.

It's all about risk at the end of the day. For me, the risk of needing an off-site back-up is probably less than that of something going awry at home. But either could happen.

Some people have a good plan in place but have never tested it. They assume that because they have X-software or Y-recovery plan that they are covered. But it's not until disaster strikes that you find out how good your plan is. If you can, test it. If you're not confident in doing that think about whether that's because you're not confident in your plan; in which case revisit it.

Louie had a disaster and had multiple options to hand; another thing to consider and I guess that's why you may be keen on keeping RAID. One of your multiple options might, for example, using Carbon Copy to periodically back up your system and/or data files to one of your spare disks in addition to your 'normal' back-ups.

Back-up procedures shouldn't be a chore. The chore is setting them up in the first place, which is why many don't bother. But when they're set up correctly and automatically (which is even easier in a desktop situation) they should take care of everything for you.
 
Paul makes some excellent points and perhaps the most glaring (and most often ignored) is to TEST your backups, go make sure it does what you think it does, that the files are there, that they are not corrupt, that they are complete (if nothing else take some directory counts, but a good comparison program to do byte by byte comparisons is not a bad idea).

I would like to comment on the Raid and backup issue a bit though also. Good backup processes are versioned - i.e. you have point in time restoration capabilities. There are a number of ways to do that. Some do it by keeping them versions all in one place, suitably separated and labeled (virtually usually); some people do it manually by keeping a rotation of backup copies, so they might have a few EHD's and back up to them. Some do it by keeping "originals" separate from edits and figure they could always re-edit (yes, you can - what if it's for 10,000 edits though?).

For people rotating through copies, Raid is fairly pointless for the backup copies themselves - more copies is better than raided copies.

For the (much better!) option of a true versioned backup, if you have only one or two such repositories, I would suggest raid be considered. In that case ALL (maybe except off site) of your copies are actually in one place, that can fail. And can fail silently, so you might not know it has failed until you try to restore and -- gone.

I personally use a versioned main backup with zfs raid copies. zfs (and similar alternatives) offer a raid-like function that also includes built in corruption checks.

My point though is that a real, comprehensive backup plan must account for the probability that at some point backup media will fail. Steps to mitigate that include testing it more frequently, making more physically separate copies, and redundancy and consistency checks such as raid and zfs, or better yet all of the above.

Thinking "what are the odds that my backup will fail when I need it" is not the right question -- the question is "did my backups fail already and I never noticed and will only find out when I need to restore". Then it is a disaster.

If you read about people who truly get stuck, a few are ones who admit "I never backed it up", but many, many say things like "I went to restore from backup and it would not work". Avoid being that person.
 
comprehensive backup plan must account for the probability that at some point backup media will fail. Steps to mitigate that include testing it more frequently, making more physically separate copies,
A point that I couldn't agree with more!
In another post on another thread, I just described the 4X redundancy of my backup plan: TimeMachine X2 (Time Capsule and local EHD) and CrashPlan X2 (Local EHD and Cloud service).
I used a TimeMachine backup when I replaced my crashed primary Drive with a new one. I don't expect to ever us the CrashPlan cloud backup to recover, but it is there in the event my home is destroyed by fire, flood or pestilence.
 
TimeMachine is great when its working, but twice now its bombed out with an Invalid Data message telling me to reformat and start again. I believe this happens when it starts pruning.
I've had that happen a few times. It's typically a problem with the Time Machine volume's disk directory, which means it's usually possible to recover those using a disk utility. For a particularly difficult one I had to run DiskWarrior on it. It's kind of a pain to do even for advanced users, because you have to do things like make sure you have the right disk image mounted and reset the invalid data flag. I go to the trouble because I don't want to lose all that version history.

Due to the size and complexity of a large Time Machine backup with lots of versions, recovery can take a long time. It seems that the way Time Machine does things tends to overwhelm the aging Hierarchical File System (HFS). Some say that Time Machine backups should become less unwieldy and more reliable and streamlined to work with once Apple transitions to the new Apple File System (APFS) in 2017:
Snapshots and clones both are going to be available in APFS. Snapshots let you throw off a read-only instance of a file system at any given point in time; as the file system’s state diverges away from the snapshot, the changed blocks are saved as part of the snapshot. This is similar in concept to Microsoft’s shadow copies, and it’s an incredibly handy feature. There are obviously huge implications here in how Time Machine works—a true file system set of snapshots could totally replace the kludgy and aging mechanism of hard links that Time Machine builds and maintains.
 
OK, I'm beginning to get the message (3 backups are better than none), but the recovery of my erased raid-5 box is now underway (see thread Stupid Me!).

Revised thinking is to redeploy the existing 4 x 2TB drives from the old USB2 Raid box to the new OWC ThunderBay 4 box and to continue using it in single volume raid-5, BUT, not rely on it being my only archive device. I do need large and fast raid-5 for video editing and the 6TB usable space will be perfect for my particular needs.

For backups, I'm considering a 5 drive NAS box in raid-6. The thinking here is that when fibre eventually reaches my house, I'll be able to replicate this to a cloud. The NAS will need a minimum of 12TB of usable storage, but I know very little about NAS. The extra features, such as being able to serve media to my Apple TVs sounds enticing.

Deciding which NAS box to buy will be very difficult. There are just so many to choose from.
 
Last edited by a moderator:
For backups, I'm considering a 5 drive NAS box in raid-6
Why a NAS? Is this NAS going to be co-located with your computer, In another building or? Presumably you have a Mac with Thunderbolt ports You can not beat the TB for I/O speed. NAS is going to involve Ethernet Even Gigabit Ethernet is going to be slower than a locally attached RAID box. Thunderbolt3 drive enclosures are becoming reasonably priced and to me may be a better decision.
 
Why a NAS? Is this NAS going to be co-located with your computer, In another building or? Presumably you have a Mac with Thunderbolt ports You can not beat the TB for I/O speed. NAS is going to involve Ethernet Even Gigabit Ethernet is going to be slower than a locally attached RAID box. Thunderbolt3 drive enclosures are becoming reasonably priced and to me may be a better decision.
NAS seems to be the where things are going. Granted it isn't very fast, but for incremental backups it should be fine. It does offer other advantages. The ThunderBay 4 raid box has thunderbolt 2 and that would be directly connect to the iMac for editing. As for TB3, my Late 2015 iMac only has TB2 and I don't expect to replace that for quite some time.
 
NAS seems to be the where things are going. Granted it isn't very fast, but for incremental backups it should be fine. It does offer other advantages. The ThunderBay 4 raid box has thunderbolt 2 and that would be directly connect to the iMac for editing. As for TB3, my Late 2015 iMac only has TB2 and I don't expect to replace that for quite some time.
A NAS is useful for files that are going to be accessed by several computers. That is the only advantage that I see offered by a NAS. You buy a TB3 device now because you will grow into it. I am doing this even though my current iMac is only TB2
The ThunderBay 4 is (as far as I can tell) the same box rebranded as the Akitio Thunder Quad) It is cheaper to buy the Akitio Quad TB3 from Amazon than the TB2 version from OWC
 
A NAS is useful for files that are going to be accessed by several computers. That is the only advantage that I see offered by a NAS. You buy a TB3 device now because you will grow into it. I am doing this even though my current iMac is only TB2
The ThunderBay 4 is (as far as I can tell) the same box rebranded as the Akitio Thunder Quad) It is cheaper to buy the Akitio Quad TB3 from Amazon than the TB2 version from OWC
I did consider the TB3 options, but then you need to add an adapter to connect to a TB2 computer. The only one I could find is made by Apple and there are too many conflicting reports of whether it will work this way round. It could be unreliable. Its primarily sold to connect an "old" TB2 device to a the new TB3 computer.
 
Its primarily sold to connect an "old" TB2 device to a the new TB3 computer.
TB3 & USB-C are bi-directional. TB devices are daisy chained. So, it does not matter which way the connection is made HOWEVER, Any TB2 device needs to be at the end of the TB3 Daisy chain as it will be the slowest component. A TB2 computer is at the end and therefore you won't get any faster than the TB2 speeds (20GBps) of the computer.
 
You buy a TB3 device now because you will grow into it. I am doing this even though my current iMac is only TB2
The ThunderBay 4 is (as far as I can tell) the same box rebranded as the Akitio Thunder Quad) It is cheaper to buy the Akitio Quad TB3 from Amazon than the TB2 version from OWC
I decided to stick with TB2. The specification of the Akitio Thunder3 Quad states that Mac is “Not supported”. They also say this on Amazon. Akitio also state connecting a TB2 computer to a TB3 device via their T3T Adapter is “Not supported”. TB3 backward compatibility requires an adapter. To date, I don’t think anyone has made one that actually works.
 
Please don't laugh, but apparently tape is coming back. This is an LTFS Thunderbolt tape backup/archive drive from mLogic. The drive is not cheap, but the tapes are and you can drag & drop files around, just like a hard disk. They say the tapes will keep your data safe for 30 years.
mTape_and_Media_1024x1024.jpg
 
Laugh? How could you not!

But no, I'd really rather pass on that one ;)
 
Please don't laugh, but apparently tape is coming back. This is an LTFS Thunderbolt tape backup/archive drive from mLogic. The drive is not cheap, but the tapes are and you can drag & drop files around, just like a hard disk. They say the tapes will keep your data safe for 30 years.
mTape_and_Media_1024x1024.jpg

Well, at least it's a far cry from when I was using tape with a Commodore PET, then a Commodore 64.. Until 5 1/4" floppy disk drives with their ~184KB(IIRC) started becoming all the rage. :cool2:

... I don't remember the capacity of those tapes, though.
 
I don't think that tape ever went away. It is still the main backup media large enterprise systems. Current technology allows for up to 8TB on a single tape. One big advantage of tape is that is takes 0 energy to archive large quantities of data.

Practically speaking, however, it probably does not scale well to individual needs.
 
There are fantastic tape based backup systems from companies like IBM for large installations. I used Tivoli IBM Storage Manager for a large enterprise level mainframe data centre backup system. You could load a bunch of tapes into a physical library, design your backup strategy and Tivoli did everything, including telling you what tapes to move to your offsite storage. Tape backup handling was once a labour intensive activity in data center operations.
 
Status
Not open for further replies.
Back
Top