Yes another question about external back ups!!! Use TM, RAID, still don't get it.

[tamarakk_1]

Lightroom Version Number

Creative Cloud, since I don't understand it I stick with Classic and don't sync it, since I did some on CC Lightroom and also on Classic

Operating System

1. macOS 10.14 Mojave

I have a RAID system, and keep most images on separate HDs that are NOT part of the RAID. (I am not a techie, someone told me to buy and use it as my invisible backup.) I also import to a second external HD on import, but I have no understanding if those would have all my LR edits and organizations.
I have LR Classic set that it keeps the sidecar files next to the CR files. I do back up catalogs regularly also. I use Time Machine also, set to back up my externals also. But feel like all this is a disorganized attempt which may or may not be right.
So what happens if the individual HDs fail, which are the primary locations of my images when I look at my LR main screen? Are my edits andLR organization still intact somewhere? I would hate to have 80k images with no LR organization and edits to comb through!

 

[Linwood Ferguson]

Someone smarter about time machine should answer, but the short version is this:

Your catalog is a database which points to your images, and contains records of your edits, changes, metadata, etc.

Your images are stored in folders, and can be in many folders scattered hither and yon, depending on where you put them. These are the only copy of the image originals stored anywhere; they are not in the catalog (ok, people say they are "in" the catalog meaning the catalog points to them, but they do not reside inside the catalog).

Your edits MAY or may not be written to sidecar files depending on preference settings. And not all edits and information lots of stuff (collections, virtual copies, some metdata) is stored only in the catalog. While some like to think of the sidecar files as a last ditch backup, you are better off treating them as not.

You must back up the catalog and the images.

The built in LR Catalog backup is NOT really a backup. At least by default it makes its backup copy on the same drive as the catalog, which means if you lose it, you lose your backups and original. The purpose of the LR catalog backup is to restore the catalog if you do something bad that corrupts the backup but the disk itself is fine. It is a decent idea to do these occasionally, especially the integrity check, but do not depend on them as backups. It ONLY makes a copy of the catalog, not the images.

You must back up the catalog and all the images (worth saying twice). If Time Machine is doing that, great. If not, make it happen.

Forget raid as a backup. Raid is nice, I'm not knocking it, don't think of it as a backup. We could do a long discussion on what raid is and isn't, but do not think of it as a backup.

Two other suggestions:

- Make sure your time machine backup goes to a completely separate disk, not a partition, completely and totally separate.
- If you can, keep the time machine backup offline when not backing up; this prevents malware that may infest your computer from grabbing your backup at the same time.

Ok, a third suggestion:

- If time machine is your only backup, consider a separate, potentially off-site backup, something like Backblaze perhaps; there are many, many. This protects against fire, flood, buglary, etc.

 

[tamarakk_1]

Thanks so much, I do understand much of what you said about LR just being a pointer. Though not that RAID is not really a back up! Yikes freinds told me to buy all that (not cheap) but I have heard it isn't ideal...though I don't know why? Backblaze says it would take 9 months to back up all my images since I have 100k. That isn't reasonable. I do keep separate HDs in a bank deposit box (lived in California with fires and earthquakes too long!). But I don't really know what that means. I do a TM backup of my computer including my external hard drives, but still don't know what that means in real world if I needed it. Should I change my settings (somehow) so the edit files are not sidecar? I guess I just want to be sure that all my LR organization, edits, tags, etc, and of course the original RAW files along with them are not a mess if I need to use a back up. I wish LR had an easy way to say "back up all files, catalogs, edits, and externals with originals, etc" on to HD X, and it would copy everything. I have asked friends and no one has an easy way.

 

[Linwood Ferguson]

Lots in there to unpack:

RAID: Raid is about redundancy and reliability. Done well, raid can go a long way to prevent data loss due to hardware failures of the drive media. It does not protect against software bugs, malware, user errors, operating system errors, bit rot ("silent" hardware errors), and the like.

As a simple case to show why it is not a backup, consider any of: LR runs amok and trashes some images, or the catalog; you make a mistake and delete 1000 images instead of 10; some other unrelated program runs amok and trashes a bunch of files; your system is infected with malware.

In all those cases, what Raid does is instantly and reliably makes a copy of all the bad stuff; so you have two copies of garbage.

I have no data to back this statement up, but I believe drives have gotten so reliable that today most data loss is NOT caused by a physical read failure of disk media (which is primarily what Raid is for). It is the "everything else", and raid is pointless for those.

Again, I like raid, I use raid -- but it's not backup.

Backup software should include a lot of features, but a key one is it should provide point-in-time restore, so you can roll back to your catalog as of yesterday (e.g. malware took over today), or even 30 days ago if you find a corruption issue that far back. Raid does not do that.

Sidecars: It's possible they slow things down a bit, but they do provide some level of redundancy. For example, if you accidentally remove a file from the catalog (but not delete it) and have the file and (up to date) sidecar, you can add it back and the edits come back. But honestly for most people who struggle to keep up with all the details of lightroom it's better to not use them. Or use them but forget about them.

LR doesn't have a "backup everything". Why? Don't know, but I can speculate: your computer has a LOT of stuff that do not have "backup everything". You are supposed to back up your computer. If you back up your computer (and associated EHD's) you also are backing up LR. And Quicken or Taxes or your recipies or whatever other software you use. While LR is important, it's not a "LR Computer" it is a computer with all sorts of stuff you need to back up. Don't treat LR differently -- backup everything. My two cents; others may disagree.

There are many programs that will do that - general backup programs. Cloudberry Desktop Backup is a good example, it supports backing up to local drives, NAS, and most cloud services. There are others.

9 Months to backup: ok, maybe, depends on your internet speed. Some services allow you to do a jump start by copying to a physical drive and shipping it. If you are only backing up changes it may be more manageable. Or maybe not. Depends on your backup speed.

I probably missed something, but that's a shot at unpacking...

 

[tamarakk_1]

That was super clear and helpful thanks. I will need to ponder it though. I might have OK backups. For example, when I import from a card, I always import to two EHD at the same time, lets say named "Africa" and "Africa Backup". When I edit and tag an image in LR, LR shows the image as residing in EHD "Africa". So if EHD "Africa" died, then what? Could I plug in Africa Backup and all would be well?

 

[clee01l]

My 2 Cents on RAID and I'll shut up. RAID is for shops that need 7X24 uptime for data, It is overkill for the home user.
A system backup app provides backup and version recovery for all critical users data, The Lightroom catalog and the images files them selves are the critical data. Time Machine will do a nice job as your only system back. uo for your Primary disk and any other germanely attached volumes. The backup volume need to be large enough to hod backup for all of the critical data on all of the other volumes.
Sidecar files contain only part of the metadata stored in the Lightroom catalog. Some people thing this is additional insurance, The real insurance is multiple backups of the catalog file that contain ALL of the metadata collections and publish service . Side car files are a wasted overhead as long as you make a good attempt at keeping backups to you Lightroom catalog.

If data volume "Africa" fails, you get a new volume to replace it and restore that volume contents from your TimeMachine restore process.

Even backup volumes fail. I alternate Time Machine between two backup volumes. If one fails, then I am never more than 30 minutes out on the other backup. Was an additional precaution I also use Acronis to make a third local backup to another volume.

 

[tamarakk_1]

CLee01l Thanks. I have the RAID now so I guess I will continue to use it. Your remarks on sidecar files makes me think perhaps I should change that setting? That would make them part of the main file from what I understand. Any reason not to change that setting (if I can find it?)
Other question, dumb I am sure, is a total Time Machine backup backs up the catalogs, right? I keep hearing to be sure to back up the catalogs, and I do that on exit from LR weekly, or more often if I have done a lot of work. Then when I plug in Time Machine (once month maybe or more if I have done lots of work). Also, why shouldn't I have the RAID doing a TM backup? And if I use TM to back up my EHDs also, wouldn't that make the Raid more useful, though for the reasons mentioned by Furguson, though still could be corrupted by virus? Thanks I am looking in to Acronis. Wow I wish I had used this forum ages ago. Thanks SO much. I try to give back too, but can't on this topic since I don't know much!

 

[davidedric]

My 2 cents on RAID 1

No, it's not a back up. But, when one of your mirrored, RAID, disks fails you simply carry on - no intervention required. Then, replace the failed disk at your leisure (how long depends on how lucky you feel), and let the mirror rebuild.

At least, when one of mine failed I was relieved it was mirrored!

 

[clee01l]

CLee01l Thanks. I have the RAID now so I guess I will continue to use it. Your remarks on sidecar files makes me think perhaps I should change that setting? That would make them part of the main file from what I understand. Any reason not to change that setting (if I can find it?)
Other question, dumb I am sure, is a total Time Machine backup backs up the catalogs, right? I keep hearing to be sure to back up the catalogs, and I do that on exit from LR weekly, or more often if I have done a lot of work. Then when I plug in Time Machine (once month maybe or more if I have done lots of work). Also, why shouldn't I have the RAID doing a TM backup? And if I use TM to back up my EHDs also, wouldn't that make the Raid more useful, though for the reasons mentioned by Furguson, though still could be corrupted by virus? Thanks I am looking in to Acronis. Wow I wish I had used this forum ages ago. Thanks SO much. I try to give back too, but can't on this topic since I don't know much!

The sidecar file are created for proprietary RAW files only otherwise the same data is written into the XMP section of the file header for DNG, JPEG etc. Turning it off in Lightroom will mean that all of your metadata will continue to be collected in the catalog file Only a portion of that metadata ever was duplicated in the sidecar/XMP section. By not writing to XMP, there will be a minor performance boost (work that Lightroom Classic does not need to do) but probably so little that you won’t detect a performance boost.
I’m not sure that TimeMachine will let you write to a RAID 0 or RAID 1 device. Because of proprietary file systems that are often used in RAID devices, The other thing is if you have a proprietary RAID filesystem to are dependent upon the 1 RAID controller. If that piece of hardware fails, then non of your RAID disks are available until you replace the RAID controller with an identical one from the same RAID manufacturer Ask me how I know this ;-).

Tome Machine backs up the folders that you designate. They can be on many different volumes though by default TimeMachine does not automatically backup attached EHDs
The reason that you wants to make a system backup of the zipped file copies that LR makes on exit is that is the only time the catalog file is closed and stable. Many backup software will back up the active open catalog file and these backups can be unstable. The only stable catalog file is the zipped copy that LR makes on exit.

TimeMachine should be running constantly in the background. It works in conjunction with MacOS to take advantage of idle CPU c Yale’s to do a backup every thirty minutes If you designate multiple TM backup disks, Time machine will rotate through each in turn creating a fresh version on each.
I use Acronis in Addition to TimeMachine because I am a “belt and suspenders” type person. I have at any time 3 backup disks that I can use to recover my critical user data. Ideally I would have a backup in an offsite location for the risk of Fire, Flood or other pestilence. I’ve already survived flood (Harvey) without loss of any of my critical user data.


Sent from my iPad using Tapatalk

 

[Roelof Moorlag]

Backblaze says it would take 9 months to back up all my images since I have 100k. That isn't reasonable

I did it (with Crashplan) and i'm very happy with it. It is one less wory to have the offsite backup realised.

 

[DCBolton]

My 2 cents (or maybe 5 cents) on RAID. The discussion here uses the RAID term as if it is only one type. Most of the references seem to be to RAID 1 (mirrored type). Other RAID types (5, 6 10) are more useful for storing large volumes of data like 100k images.

I use RAID 1, RAID 5, mirrored RAID 5 (e.g., RAID 10) at work and at home. Each has its own appropriate application. RAID 1 saves data to (usually) 2 identical drives. Thus, 2 x 4TB drives gives you 4TB of useful storage. RAID 5 stores data in 'stripes' across multiple drives (usually 4 or more) with a parity bit for every 2 bits. I won't delve into parity and XOR calculations but the parity bit allows the data to be rebuilt if a drive fails. In RAID 5, 4 x 4TB drives will give you 12TB of storage as opposed to 8TB in RAID 1. The data redundancy allows 1 drive to fail (1 of 4), while RAID 1 would allow 2 (1 of 2 and 1 of 2).

At work, two RAID 5 drives (4 × 3TB = ~8TB storage) are mirrored as RAID 10. In this case, 1 drive failure on both RAID 5 looses no data. 2 drives can fail on one RAID and 1 on the other and no data is lost. The failed drives are hot swappable and the RAID data array can be rebuilt with no down time. HOWEVER, rebuilding an 8TB RAID 5 array takes ~48 hours so it is crucial that no other drives fail during the rebuild.

RAID is a hardware configuration. It's important not to confuse the HW configuration with how it is used. For example, at home I have 2 RAID 5 drives and 1 RAID 1. One RAID 5 stores my LR images. It is backed up on the RAID 1. The other RAID 5 stores media files (as media server) and is the backup target for my desktop computer via Win 10. That RAID 5 is backed up on 2 different USB drives that alternate. The RAIDs are drives; they are not magic.

Finally, all storage media fail. It is only a matter of time. I keep an Excel spreadsheet on all HDDs including the date of manufacture as well as the time in service. Different manufacturers drives and types have different service lives but all will fail. I have old IBM drives at work that are running after 15 years and Seagate drives that failed catastrophically after 1 year. Be ready to replace drives as they approach the end of warranty. RAID gives you flexibility to wait until failure occurs without suffering the downtime of restoring from backup, which is more like 1 week for 8TB.

YMMV. Best wishes.

Sent from my SM-N960U using Tapatalk

 

[Roelof Moorlag]

And there is Raid 0 also, not adding faulttolerance but for performancereason only

 

[clee01l]

My 2 cents (or maybe 5 cents) on RAID. The discussion here uses the RAID term as if it is only one type. Most of the references seem to be to RAID 1 (mirrored type). Other RAID types (5, 6 10) are more useful for storing large volumes of data like 100k images. ...
RAID is a hardware configuration.

Your description of RAID is excellent. But you overlook the single point of failure in any RAID. configuration. The hardware component and the use of proprietary file systems. When your RAID controller hardware fails you still have 2 or more perfectly good disc drives. But with out that failed piece of hardware or one identical to it, it is not possible to read any of the data on the proprietary file system disks.

Also, remember, the issue is not about data integrity but about data recovery. Delete a critical file, write over a good version with a version in error and all you have done is replicate that mistake through out your RAID system.

Commercial ventures that need 7x24 up time for their users, use RAID for this reason - uptime. They also use redundant system backup for data recovery.


Sent from my iPad using Tapatalk

 

[Linwood Ferguson]

Another consideration for Raid is the level of technical ability of the user. Companies package raid up and make it easy to use and install, and many descriptions make it all sound very simple and standardized, but it is not.

Here's the issue to be mindful of- in terms of reliability, raid benefits you when something fails. Lots of things can fail, and raid is an often ill documented layer between your PC, and the data. For most systems, a media failure that is detected on one drive is the selling point -- you get a failure indication (somewhere), you pull the bad drive, replace, it rebuilds -- done. It's great in that mode.

But there's so many other things that can happen.

Your motherboard fails, needs a replacement of slightly different flavor, can your raid drives just move over? Answer: It depends, especially if the raid "hardware" was on the motherboard not a separate device.

The raid controller itself fails; do you have to replace it with the exact same model to access your data? Can you even GET the same model any more? How long will that take?

One of your mirrored drives fail, you buy another, pop it in and --- it has different firmware or other minor variation as it is 3 years newer. Will that matter, can it still mirror? What if you can't get precisely the same size drive any more to replace it, will that matter? Can you just buy a bigger drive and it will use only what it needs?

You upgrade Windows or Mac -- will your raid drivers still work? Can you get newer ones if needed?

Does your raid have native support out of the box from Windows or Mac, or need special drivers. It's easy to install special drivers when adding raid to the system, but what if you were installing a new system and your system drive is on raid -- can you even boot it to install the drivers?

And back to that drive failure -- will you even be told? Or will it just keep running, no longer redundant? Depends on whether you carefully set up the software to alert you. Or maybe it's a little yellow flashing light, now hidden by that bookshelf you put in.

The list of complications that raid can introduce is really quite long, and is actually harder in some ways for home systems, as manufacturers take short cuts to hit a price point. ALL of these systems go in easily, and work great when it all works. It is years later that people often pay the price, by hitting a complex problem without good instructions to resolve.

I love raid, I use it all the time, but I used it in my job from near day one. It really worries me when I see people who don't live and breath technology put Raid in, as I think the odds are high one day they will end up in a tangled up mess.

To me it is the same answer someone told me once about the high end Super-Tele's. They said "if you have to ask if you need it, you don't. If you needed it, you already know".

 

[Linwood Ferguson]

OK, that last reply got too long, but even so I want to add a few things that are more specific:

Raid 1 (mirroring) makes it more likely to be able to move your raid set to a new computer or raid controller or newer version of the raid controller without issue. Other flavors of raid are less likely to transfer without jumping through serious hooks.

Software vs Hardware raid is often debated, and hardware raid can perform better. Manufacturers (especially intel) fake this, claiming hardware raid that is mostly software. Pure software raid (e.g. windows has a couple flavors) is also available. From a reliability standpoint software raid is just as good as hardware, and pure software raid is often easier to recover from failures. For home use I've migrated to pure software raid.

And most importantly:

Raid is not backup, a good backup is ALWAYS more important than Raid.

 

[DCBolton]

All good points but perhaps I should clarify my thoughts. Regarding use of RAID systems for storing images, I was specifically speaking of NAS (network attached storage) RAID systems rather than on-board (within the computer). I use both in certain situations but in this case, I was focused on NAS.

Yes, using RAID can be intimidating but less so if properly researched ahead of time. Purchasing from a reliable manufacturer also is advisable. RAID controller failure will happen but is less frequent than drive failure. A NAS box from a reputable source can be replaced with same model or upgraded.

AsI said in my previous post, RAID is a hardware configuration and should not be confused with how it is used. RAID can be used in a backup strategy for additional protection (as I do) but it is not a backup strategy in itself.

Finally, a quick and possibly useful story from my past experience with USB backups. (I have used many backup systems over the years; floppies, tape, CD's, DVD's. I go all the way back to paper tape and 80-column punch cards.) I had spent many hours digitizing ~70 hrs of analog family videos. I had them on the computer and transferred them to a new WD USB EHD. Tested, confirmed, all good. Disconnected the drive and stored it with the intent of assembling the videos with digitized slides that were the next step. When I got around to assembling the package, the USB drive would not start. Controller had failed. Data not recoverable because WD mates those drives with the controller. Thankfully, I still had the analog tapes, a player and digital video camera to repeat the process.

Sorry for the lengthy post. I hope it is useful. Multiple backups are absolutely essential, either in single storage or RAID confirmations. Just don't confuse the two.

Sent from my SM-N960U using Tapatalk

 

[Linwood Ferguson]

Yes, using RAID can be intimidating but less so if properly researched ahead of time. Purchasing from a reliable manufacturer also is advisable. RAID controller failure will happen but is less frequent than drive failure. A NAS box from a reputable source can be replaced with same model or upgraded.

Absolutely, getting something that is well supported and documented is a key. There is a LOT of junk out there in the SOHO market just aiming at a price point.

If you lump driver issues, user error and other causes of failure in with controller failure I'd bet heavily that drive failure is not the most frequent.

Since we are telling old stories: I managed several I.T. organizations in the days of VMS (later OpenVMS) and clusters. Clusters were like raid for the server. I used them in one organization where I had really smart, capable staff. In two other organizations, where size and budget were a constraint on staffing, I was absolutely convinced we would have more downtime from administrator error than from actual server failure, and would not let them cluster. I did not say exactly "you guys are more dangerous than hardware failure" with the complexity it added but that was what it boiled down to.

It's a tough discussion to have, especially if you are having it with yourself, but it is very important to pick technology that your staff (e.g. "you") can manage properly through its lifecycle. The owner/operator is a key component in all technology that is often overlooked.

 

[davidedric]

Another old back up tale. This goes back to when departmental LAN's were becoming popular, with the result that system management migrated from Operations to Application groups. I was head of the group supporting Finance, and so had a couple of guys managing the local servers.
Well, one weekend we had to rebuild the production server, so we made sure we had the tape backups, did the work and went to restore.
Disaster. The tapes were unreadable. All of them. It turned out the tape writing heads were mis-aligned. So the company's financial records were trash.
The ending was more or less OK. We engaged a data recovery company who managed to piece together the data from several generations of tapes, and I did keep my job.
The moral, of course, is not only do your backups, but make sure you can recover from them too.

 

[LRList001]

RAID has its good points indeed, however another consideration is reliability. RAID systems introduce more failure points, improving availability at the 'cost' of reduced reliability. Having a large number of small drives brought to create a bigger container is generally a bad idea as the chances of failure are going up all the time, ie it is generally better to use fewer, larger drives (however the calculation of the optimal solution is complex, taking into account use patterns, performance and other issues). As RAID systems tend to be bought all at the same time, the disks age together. Having hot spares are also only so useful, a hot spare is running and ageing, just like the live disks. Working out when and how to replace the drives in a RAID system is non-trivial.

Ferguson has already touched on some of these points and more of some of the other joys - such as later HDDs having different firmware versions.

For home use, yes RAID has a place, but a really good backup regimen might be more important.

The backup disasters keep coming. We have a policy of keeping things for longer than whole month, year, week etc. increments. Eg we treat a year as at least 13 months. Many, many moons ago, the file system failed, only discovered as the annual accounts were being prepared. This was the point that the 12 month old backups were erased, just the day before it was realised they were needed. Ouch.

 

[Linwood Ferguson]

OK, if we are telling stories...

Burbank News Agency had a new Vax system with a couple hundred megabytes (yes, megabytes) of storage, and backed up on mag tape. They dutifully had Iron Mountain (I think it was) come pick them up and store them off-site every day, in a carefully orchestrated sequence.

Moreover, they took the time on each backup to rewind the tape, and read-verify against the disk drive data, to make sure the tapes were usable.

All good, right?

This was a big warehouse, and had a big power transformer near the computer room. While waiting for tape pickup they stored the tapes on the transformer as it was convenient to the pickup door. Yep... each tape was being erased (or at least made unreadable) before it was picked up.

The best intentions and even careful practices are sometimes inadequate. No one thought to test them on return from Iron Mountain.

Fortunately someone visiting noticed before harm was done (we had a travelling I.T. audit group).

It's interesting to think back to that time, how very expensive disk drives the size of a clothes washer would not hold enough data for one uncompressed shot from my Sony A7Riv.

 

[davidedric]

and then there was the time when we were running a disaster planning exercise (fortunately a trial), and part of the plan was to take the back up tapes from their secure location to the site where the backup servers were. Unfortunately, the vehicle with the tapes was rear ended on the motorway - end of test!

 

[clee01l]

OK, I think we have answered the OPs question. The Lounge forum is the place to swap "War Stories". Everyone is free to start a new thread there. (I have more than a few "War Stories" including stories about a real war, and can be as guilty of this as any of you).

 

[davidedric]

Agreed and apologies. Got a bit carried away