A (long) tale of a dead drive & other things

[JBtheLD]

Well I guess it had to happen to someone....
While away from my computer for a couple of hours, my Windows Vista PC lost it's connection to my 5''Gb Western Digital MyBook Pro External Drive on which all my data and photos are stored. So I thought the best thing to do would be to power the drive off then back on. I had forgotten that the WD instructions to power down the drive are to hold the power button in for 4 seconds. After briefly pressing the power button off a couple of times to no avail, I thought my only course of action would be to pull the low voltage power lead out of the back of the drive.
Well it certainly powered down the External Drive!
So I plugged the drive back up, zilch, nada, zero, no lights, nothing. Aargh!
Anyway, being an optimistic chap, I hoped that what I had done was only destroy the External Drive's controller circuit and not killed the actual drive.
After much research on WD's support site and a call to their Tech Support people, I knew that I could return the drive under warranty for an exchange unit, but as they don't recover data from their drives my data would definitely be lost. So, I opened up the box (invalidating the warranty, of course) and removed the drive. A quick trip to the Tech Guys at PC World ascertained that the drive was still functioning and that my data was intact. I bought a cheap external drive enclosure, fitted my drive and was up and running again.
Big lesson learned. Phew!!!!
So the moral of my tale is: "Power down properly, don't just pull the lead out."
And a backup is useless if it's not up to date all the time. My last backup was from June 2''7. Useless.
So, I splashed out on a new WD 2TB "World Edition" Networked Attached External Drive, to sort out my storage problems. As if...
I figured that if I set the drive as a Raid 1 drive, my backup problems would be sorted because the 2TB would actually be 2 x 1TB in sync all the time. Well some 5 hours later, Raid 1 mode was complete.
I then started the copying of all the Digital photos from my old 5''GB drive to the new Raid 1 drive. 34'GB, 59,''' files and some 28 odd hours later all the photos were on the new drive. I instructed Lightroom to find the top level folder and Voila! - I'm up and running, or I should say crawling.
So, after some hours of trying to work with my master catalog (master.lrcat 725MB) at a snails pace I selected a partially processed single folder (called "Demon Juice") containing 15'' show photos that I wanted to complete the processing on and exported it as a new catalog (without the original RAWS) onto my internal C drive.
I opened the new catalog (45MB), pointed Lightroom to the existing RAWS on the Raid 1 drive and have just finished all the cropping and other adjustments that I wanted to do to them. Lightroom was much more responsive in the Develop module than with my Master catalog, but nevertheless it is still frustrating that there is a time lag while the RAW file is loaded. (I know it's reading 14MB Canon RAW files quickly, which is pretty remarkable, just I so want to get on with it!)
So now I want to replace the part of the master catalog that refers to the "Demon Juice" folder with the new "Demon Juice" catalog.
What should I do?
Should I:
1. Import from Catalog and select my new Demon Juice catalog? or,
2. Right click then delete the Demon Juice folder (will this remove the folder from the catalog or delete the folder from disk) from my Master catalog then Import from Catalog and select my new Demon Juice catalog? or,
3. any other way of doing this right first time?
So all in all killing my hard drive by impatiently / carelessly just pulling the lead out has set me back about 1' days. And written off a perfectly good drive. And cost me a new drive. And I'm still trying to get to grips with using a Raid 1 drive for the first time - is it really backing up on the fly? How can I tell? Is the Raid 1 drive too slow to use with Lightroom - Windows Explorer browses it pretty darn quick, so all in all I don't know. Is the Raid 1 synching a sensible backup option, because what’s deleted from one drive is deleted from both and therefore it only protects against dead hardware, not file corruption or (my) human stupidity. So much to learn.
It's been a lerning curve and a half, I hope I've learnt my lesson!
Luckily I didn’t have clients waiting for these photos otherwise I would have looked a real chump.
I do hope that reading this diatribe of depressing drivel saves all of you from the distress and expense of a (nearly) dead drive. Please sort your backups out NOW!

thanks for any advice,

John

 

[Denis Pagé]

3. any other way of doing this right first time?

A guru will soon answer this one as I do not do it frequently...

And I'm still trying to get to grips with using a Raid 1 drive for the first time - is it really backing up on the fly? How can I tell?

As you say, errors created on one drive will be reflected to the other. This solution (RAID 1) is good at protecting from a drive failure but it never relief you from doing good (should I add frequent?) backups. For example, I am using RAID 1 and last summer a drive went down. By the time the drive was sent to Western Digital for a replacement, the second drive failed before I received the first new one! This is when I most appreciated my AIT Tape backup unit with near a Terabyte of data... Otherwise, the only solution is to power off the computer until the RAID is fully reconstructed with a replacement. The reason why I lost the two is that with time and heat from the drives, the SATA connectors became brittle and a crack appeared on both sides weakening their grip making on off on off contacts... So, check your cables each time you have to open the computer and I recommend Western Digital's steel reinforced ShureGrip cables as they do!

Is the Raid 1 drive too slow to use with Lightroom

The best is generally having the RAID in the computer rather than remotely unless you use 1'''' RPM SCSI drives ($1'''+ each) in a SAN. Then, use externals for backup purposes (or tapes).

I hope I've learnt my lesson!
John

You can stop crawling don't you?

 

[JBtheLD]

Thanks Denis for your advice,

Do you know if the syncing of the second drive slows down the PC? What I mean is would I experience better performance if I was using the Raid 1 drive (2 x 1TB) as a Raid ' (?) single 2TB drive. Is it that Lightroom / Windows rights to the first drive, and then the second drive copies the first, or does Lightroom / Windows write to both drives either simultaneously or sequentially?
Thanks

John

 

[Kiwigeoff]

Well I guess it had to happen to someone....

So now I want to replace the part of the master catalog that refers to the "Demon Juice" folder with the new "Demon Juice" catalog.
What should I do?
Should I:
1. Import from Catalog and select my new Demon Juice catalog? or,
2. Right click then delete the Demon Juice folder (will this remove the folder from the catalog or delete the folder from disk) from my Master catalog then Import from Catalog and select my new Demon Juice catalog? or,
3. any other way of doing this right first time?

Yes I would just remove the folder from the catalog by selecting it and then crtl/right? clicking it which will bring up a menu where you can choose "delete".
Selecting "delete" will give another menu which tells you that the folder will be deleted and the files will remain on the disk.
Then import the other catalog.

 

[Denis Pagé]

Thanks Denis for your advice,

Do you know if the syncing of the second drive slows down the PC? What I mean is would I experience better performance if I was using the Raid 1 drive (2 x 1TB) as a Raid ' (?) single 2TB drive. Is it that Lightroom / Windows rights to the first drive, and then the second drive copies the first, or does Lightroom / Windows write to both drives either simultaneously or sequentially?
Thanks

John

You will get better performance with RAID '. Altough, this is not fail proof as RAID 1 and not a price I am willing to pay just for added space/performance. Unless you have at least 4TB (2 x 2TB) of backup space as one is never enough. I my LAN administrator's life, this happened more than once that a backup wasn't good and the second saved the company.

What will slow down the PC with RAID 1 is if it is software RAID. You should have a RAID controller. The RAID controller make the drives work in sync. It is not copying in the background what was written on one; It does both.

Now just wait a moment for someone answering your "Daemon Juice" question... to finish with this topic

 

[JBtheLD]

Geoff, thanks for confirming that route. It's what I was hoping you would say.
Denis, thanks for the Raid 1 info. How could I tell if I'm using software RAID? The drive is a Western Digital MyBook World Edition in it's own enclosure. It came with "Mionet / Access Anywhere" to allow access over the net, but otherwise seems to be self contained.
Anyway, I'll keep digging around to learn more, many thanks to you both,
John

 

[Denis Pagé]

Geoff, thanks for confirming that route. It's what I was hoping you would say.
Denis, thanks for the Raid 1 info. How could I tell if I'm using software RAID? The drive is a Western Digital MyBook World Edition in it's own enclosure. It came with "Mionet / Access Anywhere" to allow access over the net, but otherwise seems to be self contained.
Anyway, I'll keep digging around to learn more, many thanks to you both,
John

In the Western Digital documentation, they write: "The included RAID software makes it easy for you to take advantage of RAID mirroring for extra data protection.".

And for your Catalog Import/Recovery, see how easy it was?

 

[Andrew Hayton]

John,

On the catalogue front, what I do is import from catalogue and ask LR to overwrite/update duplicates. LR will then see what is new and change only that. The other way is to open the small catalogue and in grid view in Library do a cmd/ctr-s and save the metadata to the files. Now open the main catalogue and do a synchronize on the folder. This will import all the changes but not the history.

 

[Mark Sirota]

Numerous errors in this thread I'd like to correct.

(1) RAID 1 == mirrored drives.
(2) RAID ' == striped drives. This is not the same thing as JBOD.

RAID ' is typically faster for large data transfers (both reading and writing), as each disk has to handle only half the data (in a two-disk setup).

In RAID 1 for writing, both disks have to handle all the data (four times as much total work as a write in RAID ').

In RAID 1 for reading, multiple smaller reads can be faster than RAID ', because while one disk is handling one read the other can handle another read request. But for larger, less random reads, RAID ' will typically be faster.

All this speed discussion is also dependent on other factors. What is doing the striping or copying -- is it software running in the operating system, or in firmware in the disk controller? If it's in the OS, is there enough CPU capacity? Is the disk bus (USB, firewire, etc.) saturated?

That said, RAID ' is terrible for reliability. The MTBF (mean time between failures) is half that of a single disk, because a failure in either disk takes out the entire system. If you're using RAID for protection from total disk failure, you want RAID 1.

That said, if you want to protect against disk corruption, you don't want RAID 1 -- any corruption on one disk will be immediately propagated to the other disk, leaving you unable to recover. This is also true for, say, accidental deletion of your files due to human error.

To protect against this case, you want multiple single drives (JBOD), which you would synchronize by hand or using a backup utility.

 

[JBtheLD]

Thank you Denis, Andrew & Mark.
First of all, I'm going to try replacing / reimporting my Demon Juice catalog into my main catalog this evening. I'll let you know how it goes.
Second, I think that I'm stuck with the software Raid controller that came with the external drive if I continue to use my 2TB as a 1Tb Raid 1 drive.
Thirdly, I don't really know which is the best way to proceed as synchronising multiple single drives (JBOD) using a backup utility or synchronising utility would also propagate any file errors / corruptions / deletions unless I had the extraordinary wit or good fortune to spot that there was a problem on my master drive before initiating a backup or synchronisation, even more unlikely if this was scheduled to take place automatically at the end of each day.
I guess I would be entirely happy that I'm guarding against hard disk failure by using a Raid 1 drive if my Lightroom access to the digital photos on it was very much quicker.
What are the steps that I can take to make Lightroom perform as fast as possible (afap), while storing my master catalog (725MB) on my internal c drive and my Canon RAW files (34'GB) on the Raid 1 External Drive, which is connected by Cat 5 cable to my wirleess broadband router to which my PC is connected in turn? The router is the network "boss".
Should I disable Norton Internet Security "file access scanning", if that's the right term?
Do I need to connect the Cat 5 Raid drive to the PC differently?
Do I need a faster Processor and motherboard? I have an AMD Athlon 1.3GHz processor.
Do I need more RAM? I have 1.5GB RAM.
Should I revert to multiple single drives, connected via USB as in JBOD?
Or, should I import new photos into an empty new catalog on the C drive, process them, export them, sildeshow, web and print them and only when all of that is complete import that catalog into my master catalog, for the sake of keeping all my photos in a single catalog?
I know some of these questions are referred to in other threads (ram & processor), but I would love the benefit of all your wisdom to set me on the right course, once and for all.
Many thanks,
John

 

[Mark Sirota]

Thirdly, I don't really know which is the best way to proceed as synchronising multiple single drives (JBOD) using a backup utility or synchronising utility would also propagate any file errors / corruptions / deletions unless I had the extraordinary wit or good fortune to spot that there was a problem on my master drive before initiating a backup or synchronisation, even more unlikely if this was scheduled to take place automatically at the end of each day.

JBOD (vs. RAID 1) saves you from that "oops, I didn't mean to delete that!" effect. Beyond that, I agree, the benefits are minimal unless you're unusually careful and attentive.

In terms of performance, your software RAID is probably hurting for writes (imports, and perhaps for writing XMP if you make changes to many at once with auto-write on, or do a mass Save). Whether it's hurting enough to matter is something only you can decide.

Certainly the immediacy of RAID 1 is helpful for guarding against disk failure while you're working, and is a lot easier and more convenient than copying separately.

What are the steps that I can take to make Lightroom perform as fast as possible (afap), while storing my master catalog (725MB) on my internal c drive and my Canon RAW files (34'GB) on the Raid 1 External Drive, which is connected by Cat 5 cable to my wirleess broadband router to which my PC is connected in turn? The router is the network "boss".
Should I disable Norton Internet Security "file access scanning", if that's the right term?
Do I need to connect the Cat 5 Raid drive to the PC differently?

Are you saying that your RAID device is a Network-Attached Storage (NAS) device, connected with Ethernet? These are a typically a lot slower than a locally-connected USB2, FireWire, Firewire 8'', or eSATA device.

Your Norton stuff is probably causing a MASSIVE decrease in write performance. I'm not familiar with it, but if you can disable the bit where it scans files as they are written (at least for these filesystems), that will probably make a noticable difference.

I'd suggest those as the first two things to address. More memory and CPU (especially for software RAID) can help, but if actually writing the files to disk is the bottleneck, then improving performance in other areas won't help much.

 

[Brad Snyder]

John, I can add a little to the discussion.

Specifically your concern about propagating problems from the source to the backup.

I won't add any confusion by detailing my backup strategy, except to say that DNGs end up on two separate drives, and the original RAWs go to writable removable (CD, DVD, DVD/DL) depending on size; all of this happens at import.

Now, my /images directory (wherein lives everything image data related on my machine; including currently, the LR catalogs and previews)

I use Microsoft's SyncToy, auto scheduled every morning at 2AM to copy everything new from the live /images directory to the mirrored backup /images directory on a different external harddrive.

* Here's the bit that concerns you, SyncToy has several different sync modes. For the automated process, I use the "contribute" mode. In this mode, "New and updated files are copied left to right. Renames on the left (source) are repeated on the right (backup). No deletions." (Sync toy uses a metaphor of Source on the left, Target on the right to keep things organized).

This mode allows me to fearlessly let updated files source files get automagically sync'd to backup. Of course, eventually the backup slowly wanders out of sync over time, due to moves and deletions on the source side. Every couple of weeks, I take time to do an attended/reviewed sync in "Synchronize" mode, where " New and updated files are copied both ways. Renames and deletes on either side are repeated on the other". You have the opportunity to review the actions for sanity, before committing.

Like you, I'm scared of the unattended full sync because of the possiblity of replicating source problems in the backup set.

If I'm understanding your concern properly, perhaps give it a look.

 

[JBtheLD]

Mark, Thank you.

Are you saying that your RAID device is a Network-Attached Storage (NAS) device, connected with Ethernet? These are a typically a lot slower than a locally-connected USB2, FireWire, Firewire 8'', or eSATA device.

Yes, it is NAS. The drive is a Western Digital "My Book World Edition". More info at http://www.wdc.com/en/products/products.asp?driveid=34'
The info at http://www.wdc.com/en/products/resources/DriveCompatibilityguide.asp#external persuaded me that an Ethernet connection would be faster than USB2. Should I have bought the USB2 version of the drive? Anyway, I'm stuck with the NAS one that I bought!

I'll try disabling the Norton stuff, but: since my Lightroom catalogue is on my internal c drive, does Lightroom ever write to the external drive since only the RAW negatives are stored there. Isn't all the Lightroom writing going to the c drive? Oh I see, disable the Norton stuff from getting in the way of Lightroom writing to whatever drive should speed it up.
Thanks,

John

 

[JBtheLD]

Brad, SyncToy has been suggested to me in another thread and I was tempted to start using it before I killed my drive. Perhaps all this wouldn't have happened if I had got around to using it earlier!
Now I figure that what I want is a foolproof method of securing my photos that is simple and yes, I guess, fully automated which is what RAID 1 should give me. If I accidently delete a RAW negative or folder of negatives, yes I guess that would be bad; hopefully worse than that will never happen unless I do something really dull!
BTW, is it posible to set an operating system folder property switch so that all the folders below my top folder ("Digtal Photos") are non removable? If so, that would allow me to add new folders of photos but not delete or mess up the existing folders. After all, isn't LR only reading the RAW files and then writing all my edits to it's own catalogue file *.lrcat.
Thanks,

John

 

[Brad Snyder]

Off the top of my head, in Windows, you could make the folders read-only. I don't think that's what you want though. You couldn't accidnetly delete them, but they'd be very hard to work with, and that r/o attribute setting is time consuming. There may be third party tools, but I'll have to do some research.

(Off topic a little: I spent many years on Digital Equipment VAX/VMS, and you could configure all that kind of stuff per file/directory, etc.) plus it automatically would keep up to 64k archive versions of a file. It was marvelous. Then Microsoft bought the kernel, took out all the good parts and called it Windows NT)

 

[JBtheLD]

Brad, thanks for your reply. Maybe the read only switch isn't a good idea after all.
So - here's the good news:
I backed up and then opened my master catalog, right clicked my partially processed Demon Juice folder, selected delete and then confirm.
A couple of minutes later that was done.
Optimised the master catalog and made a new backup of that catalog. Just in case.
"Imported from catalog" my fully processed, separate Demon Juicecatalog.
About 2' mins later the master catalog had completed importing the other catalog with all the edits, history, virtual copies ... the lot. Edits for 1,5'' Canon RAW files. Amazing.
Brilliant, Lightroom worked flawlessly and did "exactly what it says on the tin".
I'm a very happy bunny once again.
Now, if only I get the NAS RAID 1 drive and Lightroom to work together at the speed of light...
many thanks,

John

 

[Denis Pagé]

BTW, is it posible to set an operating system folder property switch so that all the folders below my top folder ("Digtal Photos") are non removable? If so, that would allow me to add new folders of photos but not delete or mess up the existing folders. After all, isn't LR only reading the RAW files and then writing all my edits to it's own catalogue file *.lrcat.
Thanks,

John

Easily done on XP Pro with Rights Management. You set the security settings so one user (or a Group of users) can Write, Add, Modify but not Delete... Vista Ultimate and a couple others Vista flavors should do it. But I discovered that Vista Home is so much limited in this way, probably it won't do it.

 

[Brad Snyder]

Yeah, I forgot about that, principally, because I screw it up every time I try it. :shock:

If you want to go that route, make pretty darn sure that you've got all your users set up kosher(ly?) and have all the passwords available (and tested).

 

[JBtheLD]

Denis and Brad,
Thank you for the tip and for the caution.
I think that for now I'm going to concentrate on trying to get faster access to & from my RAID 1 NAS drive and then when I'm sure that's really running tickety-boo I'll come back to securing the RAW files as read only.
I think your idea of setting User permissions will be the best way to implement this, because at anytime I would be able to switch to a User with full permissions and move or remove whichever files if I wanted to.
Whereas setting the file properties to read only will: a) take a while to propagate through all the folders and sub-folders; b) have to be selectively changed back on files for moving / deleting; c) be subject to error & accident, as in "Damn, I thought I had set those files to be read-only" as you watch them sailing away.
Switching to the Full Permission User, carrying out file maintenace, then switching back to my Normal User seems simpler, quicker and inherently safer.

Thanks,

John