Corrupt CR2 files after importing from Compact Flash to HDD using Lightroom

[Sheepdisease]

I appear to be having similar symptoms as others (http://www.lightroomqueen.com/commu...ightroom-possibly-corrupt-files-during-import) but want to explain my situation in case anyone can shed some light.

I am running Windows 7 Ultimate 64bit Service Pack 1, Intel i5-2500k CPU @ 3.30GHz 3.60GHz with 8GB of RAM.

I have not had problems until recently, when I started noticing that some of the images imported from my CF card were corrupted upon viewing them in Lightroom.

I have not changed the way that I do anything or the equipment that I use. I take the CF card out of the camera, insert it into my card reader (which is connected directly to the motherboard), Lightroom opens and I import the images into the catalogue and onto my HDD.

To remedy the problem, I close Lightroom and I follow the same process, but instead of importing again into Lightroom via Lightroom directly, I copy the images from my CF card using Windows Explorer to copy the images to the newly created folder that the images were originally imported into by Lightroom.

When I open up Lightroom again, it refreshes the images and voila, they appear without any corruption.

I thought it was a problem with image caching so I tried clearing the cache several times to no avail. I thought it may have been a problem with my hard drive, so I scanned it using the utility provided by the manufacturer.

What is going on? How can this be fixed?

 

[Hal P Anderson]

You've done a nice job of explaining your problem. My guess is that you've got some flaky hardware that intermittently fails to import your images successfully.

CF card. Try using a different one and see if the problem goes away.

Card reader. Try one that attaches via USB.

Disk drive. If you have more than one, try importing to a different one for a while. If that doesn't help, you can move those images back to your normal drive using Lightroom.

RAM. Run memtest86 to diagnose any problems.

You also might try downloading prime95 and running it in "Torture Test" mode. Its easy to find using Google.

Hal

 

[Sheepdisease]

You've done a nice job of explaining your problem. My guess is that you've got some flaky hardware that intermittently fails to import your images successfully.

CF card. Try using a different one and see if the problem goes away.

Card reader. Try one that attaches via USB.

Disk drive. If you have more than one, try importing to a different one for a while. If that doesn't help, you can move those images back to your normal drive using Lightroom.

RAM. Run memtest86 to diagnose any problems.

You also might try downloading prime95 and running it in "Torture Test" mode. Its easy to find using Google.

Hal

Ran Prime95 with no glitches:

[May 3 10:02] Torture Test completed 268 tests in 10 hours, 45 minutes - 0 errors, 0 warnings.
[May 3 10:02] Worker stopped.

Ran Memtest with 9 passes without 1 failure.

Still to try importing into Lightroom using a different computer to see if I get the same corruption when I import from the same CF card and connecting using USB from camera.

 

[Sheepdisease]

Additional test carried out. I transferred the images from the CF to laptop using camera USB lead right from Lightroom and checked the sample image I showed you above and there were no problems. This suggests to me that it is not the card at fault, nor the camera itself. The card reader on my PC was fine when I transferred the images using Windows Explorer as opposed to Lightroom directly and the images appeared fine in Lightroom afterwards.

Given the fact that I have tested my CPU, RAM and HDDs with the respective software, this would lead me to believe that there is a problem with importing CR2 type images into Lightroom directly. I know I am not the first person to report this, but this has satisfied me somewhat that the problem lies with Lightroom.

What can I do to fix this?

 

[Allan Olesen]

What happens if you once again - or 10 times again - import the photo with Lightroom directly from the card?

Right now you have experienced corruption once. You used Lightroom when it happened. But until you can show that this happens more often when you use Lightroom, you have no way of knowing if Lightroom caused the error.

 

[Sheepdisease]

That is most likely true. But this isn't the first time I have had this problem, just the first time I have bothered to see what I can do about it.

This has happened upon import on 4 separate occasions now using the same CF card and importing with Lightroom. It always appears to be around the same number of images, more towards the end than beginning of the transfer and has happened on both the laptop and PC (now that I come to think of it) whilst using the USB cable and connecting the CF card directly, so long as I import using Lightroom. The workaround afterwards is to transfer directly which fixes the corrupted images, but that should really be unnecessary. The given process of checking whether any images are corrupted is laborious to say the least, having to wait for Lightroom to generate the preview for each image.

 

[Allan Olesen]

Is it correct to assume that you only do the direct import if the Lightroom import fails?

Let us assume that this happens randomly and not every time. Every time it works in Lightroom, you will not do a direct import. Every time it fails in Lightroom, you will follow up with a direct import. So your experience will of course be that the errors mainly happen when you use Lightroom. And because it only happens sometimes, the following direct import will probably be ok, because most imports ARE ok.

But now imagine that you always imported directly. If the direct import fails, you will follow up with a Lightroom import. So now your experience will be that the errors mainly happen when you use direct import.

 

[Brad Snyder]

Well, if they're CR2 raw files, you can rule out Lightroom as the cause. PERIOD. With one tiny exception, which is not pertinent here, Lightroom *NEVER* writes to a raw file type. Hence any corruption has to derive from some other operation. Even when the file is copied from the flash card, Lightroom uses the standard Windows built-in file services.

Lightroom is quite good at identifying corrupt files however, as you have noticed.

You seem to have drawn a very firm conclusion that somehow Lightroom has a problem importing CR2 files. I believe that you are incorrect, and that you will eventually identify a fault in your system somewhere.

I know it's a frustrating problem, and I sympathize, but until you can explain how Lightroom is corrupting your files without writing to them, I'll continue to believe you're barking up the wrong tree.

 

[Sheepdisease]

So what am I to do next?

 

[Brad Snyder]

Honestly, I don't know. We've seen these cases reported numerous times over the years this forum has been in operation, and the problem:
1) isn't resolved to the user's satisfaction, or
2) turns out to be related to some piece of the hardware chain.

If I'm reading right, you have not tried different CF cards, or USB cables, or card readers or cameras. Maybe you explained that somewhere, but it's not obvious to me. In fact you emphasize that you've used the same cable and card. Try changing these things one at a time and see if your problems disappear.

 

[Jim Wilde]

I've been thinking about what I would do if I was experiencing this problem. Obviously I would run all the same hardware tests that you have, and assuming they came up clean then yes, what's next? One of the problems is the intermittent nature of the problem, so all the hardware testing AND the testing on a different computer may not actually prove anything, so I would start thinking about changing some things on a semi-permanent basis to see if the problem still happens after an extended period of "doing things differently".

Looking at the pipeline from camera to corrupted files as seen in Lightroom, probably the only things you can rule out are the camera and the CF card (on the basis that you are able to recopy the original file from the CF card and this overcomes the corruption that you see in Lightroom, so the original files appear to be good). Next in line is the card reader, so yes that could have an intermittent fault....so have you got a USB card reader that you could use in place of the built-in one? Sure, not as convenient, but certainly worth trying for a while to see if the problem goes away.

After that, there are a few places where data could be written during a Lightroom import, but the prime suspects are likely to be the image files that are written to your destination drive, and the ACR cache entries (although these may not always be written during an import - it depends on your preview rendering choice - but they would be written as soon as you attempt to open a file in the Develop module).

I note that early in the thread you said that you "cleared the cache" several times....so which cache was that? ACR cache or the previews cache? Personally, I would purge the ACR cache, then I'd probably relocate it on a different drive. Next up would be trying to use a different drive for the import destination folder for a while, again in an attempt to isolate the cause of the corruption that you are experiencing.

Although I've suggested three things that you could change to try to pin down the cause, obviously I would NOT change them all at the same time....I'd start with the card reader, run with a new one for a few weeks (or until a corruption occurred). Assuming this didn't fix the problem, then I'd do the ACR cache....again run with it for a few weeks or until a corruption. Last would be using a different drive for my new imports....but thinking about this, where are your images currently written? Internal drive, external drive, network drive? If external, how connected? USB? Have you tried a different port?

As you can see, there are a few things you could try. Would be interested in hearing what changes you try, and if one of them fixes the issue.

Edit: I see Brad's already responded, in a similar vein.

 

[Allan Olesen]

I would take this action for all future imports:
1. Copy the files manually from the card to the computer's internal drive.
2. Copy those files from the internal drive to a backup location on the same drive or an external drive.
3. Import the photos from step 1 into Lightroom, using the Move option.

The next time you see a corrupted file in Lightroom, check the backup copy from step 2. If that backup copy is also corrupted, you will know for certain that the corruption occured before importing to Lightroom.

(In order to save some disk space, you could skip the backup step and instead use an md5 utility to create an md5 checksum of all files after finishing step 1. If those later turn out to be corrupted, you can check if their checksums have changed. But since a backup is always a good idea, I wouldn't skip it.)