• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom Classic questions. All the information you need is in Adobe Lightroom Classic - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. And better still, the eBooks are updated for every release, so it's always up to date.

Catalogs My Classic CC catalog was corrupt for months but I didn't know - help needed

Status
Not open for further replies.
N

noborg

New Member
Joined
Feb 15, 2018
Messages
15
Lightroom Experience
Advanced
Lightroom Version
Cloud Service
Operating System:
Exact Lightroom Version (Help menu > System Info):
 
Sort by date Sort by votes
Long story short: My catalog became corrupted when I first converted it to Classic CC in October 2017, but I didn't know it until now because:

(a) LR can run OK under some circumstances with a catalog that can't pass the integrity test on startup - i.e., if I start LR with "ask for catalog every time LR starts" and tick the "test integrity" box, the test fails and LR exits. But if I start it directly, it runs for a short time before crashing.


and

(b) Worse, if I back up the corrupted catalog (while LR still running) and specify "test integrity", the test DOESN'T fail, and the catalog is merrily backed up.

Result is I now have an unusable catalog, and all my backups since Oct. are corrupted. So I've lost edits on some major projects - bad situation; particularly frustrating since I've been diligent wrt backups.

I've tried the sqlite3 reconstruction route - the text file gets generated OK, but sqlite3 -init fails with error messages.

I've seen elsewhere in this community that there are some experts who know how to fix such problems. I'd appreciate any help available - would even be willing to pay for repair.

Thanks.

js
 
Last edited:
Upvote 0 Downvote
Upvote 0 Downvote
Victoria Bampton said:
Zip up the lrcat and use www.wetransfer.com to send it to [email protected] and I'll pass it on to an engineer at Adobe.
Click to expand...
You're the best! Thanks. It's on the way (looks like a 20 min upload).

Before you take any (more) trouble, however, I want to mention that after some discussion in a thread on the Adobe forum (You may not know your Classic CC catalog is cor... | Adobe Community ), someone named AviMathur kindly offered to look at the catalog. I think he's with Adobe, but I'm not sure.

I uploaded the file for him a couple of days ago, but he's been silent (and unresponsive) since then - don't even know if he got the file. I posted here because - after searching - it appears the folks here are more knowledgeable and helpful about this type of problem.

I'd be delighted for you to use your contacts to help me, but I don't want to put you in an awkward position or have you waste your time. So your call on how to proceed!

Also, after several experiments, I'm confident that - at least for my catalog on my system - having LR test integrity as part of backup isn't safe, as it does not catch/report the corruption. Since I presume the integrity test is the same as the one you can have invoked when LR opens a catalog, this is quite a puzzle (understatement). Have you run into this at all?

One other thing: I have tried to use sqlite3 (echo .dump | sqlite3 corrupt.lcat > lcat.txt; sqlite3 -init lcat.txt repaired.lcat) to produce a hopefully-clean catalog. In some cases, this process fails with sqlite3 reported errors , which is not too surprising. But in some cases, although it completes the creation of a new .lcat, that lcat turned out to be corrupt as well. That was surprising, as you'd think that lcat -> text file -> lcat would either fail or generate a clean lcat. Maybe that's a clue.

Many thanks.

John Shore
 
Upvote 0 Downvote
@nobrg, that the exact same statements produce different results (in that dump/load case) is concerning, that would seem to indicate a problem with your computer. While having it not work is not too surprising, having it work differently each time you do it is a sign of a bigger problem. Database dump/import is not a random number generator.
 
Upvote 0 Downvote
Ferguson said:
@nobrg, that the exact same statements produce different results (in that dump/load case) is concerning, that would seem to indicate a problem with your computer. While having it not work is not too surprising, having it work differently each time you do it is a sign of a bigger problem. Database dump/import is not a random number generator.
Click to expand...

Thanks, Ferguson, for taking the time to comment. My bad - I didn't phrase it carefully enough. What I was trying to say was that, for some corrupt lrcats, sqlite3 dump/load fails. But for some other corrupt lrcats, dump/load succeeds, but the resulting rebuilt lrcat is still corrupt (i.e., doesn't pass "test integrity" on LR startup). I didn't mean to imply that, for a given lrcat, sometimes dump/load fails and sometimes it doesn't.
 
Upvote 0 Downvote
While continuing to hope that my main lrcat can be fixed, I’ve been cobbling together a partial lrcat that lets me continue working. Along the way, one of the pieces is an lrcat that I created by

1) managing to export a small part of my main catalog resulting in a file corrupt.lrcat (don’t know if the extracted lrcat is actually corrupt)

2) reconstructing that catalog with sqlite3 via: echo .dump | sqlite3 corrupt.lrcat > corrupt-lrcat.txt; sqlite3 -init corrupt-lrcat.txt repaired.lrcat)

This reconstructed catalog has the following properties:

1) it passes “test integrity” when the integrity test is run on LR startup

2) it backs up on LR exit with both test integrity and optimize enabled – no error messages from the test

3) Nevertheless, when I create a brand new catalog and then try to import repaired.lrcat, the import fails with “LR could not import this catalog because of an unknown error”.

Perhaps that’s a clue to what’s wrong with my full catalog. I’ll go ahead and upload repaired.lrcat in case someone is looking into my problem and thinks it's useful to look at this catalog fragment.

If nothing else, this confirms that the LR "test integrity" is not trustworthy. Obviously, that's dangerous. In my case, it's why I didn't notice until now that my main catalog got corrupted when I switched to Classic CC (i.e., when the catalog format was updated). I know this because my last backup pre-Classic is OK, but my first backup from Classic is corrupt (likewise all subsequent backups).


js
 
Last edited:
Upvote 0 Downvote
Update:
My troubles have shown clearly that Classic's lrcat integrity test is untrustworthy, but I just ran into hard evidence that Classic's backup process itself is suspect.

Indeed, I investigated this because some other users experiencing similar problems (all Classic backups corrupt) have suggested that the backup itself might be at fault. And that's just what I found.

I started from scratch: created a brand new catalog, imported 3K images from various sources, created a lot of virtual copies, edited a little, and exported some jpegs. The result is a presumably pristine small catalog (40MB). However:
  • It passes the integrity test on startup (no surprise), and you can work as usual with the catalog.
  • But if it's backed up on Classic exit and then restored, the restored version does not pass the integrity test on startup. This happens whether or not test integrity and optimize catalog are ticked on Classic exit.
It's not clear, however, whether or not the catalog is actually corrupt. Just as the integrity test isn't reliable when reporting a clean catalog, perhaps it's also unreliable when reporting a corrupt catalog. Indeed, if I open the supposedly-corrupt restored catalog directly with Classic (bypassing the startup test), it opens and appears to operate just fine. On the other hand, I know from recent experience that Classic can sometimes run for a long time on a corrupt catalog without encountering the corruption and crashing. Kind of feels like running in circles - maddening.

I tested this using the same initial pristine catalog on two different (up to date) Windows systems. The results were identical. That said, the test isn't definitive globally, since the two systems (my desktop and laptop) have a lot in common wrt configuration. I'll upload the presumably pristine catalog (20180222 jshore small backup test), in case someone wishes to check this out.

Bottom line is that there doesn't seem to be a foolproof way to know whether a given catalog is corrupt. Classic's integrity test can say it's corrupt when it isn't, and Classic can run happily for quite a while on a catalog that is.

Pretty alarming.

js
 
Upvote 0 Downvote
Paul sent your catalog back late last night, so its on its way back to you.
 
Upvote 0 Downvote
Upvote 0 Downvote
Victoria Bampton said:
Paul sent your catalog back late last night, so its on its way back to you.
Click to expand...

First, many thanks for arranging this. But now I'm really puzzled. Classic throws an "assertion failed" error when trying to open the catalog that adobe believes to be repaired. This happens on my laptop as well.

And it happens both when I start Classic with or without the integrity test on startup, which I think means that the catalog pases the integrity test but still fails on startup.

If someone on your team has a Windows 10 system, would it be possible for them to launch Classic on the catalog to see if it opens (i.e., no assertion)?

js
 
Upvote 0 Downvote
noborg said:
If someone on your team has a Windows 10 system, would it be possible for them to launch Classic on the catalog to see if it opens (i.e., no assertion)?
Click to expand...
Busted here on Mac too. I'll ping Paul.
 
Upvote 0 Downvote
noborg said:
Well, I guess the good news is that Paul now has a more-compelling example of the integrity test failing to detect a corrupt catalog. :)
Click to expand...
It's not good news I'm afraid. He couldn't recover it using his normal tools.

You said it runs without crashing at times? And the same is working ok for me here. So if it was me, I'd use Export As Catalog to export chunks into individual catalogs that you can integrity check on opening. The corruption is probably limited to a small chunk of photos, so you can likely narrow it down. And then once you've got all of the "bits" in uncorrupted catalogs, you can merge them back into one.

In the meantime, I can reproduce the backup integrity check not picking up the corruption, so I'll get that bug filed.
 
Upvote 0 Downvote
Hmmm, you sent me a small catalog that you said did the same? I can't reproduce it with that one. It passes on opening, I let it back up using 7.2 and the backup still passes on opening. Can you go to Help menu > System Info and confirm which version please? I'm wondering if there was an issue in an earlier version that they've fixed.
 
Upvote 0 Downvote
Victoria Bampton said:
It's not good news I'm afraid. He couldn't recover it using his normal tools.

You said it runs without crashing at times? And the same is working ok for me here. So if it was me, I'd use Export As Catalog to export chunks into individual catalogs that you can integrity check on opening. The corruption is probably limited to a small chunk of photos, so you can likely narrow it down. And then once you've got all of the "bits" in uncorrupted catalogs, you can merge them back into one.

In the meantime, I can reproduce the backup integrity check not picking up the corruption, so I'll get that bug filed.
Click to expand...

Indeed, given I couldn't count on repair, I've been slowly rebuilding a catalog like that - combo of export/import catalog fragments and just import of image folders when that doesn't work, or when I know the metadata is up-to-date and has changed since the last readable catalog. Likewise for collections (too bad you have to do that separately). The biggest impediment is that I have a large number of published catalogs, which can't be reproduced by export/import (grrr), but I'm having some luck with the voyager plugin (https://alloyphoto.com/plugins/lrvoyager/).

I'm disappointed that Adobe couldn't repair my catalog, but I greatly appreciate the effort by you and by them, and likewise that you'll file a bug report regarding integrity check not being reliable in picking up corruption.

Plus, I've certainly learned a lot of useful things about catalogs!
 
Upvote 0 Downvote
Victoria Bampton said:
Hmmm, you sent me a small catalog that you said did the same? I can't reproduce it with that one. It passes on opening, I let it back up using 7.2 and the backup still passes on opening. Can you go to Help menu > System Info and confirm which version please? I'm wondering if there was an issue in an earlier version that they've fixed.
Click to expand...

Yes, I had sent you "20180222jshore small backup test.lrcat" - a clean catalog that passes the integrity test on startup, but after backup/restore no longer passes the test (but doesn't seem to have been actually corrupted, as you can open it directly and work as usual.

My desktop is at 7.2, but I hadn't upgraded my laptop since 7.0.1, and I decided to keep it that way while sorting out my difficulties - figuring that could facilitate testing and possibly recovery. The backup/restore behavior was the same on both 7.0.1 and 7.2, which makes me wonder whether the bug only affects Windows. But to be sure on my end, I'll retest on both systems, and let you know.

FYI (you may know this), I'm not the only one reporting that all Classic backups are reported corrupt by the integrity test on startup (see Lightroom Classic CC Backup Catalog Corrupted | Adobe Community). (And that's why I started my reconstruction efforts using my last pre-Classic backup.)
 
Upvote 0 Downvote
Upvote 0 Downvote
Victoria Bampton said:
Zip up the lrcat and use www.wetransfer.com to send it to [email protected] and I'll pass it on to an engineer at Adobe.
Click to expand...
Hi, This is my first time posting to the site and I have the same issue as noborg...I have a corrupt catalog in Lightroom classic CC. I tried SQLite3 repair with same error issue of step 1 of SQL repair seeming to go Ok but errors on writing to the .lrcat file. I would appreciate if you had advice or could also send it to an Adobe engineer or suggest other methods for repair. thank you so much!
 
Upvote 0 Downvote
Hi, This is my first time posting to the site and I have the same issue as noborg...I have a corrupt catalog in Lightroom classic CC. I tried SQLite3 repair with same error issue of step 1 of SQL repair seeming to go Ok but errors on writing to the .lrcat file. I would appreciate if you had advice or could also send it to an Adobe engineer or suggest other methods for repair. thank you so much!
 
Upvote 0 Downvote
Hi - just saw this (been traveling). I hope that Adobe (via Victoria) was or will be able to fix your catalog. In my case last year, Adobe also worked on mine (thanks again, Victoria). They sent back a catalog that they said was no longer corrupted, but unfortunately that turned out not to be the case. In the end, I had to start over with a 6-month-old backup (the most recent one prior to converting to Classic CC) and painstakingly reconstruct it. I have no idea if Adobe has fixed the bug, but since then I've never used the built in LR backup. I just set up my own system.
 
Upvote 0 Downvote
Sorry to hear about your catalog issue. I did some research and was able, miraculously, to repair it myself and I am anxious to download the repaired catalog from Adobe from Victoria which I greatly appreciate their efforts. I attached an instruction sheet I created for us in case it happens again, I have attached it to this post...I hope that is OK with group.
 

Attachments

  • Steps to fix a corrupt Lightroom Catalog.pdf
    523.3 KB · Views: 1,002
Upvote 0 Downvote
Status
Not open for further replies.
Back
Top