Broken Catalog

Status
Not open for further replies.
C

chris_11

Member
Joined
Sep 27, 2021
Messages
25
Lightroom Version Number
6.14
Operating System
  1. Windows 10
Hi there,

I really hope you can help me please, have uploaded through we transfer already and would much appreciate some assistance please.
I do have a backup but every back I have is since discovering a problem, my latest is 8 days ago so I have a "working" copy but.....

For some months I have had a problem where accessing some keywords (or some combination) results in the keyword panel showing "loading" and not being able to progress further. Definitely keywords, everything else works. I do encounter keywords that have a photo count but no photos show (or an incorrect count) so I wonder if there is a linkage issue in howeve the database stores the data.

I have been re-keywording into a hierarchical structure (if you get that far they are the top level ones starting with an underscore to differentiate for now).

However, today when opening the catalog, lightroom appears as a background process and does not display to me, this was after the latest of it's "I'm not playing ball anymore on keywords".

Other catalogues, if opened first are no problem, I can then change to this catalog but I get an incomplete display (no photos, the library panels on the right hand side appear partially drawn and not populated with data).

The first question is can it be rescued please?

But perhaps the more important question is are you able to identify what they keyword issue is and is it fixable - I've spent many frustrating hours try to solve the issue, photos that have been rekeyworded appear ok but i's haphazard trying to fix what's outstanding, would very much like to change to a newer version of lightroom but cannot see the point of moving if my catalog is already damaged.

Any help greatly appreciated, thanks

Chris
 
Sort by date Sort by votes
Hi Chris. Yep, there is a problem. It upgrades to the current version ok, but the Folders panel is empty. If I try importing it into a clean catalog, it fails.
I think there's an incomplete record somewhere as the photo count shows as 13757, but more than 16000 when trying to import it into another catalog. We can pass it on to Adobe to see if they can rescue it, but the result might be a Lightroom Classic 11 catalog, so you'd probably need to upgrade to the subscription to be able to use it.
 
Upvote 0 Downvote
Hi Chris

Unfortunately bad news. It can't be recovered. So, back to a backup, I'm afraid.

You could write out metadata for the photos you updated Keywords in and then read metadata once you have the backup catalog running (take an extra backup of the one that works first) - that could recover some of the work you've done.
 
Upvote 0 Downvote
Thank you for taking a look, it's disappointing but not entirely surprising, I'll keep trying to clean up the older version, back to the grindstone for me :-(
 
Upvote 0 Downvote
I am quite stumped, I have tried exporting all of my photos to a new catalog and I am still seeing issues in a new catalog. I have also tried moving the catalog to a different machine (unfortunatley ancient laptop being my only option) and that made no difference. I am beginning to wonder if I have found/triggered a lightroom "feature" somewhere :-(
 
Upvote 0 Downvote
Take us through exactly how you "exported your photos to a new catalog"? And what kind of issues are you continuing to see in the new catalog?
 
Upvote 0 Downvote
Thanks Victoria, I used the "Export as catalog" option, selecting none of the options to create a new catalogue on a different drive without moving my photos with it. That ran through with no problems.

I'm still fixated on fixing the keywords so my first test was to delete a couple of keywords that now seem of no use to me but with large numbers against them eg "holiday" and where previously that would definitely cause light room to stop responding it worked perfectly. But now, I have for example the keyword "pony" with 183 photos that I wanted to go through and add keywords for the names of the ponies, lightroom will bring up the 183 photos straight away except when I scroll down I can immediately see it is not rendering the previews and the keyword panel is stuck saying "working":
1636274594850.png


It is that state, stuck saying "working", that was plaguing me before and where my problem starts. My PC is by no means fast and I do occasionally see the word flash up so it's obviously initially there by design when loading data but sometimes gets stuck. It's also at this point that exiting lightroom leaves behind the lock & journal files. As I said before I have seen this behaviour on a different machine (I was really hoping that it was as simple as my PC, desperately needs replacing so an excuse required!)

My initial thoughts were that I have linkage problem with some (many!) keywords and hence working through reinventing into a more useful hierarchy and on photos that I know I have completely cleared the old keywords out of the problem goes away.

I had imagined that exporting the photos would recreate the data from scratch but I suspect that it's actually transferring the database content (which makes sense, I just hadn't thought to hard about it before).

Potentially related, optimize catalog sometimes gets stuck, if I try running it from the menu before I encounter a problem it works fine but once I've had any odd behaviour it is doomed (no surprise).

I have just exported the keywords (hadn't spotted that option before) and it just looks like what I might expect to see, the only oddity is one keywork has commas but I don't imagine that's a problem. Just kicked off building previews from scratch to see if that makes any odds.

Any pointer appreciated and thank you for your support
 
Upvote 0 Downvote
I did rebuild the previews in case the performance of that extra step was causing a problem but the same operation of selecting a photo (again from the pony selection, different picture) just instantly stops the keyword panel functioning on "working", when I switch to the develop panel the photo does not load and the panel contents don't load.
1636323462451.png

I can also back on the original screenshot (I didnt notice before but was just retrying) that some pictures show a file bame and some dont, might be related to the point the data stops loading but there doesn't appear to be any logic as to which dont get loaded (it's not file / name order), I can double click on a photo from the library list at which point effect happens and I now am stuck with a loading message at the bottom
1636323999475.png

I was trying to figure out the name of the file...if I picked a single picture from the library and goto the develop module on a fresh start the develop module is fine.....it does appear to be library related.

Thx
 
Upvote 0 Downvote
chris_11 said:
But now, I have for example the keyword "pony" with 183 photos that I wanted to go through and add keywords for the names of the ponies, lightroom will bring up the 183 photos straight away except when I scroll down I can immediately see it is not rendering the previews and the keyword panel is stuck saying "working":
Click to expand...

Great, you might have narrowed down which photo has trouble. If there's only that odd one, we might be able to use an SQLite editor to remove the record for that problem photo.

I suspect Adobe's corruption tool would probably be able to fix it. Or, rather than export as catalog (which tends to dump out the lot), you could try using Import From Another Catalog to pull a year at a time into a new catalog, until you narrow it down to which photos have corrupted records.
 
Upvote 0 Downvote
Unfortunately I don't think it's a photo (if it is then it is at least hundreds of photos given how things have been going), however, I didn't think of trying the other way round, so yes I'll try importing, if it does turn out to be photos at least I can work through the problem, going to take me a while (mad busy week already) but I'll come back and let you know, I would love it to be as simple as a photo or two tripping things up!

Chris
 
Upvote 0 Downvote
If you're planning on upgrading to the subscription, I'd suggest you save yourself the time and see if our Adobe guy can fix it automatically.
 
Upvote 0 Downvote
I thought I would give it a go ...... this is going to take a while, for starters my folders are a bit of a mess and this is a good reason to have a good tidy up. Then I found the very first folder has 3 photos in it and just importing those 3 caused an error. Deleting the files from lightroom and reimporting into my original catalogue has resolved that issue, many files have no edits so this might not be too bad, only 545 folders to go :)
 
Upvote 0 Downvote
Losing the will to live here, I have tried handful of folders (so about 1% of the total folders and a fraction of the overall pictures) and in every case the "import from another catalogue" has errored. In all cases except 1 recreating keywords solved the problem but I've got no evidence as to why. One folder (the biggest typically, a whole 10 photos) was reporting as 11 photos and I ended up exporting and reimporting into the source catalogue and that worked, the problem is that loses on the development steps. Doing > 500 folders just isn't viable. Can I pass you the catalog to see if it is fixable please? The value of lightroom from my perspective is entirely in the library module and retention of development steps so really dont want to lose that part.
Thx
 
Upvote 0 Downvote
Will do.
 
Upvote 0 Downvote
Dead as a dodo, sorry Chris. It’s backup time
 
Upvote 0 Downvote
Apparently @Paul McFarlane already sent the catalog to Rikk in message 3 and I completely missed it! It's a good thing he knows what's going on!!!! I'm sorry it's not better news. I might be tempted to try with parent folders... import from catalog got to about 5000 before failing, so that could be a good place to start.
 
Upvote 0 Downvote
Thanks, trying to work through bit by bit to see what I can drag out and save.....the product loads and operates, it just trips up in the library so I am trying to figure out what I can rescue out into a nice new catalog.
 
Upvote 0 Downvote
Hi Chris

One other approach you might consider is to save metadata to files (Ctrl+S) then import to a new catalog; perhaps try it with a folder. You at least save the edits you've done that way.
 
Upvote 0 Downvote
Thank you I will give that a go later, I was struggling to get keywords to reimport let alone development settings so perhaps that the answer I hadn't spotted!
 
Upvote 0 Downvote
A wander through my discoveries - fundamentally because my original photos are largely untouched by lightroom I do at least know I can get back to where I started in some form and worst comes to the worst for the last 3 years I have been copying the original files to a dumping ground on disk (haven't looked there for a while, that's 14,000 files I've never needed to touch).

It seems I can save metadata to a file, that's going to play havoc with my backups!
If I then create a new catalog and import the file it delivers the final photo version as opposed to the original.
  • it's annoying not to have a practical means of recovering my actual edits but they are back there in the broken catalog, would be a pain to manually recreate but hey needs must!
  • my PC is slow....so slow I see it apply the develop updates when the preview is created and indeed if I just open a source JPG in Windows I can still see the original. I like this but I thought it would be good to have XMP files as clearly separate, but that option crashes the broken library and reopening it's stuck on creating xmp files all the time but not progressing, there was no obvious logic to where it started creating xmp files but the very next files after it stopped were showing a metadata conflict so may be some logic there.
It also brings across the metadata (I am talking about the title, camera information)

However, keywords :(
It does load the keywords into the keyword list.
It does not retain the selections for a particular photo, I have just discovered that I can "read metadata from file" and that loads the keywords but there is no keyboard shortcut and the synchronize option for a folder doesn't cover it, it is possible to highlight all the photos and read metadata, not ideal but it works.

I have also tested this between two newly created catalogs , to see if the behaviour is consistent but given that I am reading from the file not the catalog that was a pointless test (didn't think of that until after I tried!), definitely designed behaviour.

In addition whilst it brings across the colour labels and ratings it does lose the flagged / rejected (and of course stacking) as these are clearly lightroom specific features, again not the end of the world and perhaps it's time to create a new simplified library (all those stacks I kept for panoramas just in case I wanted to revisit later!)

I have tested this with different file types to try to ensure it works consistently and does what I expect everytime so in case it is of any interest to anyone :
  • ORF (olympus, it creates a .XMP so I can still see my original file),
  • DNG (which has an option to update DNG preview and metadata, seems to have the same effect as saving metadata).
  • TIF (Interesting side note is that saving meta data to a TIFF file does not update the date modified on the source file.)
  • JPG
  • PNG

Ultimately it looks like I can create a new catalog and rebuild over time (! dreading that). Of more help I have managed to recover a backup copy of my catalog from March.....and am yet to see any signs of problems in it (other than a massive cull I did on photos, all the new photos added and a large swathe of re-edits over the summer!).

My only concern about recreating all this in lightroom is that it's clear that the software has an integrity check that doesn't work so anything I do might be in vain if another issue occurs as there is no way to tell until later. Even when a problem is identified the message gives no information to support resolving any issues and a quick search on google (because I was looking for why the keywords won't load from files automatically) demonstrates that I'm not the only one who has had problems.

Having adopted the one big catalogue approach a couple of years back it does have benefits but the downsides are now clear.

Getting there very very slowly, thanks for all your help and support.
 
Upvote 0 Downvote
Honestly... going back to your last good backup from 8 days before your first post is probably your best bet. Does that have the same issues? That is what backups are for!
 
Upvote 0 Downvote
Yes :-(
I've been trying to re-keyword for months because I thought that was the problem.

I've recovered a catalog from March and it appears fine (but that was before I deleted several thousand pictures and then loaded more). I've got no catalogs that have the current photos that appear to be fully functioning so I'm sorting through what I have to hand and trying to figure out a sensible approach, if I can successfully import some folders and manually tackle others I'll be done in 6 months :)

I'm at that point of just buckling down and getting on with sorting the problem but biggest concern is that there was no indication of a problem / corruption etc and it was pot luck that I came across a new issue, I think initially I just thought it was lightroom because it's always been painfully slow tackling keywords.
 
Upvote 0 Downvote
Status
Not open for further replies.
Back
Top