Catalogs Test Integrity before backing up

[Michael Naylor]

Lightroom Version Number

Lightroom Classic version: 8.3 [ 201905061635-edecdfdb ]

Operating System

1. macOS 10.14 Mojave

Please could some explain exactly what "Test Integrity before backing up" is suppose to do? It doesn't seem to be doing anything with regard to the problems I'm having.
1) Previously entered keywords have mysteriously vanished.
2) Smart Collections filtering on "Keywords -> doesn't contain" is displaying false positives.
Any clues for fixing theses issues?

 

[clee01l]

The Lightroom Catalog is a database. It consists of many tables with records (Like a spreadsheet). These tables are related to each other using some common key. Usually a unique identifier for each image or simply a unique integer value ( for thing like keywords that can be assigned to many images). To speed things up there is an index table that can efficiently keep track of the locations of records in the tables.
The integrity check is a database process that every record in a table has a relationship to the related tables, there are no orphan records and that the index tables are up to date.

Keywords do not vanish without a little user help. If you create a keyword then you should be able to find it in the Keyword panel. If you assigned a keyword to an image, then that assignment will remain until you unassigned it or delete the keyword from the master list. If these two facts are in dispute, I would question whether I assigned or created the keyword in the copy of the catalog file that is open or an old copy of the catalog that was replaced by the one you have now open.
Smart Collections filtering on "Keywords -> doesn't contain" is displaying what it finds in the search criteria Doesn't contain > "bird" will filter out images that do not contain "Bird", "BIRD" OR "Birdland". I would look at the criteria field for an imperfect construction. Spaces in the Criteria field are treated as an OR ("Bird cat" = Bird or cat). Also look at the match criteria Match "any" to Match "all".

I can tell more if you include your exact Smart Collection in a reply.

 

[Johan Elzenga]

2) Smart Collections filtering on "Keywords -> doesn't contain" is displaying false positives.

This is almost always user error. Give us an example and we might be able to tell you what went wrong.

 

[Michael Naylor]

The integrity check is a database process that every record in a table has a relationship to the related tables, there are no orphan records and that the index tables are up to date.

I backup at least once a day and have never had an integrity warning, so I assume the database must be OK. Therefore, should I conclude that what is happing is because of bugs in Lightroom?

 

[Michael Naylor]

This is almost always user error. Give us an example and we might be able to tell you what went wrong.

I can't send the entire catalog. What kind of example could I send?

 

[Michael Naylor]

Give us an example and we might be able to tell you what went wrong.

Johan. I can't give an example of missing keywords, here's what happening with Smart Collections.
All keywords are structured, eg A>b>c, etc. Out of 25,000 images, about 5,000 have People, as in #PEOPLE>Family>Michael.
About 24,000 have #WHERE>Spain>Andalucia>Mijas>Pueblo, etc.
All the keywords are saved in the files (IPTC & XMP) - as seen by Exiftool.
All keywords are display correctly The LR Keywords & Keywords List.
I create a Smart Collection like this...
Keywords | doesn't contain | #WHERE
And a number of images that do contain, eg #WHERE are displayed along with the ones that don't.

 

[clee01l]

Therefore, should I conclude that what is happing is because of bugs in Lightroom?

I don't think you paid attention to my post. There has never been a case where a bug caused a user to lose data. Did these keyword ever show up as assigned. Are these keywords that are not missing from the keyword panel? Do you have an export showing these keywords but they now are missing from the cataloged image original? The most likely occurrence is that you deleted the keyword if it ever was assigned. Not paying attention to how many images were selected, I removed several keywords from one image and had that applied to the over 2000 image that were selected. I had to go back 6 months to find a backup that contained the assigned keywords to correct the problem. Again the most likely cause is what I call "stupid user mistakes" (Like mine).
A screen shot of the Smart Collection or export it and change the extension to "TXT" so that you can attach it to the reply.

 

[Michael Naylor]

Here are a few screen shots...

 

[Michael Naylor]

PS The Askwith Moor should not be displayed, because it has a #WHERE keyword.

 

[clee01l]

That "#" may be a special query character (I'm not sure) This may be the problem.
Try running the query with
{Keywords}{doesn't contain}{WHERE} omitting the # sign

 

[Michael Naylor]

That "#" may be a special query character (I'm not sure) This may be the problem.
Try running the query with
{Keywords}{doesn't contain}{WHERE} omitting the # sign

I've tried that, and I've also tried changing #WHERE to LOCATION, but it didn't make any difference.

 

[Johan Elzenga]

Your keyword field is set to ‘Keywords & containing keywords’. Set it to ‘Keywords’ only. Do you still see false positives?

 

[Michael Naylor]

Your keyword field is set to ‘Keywords & containing keywords’. Set it to ‘Keywords’ only. Do you still see false positives?

The options are: Enter Keywords, Keywords & Containing Keywords, and Will Export. None of these affect the outcome. And as said early, they all display exactly what one would expect.

 

[Johan Elzenga]

I’m not saying it will affect the outcome. I was thinking that if #WHERE is a containing keyword for this image, not an assigned keyword, then the outcome is correct. I haven’t tested it, though.

 

[Michael Naylor]

I’m not saying it will affect the outcome. I was thinking that if #WHERE is a containing keyword for this image, not an assigned keyword, then the outcome is correct.

It doesn't matter if #WHERE is explicitly assigned or not. In an case, the false positives are quite spurious. I've been trying to resolve this for 2 days now. That's why I suspect it is a bug, or at least a programming error. Whatever, I now have a work around, albeit extremely tedious...

The only way I've found is to create another "temp" keyword, sometimes parent, sometimes child. It's trial and error. Then move suspect keywords into the "temp", delete the original and rename the "temp". The problem is finding the false positives. I'm having to eyeball thousands of images, looking for them one by one. It's going to take weeks.

In this occasion I don't feel I have made "stupid user mistakes", as was implied.

 

[clee01l]

In this occasion I don't feel I have made "stupid user mistakes", as was implied.

You still haven't explained how the keyword were there and assigned and then disappeared from either the image or the keyword list. I can assure you that LR does not arbitrarily remove keywords from images. I also think that you are probably the only person to have this problem. So, where do you think the error got into your catalog?

 

[Michael Naylor]

You still haven't explained how the keyword were there and assigned and then disappeared from either the image or the keyword list. I can assure you that LR does not arbitrarily remove keywords from images. I also think that you are probably the only person to have this problem. So, where do you think the error got into your catalog?

The keywords were there because I added them. Two years ago I began scanning 10,000 historical family images, identifying the people, locations and date or capture as best I could. Some I wasn't too sure about, but for portraits of myself, mother and father, I was pretty sure who they were. During this time I've had these images in and out of LR, trying to find a better DAM. Some were better in other ways, but I finished up back with Adobe. In fact, I've been with Adobe since version 1. Each time I've migrated I've checked and double checked the keywording and evolved the chronology based on year and month. Almost all the dates I come up with are estimates based on other estimates, often based solely on distant memories, hair style, location. or by guessing someone's age. I have lists of births , deaths, marriages and divorce, the cars I and my father owned, and many other clues to work from, but not necessarily the precise date surrounding all those events.

I'm continuously reviewing the images and why anomalies get noticed. It's an obsession and continues to be an ongoing process and will do till someone, Adobe or other, comes up with something better. Fuzzy dates and a visual timeline would help, but I do make use of Numbers, OmniOutliner, MindNode, Ninox Database and temporary scraps of paper.

Despite my extended experience with databases, including database design and writing of SQL code in a previous life, I'd never really paid much attention to the LR checkboxes for ancestors until recently. Any alterations of that kind are a cause for concern, so I backup before changing anything and search for possible errors during and immediately after.

Finally, to further answer your question, I don't think any software engineer would be arrogant enough to claim their software was absolutely bug free, so it would be absurd for me to claim 100% certainty about missing keywords. However, with a small number of photographs which are particularly memorable and key to the dating structure, I am sure those keywords had been previous added, checked and double checked.

P.S. You haven't really answered my original question. What exactly does "Test Integrity before backing up" actually do?

 

[Johan Elzenga]

Here's something that you can try: If you add an exclamation point before the search term, it is considered as 'not'. That means that 'does not contain' #WHERE can also be written as 'contains' !#WHERE. That gives you the option to use slightly different criteria. Try 'contains words' !#WHERE.

 

[Dan Marchant]

P.S. You haven't really answered my original question. What exactly does "Test Integrity before backing up" actually do?

It checks that all the statements in the database are valid (that it all makes sense) before backing it up.

 

[clee01l]

Finally, to further answer your question, I don't think any software engineer would be arrogant enough to claim their software was absolutely bug free, so it would be absurd for me to claim 100% certainty about missing keywords. However, with a small number of photographs which are particularly memorable and key to the dating structure, I am sure those keywords had been previous added, checked and double checked.

P.S. You haven't really answered my original question. What exactly does "Test Integrity before backing up" actually do?

Note in my earlier response, I did not say that Lightroom is "bug free", but that in all of the versions released to users (including some very buggy releases) Lightroom has never lost user data. Others here with connections to Adobe will stand by that statement.
Both Johan and me suggest that the keyword missing problem is related to user error. In fact I'll go so far as to say that almost all problems presented here are caused by user error or user not understanding how LR functions.
Find me another user with your problem and I can then consider that it is more than user error.
I am not presumptive enough to think that I am beyond mistake and that includes mistakes in my responses to your "Keyword" post . However your statement "During this time I've had these images in and out of LR, trying to find a better DAM." opens additional questions. Have you always used the same master catalog or did you import into a new catalog each time you returned to LR? Do you have a backup catalog that shows that these keywords did at one time exist in your master LR catalog that you now use? Or a former catalog?

As for the integrity test, that was verbosely answered in my very first response. Lightroom will not report errors in the database integrity, it will quietly fix them.

 

[PhilBurton]

As for the integrity test, that was verbosely answered in my very first response. Lightroom will not report errors in the database integrity, it will quietly fix them.

Some time ago, I asked Victoria why she (or Adobe - can't remember which) doesn't provide user tools to fix corrupt catalogs. Instead, Victoria requests that people send her their corrupt catalogs, which she then sends on to Adobe for repair. The explanation was that by fixing users' corrupt catalogs, Adobe can learn about different types of catalog corruption, leading to better automated tools within Lightroom to fix catalog corruption.

I suppose that Lightroom could present us with a skull-and-crossbones image along with a message saying that the catalog is corrupt. And then ask us if we want to have LR perform the necessary fix. But what good would that warning do? Who would ever refuse the fix?

 

[Califdan]

Have been away so have not followed all the details of this thread and have not read all the posts in it either. But, a couple to quick ideas.

1) Open Keyword list and assure that there is nothing in the Keyword List Filter.

2) (this is far fetched so please excuse me if it's not related) You say that your images have passed back and forth between different DAM's and LR and that each time you put them back into LR you verify that the KW's are there. Fair enough. But, if on one of those "re-import" occasions the KW's had been stripped out by some other DAM, or (depending on file type) associated XMP files were lost along the way, the images would not come in with prior keywords. Then, in LR if you used the "delete unused keywords" option (I think I recall such a thing) those empty (or unused) keywords would be removed from LR. Like I said far fetched sequence of events.

3) While I understand Clee's statement about there being no known cases of lost data with LR (which I am neither validating nor denying), there have been cases of corrupt catalogs. I know because around version 3 or 4 of LR I was a victim of such. Neither myself nor several levels of tech support at Adobe were ever able to discern the cause of the corruption, and Adobe was not able to repair the catalog. Even though I had daily or weekly catalog backups, this required me to revert to a catalog backup from roughly 5 months earlier which was the most recent backup that did not also exhibit the corruption (the corruption only made itself known when accessing certain folders from several years earlier). So, in essence everything I had done in LR over those 5 months was lost. It's symantics whether this constitutes "lost data" but was a pain in the butt either way. So, it is entirely possible, in my opinion, for database corruption to be at the root of your problem. Unfortunately I am not capable of helping with such issues, but I believe there are some folks who can.

 

[clee01l]

While I understand Clee's statement about there being no known cases of lost data with LR (which I am neither validating nor denying), there have been cases of corrupt catalogs.

A corrupt catalog is the result of the open file being left in an unstable state during a crash. A properly handled catalog will not go corrupt without outside forces (OS and other apps) affecting the file or the the database operation that needs to update multiple tables before the file gets closed. LR does not update the file immediately but instead write the updates into a journal file to be updated when the OS has available Read/write I/O time.

 

[Califdan]

Clee: Yes - prime cause is process termination before all write operations are complete and buffers flushed such as power failure, system crash, or forceful process termination by hand. But, buggy code (what we used to call "incomplete instances" to avoid using the "bug" word in status reports) can also cause database corruption and many such cases are very obscure to find - especially with object oriented coding where the timing of events in the code is not as predictable as it was with linear coding.

BTW, I should note that even with the most recent versions of LR Classic, every now and again it goes into an endless loop (just to be sure I let one such occasion run for over 12 hours to see if it would ever finish whatever it was doing and it never did). When this happens one or more occurrences of "Bezel" keep coming and going on the Windows task bar and the LR window keeps flashing on and off or alternating between LR and the Desktop. The only way out of these cases is to force a power off or use task manager to terminate the LR process and in either case risk catalog corruption. I only say this to point out that sometimes one is forced to "improperly" handle (as you say) the catalog as it is the only option available. This has been reported to Adobe, but as it is not reproducible on demand they have so far been unable to correct it. The last occurrence for me was on 8.0 (last week). I skipped 8.1 and 8.2 and have only been running 8.3 for a few days so time will tell if the problem still exists.

 

[Victoria Bampton]

There have been rare cases of corruption that don't trigger to corruption or data integrity message. The main one showing up at the moment is the "ghost images" issue and I think I've seen about 5 cases in the last year. The test integrity picks up most but not all.