Changes to image storage drive and missing photos

[MickD]

Lightroom Version

Lightroom Classic version: 12.5 [ 202308091523-68a7abd6 ]

Operating System

1. Windows 10

Hi Everyone,

This is my first post.

Could anyone help me with the nature of the relationship between drive changes and missing photos?

I lost a drive recently, restored my images folder to a new drive from a backup, but also chose to change the drive label (from F:\Media Library to F:\GRAPHICS). All my images were reported as missing. This remained the case even after I changed the drive label from GRAPHICS back to Media Library.

So I synchronized the root image folder with "Restore missing photos" deselected, which re-imported my images, and am now copying and pasting my develop settings from the images flagged missing to the corresponding re-imported images. Thankfully, I'm a new photographer and hadn't done much work yet in the develop module!

Two questions arise in my mind:

1) Why didn't Lightroom repair the catalogue links when I changed the drive label back to its original state? Both the drive letter and label were then the same as before I lost the original drive.

2) If I was now to assign a drive letter of Z, for example, to make it permanent, would there be negative consequences from that? Or would it simply be a matter of finding the missing root folder for Lightroom and it will simply pick up where it left off?

Many thanks!

MickD

 

[Califdan]

So I synchronized the root image folder with "Restore missing photos" deselected,

Oops - Wrong! You should have used "Find Missing Folder" on the root folder and that would have fixed everything in one fell swoop.

1) Why didn't Lightroom repair the catalogue links when I changed the drive label back to its original state? Both the drive letter and label were then the same as before I lost the original drive.

As you are on Windows, having a different drive letter or any other change to the entire path from the drive letter all the way to that root folder could cause LrC to not believe it was the same thing

2) If I was now to assign a drive letter of Z, for example, to make it permanent, would there be negative consequences from that? Or would it simply be a matter of finding the missing root folder for Lightroom and it will simply pick up where it left off?

When you find yourself at the bottom of a deep hole, stop digging. Rather than spending the rest of your life going through your re-imported images one by one and copying settings and metadata from the old "missing" copy to the newly imported copy abandon this effort and go back and do it correctly.

If you've already removed a bunch of the "missing" images from the catalog, abandon this catalog and recover the last catalog from a backup from before you did the "synchronize". Then name the drive anything you want with any drive letter you like and use the "Find Missing Folder" command to point your catalog images to it.

If ALL the "missing" images are still in the catalog (in other words each image has two previews, one marked as missing and one from the subsequent import), then you can simply remove from the catalog (NOT FROM DISK) all the images from that mistaken import (use the folder panel. Then "find missing folder" as above.

 

[MickD]

Thanks Califdan.

The curious thing was that none of the folders had a question mark and find missing folder was not available. Synchronise with remove missing photos deselected seemed the only way of dealing with the problem. Otherwise I would indeed have used Find missing folder as you suggest.

In future I intend to run, in addition to file backups, scheduled disk imaging backups for both the system disk and my USB image storage drive. This would allow me to restore the whole shebang if I really needed to.

 

[Califdan]

Hmmm, very strange. Are you saying that after replacing the drive, the LrC folders panel did not show "?" on the folders but all the images in those folders were marked as missing? Perhaps did your restore tool (whatever you used) rename the files? I've seen some such tools append something like "-Recovered" to the end of the file names.

 

[MickD]

I didn't use a proprietary restore method. It was a straightforward copy and paste of my backed up image folder to a replacement drive, using File Explorer in Windows. I think the issue has to be a result of the replacement drive having a different drive label, and changing the drive label to match the one on the old drive did not persuade Lightroom to simply pick up where it left off.

The apparent fragility of the catalog-drive linkage does concern me going forward. Possibly my issue stems ultimately from security settings in the NTFS file system. Backup software often allows a choice of whether or not to retain these in the backup. If they are retained then accessing the backup can be problematic; if they are not restrained then maybe Lightroom accepts the restored files without complications. I'm just speculating here.

 

[MickD]

Hmmm, very strange. Are you saying that after replacing the drive, the LrC folders panel did not show "?" on the folders but all the images in those folders were marked as missing? Perhaps did your restore tool (whatever you used) rename the files? I've seen some such tools append something like "-Recovered" to the end of the file names.

Yes in answer to your question tlno folders had a question mark, and images marked as missing remained so after the synchronise. Synchronise reimported the images which appeared, with a date appended to the filename of no significance that I can currently identify, alongside the missing ones, and with no develop settings applied (i.e. they were original raw files).

 

[MickD]

Hmmm, very strange. Are you saying that after replacing the drive, the LrC folders panel did not show "?" on the folders but all the images in those folders were marked as missing? Perhaps did your restore tool (whatever you used) rename the files? I've seen some such tools append something like "-Recovered" to the end of the file names.

Sorry for the typo ... I meant "no folders had a question mark".

 

[Califdan]

In the Folders panel, do you see entries for both the new and the old drives?

Are the 'not missing' folders listed under the name and drive letter of the new drive or old drive or both?

Can you post a screen shot of the your entire Folders Panel (showing all drives) with some of the cuprit folders showing?

 

[Califdan]

Launch your Adobe Creative Cloud app (rainbow infinity sign icon). Then check the "Files" tab and look at all 4 of sections. The only one that should have anything in it is the "Your Libraries" which may have one item called "My Library"

Also check your storage usage in the same app by clicking on the cloud icon just to make sure it indicats nearly full.

 

[MickD]

Only the new drive shows in the folder panel and 'not missing' folders listed under it accordingly.

What I can tell you is that I started up the drive after starting Lightroom just now, and Lightroom offered to import the images from the drive. All the images - I think, because I didn't check them all - were marked as duplicates (don't import suspected duplicates option was ticked).

A couple of screenshots attached.

 

[Califdan]

In your screen shots, I'm seeing pairs of duplicate image (for example _DSC0200.NEF). One of each pair is not missing but has a date appended to the file name. The 2nd if the pair does not have the extra date text included but is missing even though it has a Smart Preview. Am I correct in assuming that the first one (not missing with date in file name) appeared with the full import you mentioned in your most recent post, and the 2nd one being shown (missing with smart preview) was what was there when you made your first post on this topic?

If that's the case, then indeed, something appended the date to file name I can't see the entire file names on thise but the part of the dates I can see are all 2023_09_12 13_2... Is that date the day you copied the images to the new drive and if not what does that date represent? If the date was either when you took the backup or did the restore, it looks a lot like something in either your backup process or the process you used to recover those images appended the date text to the end of every file name which is why LrC can't find them.

Can you either confirm or reject my hypothosis?

 

[MickD]

Yes, the non-missing images with appended date appeared after synchronize (with remove missing photos option deselected). The missing images with smart preview were non-missing with smart preview before I performed the sychronization.

The appended date in full is: (2023_09_12 13_25_34 UTC). This is the same for every image which has been appended with it.

The missing images are not present on the storage drive. If I look in File Explorer, in Windows 10, all I see are the non-missing images with date appended.

The backup and restore:

1. For backup I used the Windows backup utility called File History, and did a straight copy and paste of the Media Library folder in this backup, to the new hard drive.

2. Before starting up Lightroom I changed the drive label on the new drive from "Media Drive" (which I had just given it in the drive formatting step) to "Graphics PC", merely as a naming preference. Simply a change of mind on what I wanted the drive label to be. Originally, before I lost the old drive, that old drive was labelled "Media Library".

3. Then when I started Lightroom, (a) all 'previous' images, with no appended date, were missing, (b) no folders were missing.

4. In the hope of getting back my missing images, I changed the drive label back to "Media Library". Throughout this whole story the drive letter has remained unchanged (i.e. F). This did not make the missing images non-missing.

5. So I did synchronize on the Media Library folder, with remove missing photos deselected. Result: all my images were now present as non-missing images with the same original (Nikon) filename (_DSC...), date 12th Sep appended, no longer .NEF but now .DNG, and without any develop settings.

The appended date doesn't strike me with any obvious significance. Possibly it was the date I first used File History to do the backup? I mean there must be some significance to it, but it is not the creation, modification or access date of a randomly selected file in the sub-folder "2023-08-28_Shropshire Steam Rally".

 

[MickD]

In your screen shots, I'm seeing pairs of duplicate image (for example _DSC0200.NEF). One of each pair is not missing but has a date appended to the file name. The 2nd if the pair does not have the extra date text included but is missing even though it has a Smart Preview. Am I correct in assuming that the first one (not missing with date in file name) appeared with the full import you mentioned in your most recent post, and the 2nd one being shown (missing with smart preview) was what was there when you made your first post on this topic?

If that's the case, then indeed, something appended the date to file name I can't see the entire file names on thise but the part of the dates I can see are all 2023_09_12 13_2... Is that date the day you copied the images to the new drive and if not what does that date represent? If the date was either when you took the backup or did the restore, it looks a lot like something in either your backup process or the process you used to recover those images appended the date text to the end of every file name which is why LrC can't find them.

Can you either confirm or reject my hypothosis?

Attached is a screenshot of file properties in File Explorer.

 

[MickD]

I think I'm starting to understand what's happened here. I believe it's an NTFS file security issue.

1. Missing images can be removed from Lightroom. Imported images with date appended cannot be sent to the Recycle bin (deleted) in Lightroom.

2. Deleting date appended images in File Explorer requires agreeing to Continue as an administrator. This occurs when the account in use has insufficient intrinsic privileges to just simply delete it.

It seems that when I copied the images to a new drive, it inherited permissions that were insufficient to make them mine, and I need to Take Ownership of the root folder (https://windowsreport.com/administrator-permission-delete-folder/). See also attached screenshot of "Preserve file security settings" option in a well known backup software. I reckon File History backup utility preserves the file security settings, hence Lightroom could not use them to make my missing images non-missing.

But with my 'workaround' method of copying and pasting develop settings to the relatively few images required, I'm reluctant to do anything that could cause further trouble. I think what Lightroom has done is re-import the images during synchronize, and change the filenames so there wouldn't be two copies of any given file in the catalog. But Lightroom can't change the security settings relative to my user account.

 

[Califdan]

Permissions may be involved but I don't believe that permissions are the problem here. The problem is that recovering files using File History appears to have appended a UTC DTS (Date Time Stamp) containing the restore date to all the file names.

So, a lot to unpack in your post....

1. Missing images can be removed from Lightroom.

THEY CAN BUT DON'T! if you remove the missing images from LrC you wll be removing all the work you did on those images. This includes titles, captions, keywords, collections, ratings, flags, and everything in the Develop Module. If, prior to your drive failure you had LrC write updates to XMP, some - but not all - of this would be in the non-missing files. Your screen shot is not showing file extensions so I can't tell if any of the files are XMP's but it doesn' appear that they are. But even if you did save Metadata to disk you would still be losing your Collections, flags, and file history among other things/

1. Imported images with date appended cannot be sent to the Recycle bin (deleted) in Lightroom.

I don't know why they can't but DON'T as these are your only copies of the images you have (that I know of)

2. Deleting date appended images in File Explorer requires agreeing to Continue as an administrator. This occurs when the account in use has insufficient intrinsic privileges to just simply delete it.

DO NOT DELETE THE IMAGES

. I reckon File History backup utility preserves the file security settings, hence Lightroom could not use them to make my missing images non-missing.

Not the problem. LrC doesn't doesn't recognize the images as they are not the same file name as before - rather they all have the UTC DTS on them

I think what Lightroom has done is re-import the images during synchronize, and change the filenames so there wouldn't be two copies of any given file in the catalog. But Lightroom can't change the security settings relative to my user account.

The synchronize folder believed that all the images were new as the catalog did not already contain images with those names. The File History restore added the UTC data - Not the LrC Folder sync process. Do some Google Searchs for restoring from File History and File history date stamps (or similar searches) and you'll find loads of info about this problem.

Here's what you need to do:

As an experiment, do the following. Take one image and using File Manager, rename it by taking off the UTC date part of the file name (including the parenthesis and any left over trailing spaces). Then go into Lrc and look at that image. The copy with the short name should now no longer be missing and the copy with the long name will be missing. If that works it proves the theory and you can continue reading......

In LrC remove (DO NOT DELETE FROM DISK) all the images with the long file name (the ones created when you synced the folders. Then there are two choices.

1) You can go back and see if there is a way to restore from file history without it adding the UTC Date to the file names. I suspect that there is but you'll have to research that on your own. If there is, then try that. If it does restore without the UTC date being added to each file name then LrC should be happy without any additional actions. All the "misisng" images will no longer be misisng (you may have to fix a drive letter or folder path)

2) If you can't find a way to have File History behave then you will need to find some way to remove the UTC date from each image file. There are many 3rd party programs that can do this. I use one called "Bulk Rename Utility"

(the UI is a bit confusing, but it works) - however there are countless others. What you want it to do is to remove the UTC date, along with the parenthesis and any trailing spaces from all the images. As the UTC dates are all the same this should not be too hard for such a program to do. Once the files are renamed, LrC should be happy as in the prior paragraph.

 

[Califdan]

I did a bit more research.

The problem you had is is this. You recovered using File History by using File Manager to drill down the "file history" folder till you found the images folder then dragged (or copy paste) that folder to your new drive. This method adds the DTS to the file name. You'll notice that all the file names have the Date appended so if you copy or drag the date comes along.

What you should have done is click the "Windows" icon on your task bar, then type in "File" and select "Restore your files with File History". You can also get there through the Control Panel

This brings you to the File History tool.

Use the left/right arrows at the bottom to go to different "as of" dates. Right click on the green restore button to use "restore to a different locaiton" optiuon rather than the "restore to same locaton" option. then click the green circle icon. Your files will be restored without the Date code appended

 

[MickD]

That's done it. I used batch rename. Thanks very much for your help Califdan. Much appreciated.

All the best

 

[MickD]

PS I had to take full control on the permissions for the Media Library folder. Otherwise the bulk rename gave Access Denied. I believe this stems from copying from one NTFS volume to another. The copy of the object in the new location inherits the permissions of the new location. So the permissions can change. Permissions only stay the same for moving data within one and the same volume.