Problems with folders and files containing Swedish letters (åäö)

[KMagnus]

Lightroom Version Number

Adobe Lightroom Classic CC 7.4

Operating System

1. Windows 10

Hi,

I have a problem which I can't seem to solve on my own or by Googling.

Scenario:
Recently moved my catalogue and all photos from an internal drive of an iMac to Windows 10 and an external G-Raid disc. While doing this I also moved up one or two version of Lightroom but can't confirm exactly from what version atm. Everything went smooth with about 4/5 of the files and photos but the rest were marked with "?", not found.

Problem:
It's easy enough to click and execute "Find Missing Folder" - results in the folder being properly added to the catalogue, however the photos in that folder is not if the filenames of the photos contain any Swedish characters (åäö). It's also easy to click one of those photos and select "Locate", find the exact photo and get it added back in (even if it contains a Swedish character). Now this is where the huge problem begins. Even with the box "Find Nearby Missing Photos" checked, Lightroom constantly just adds one photo at a time. When clicking "Select" I get a warning saying that the selected file does not have the same file name as the one selected (see attached screen grab). Challenge is, as far as I can see, they are named exactly the same. This problem becomes huge because of scale, it would take me a long time to fix this manually, one photo at a time.

Any help or suggestion would be greatly appreciated.

Best regards,
Karl Magnus Troedsson

 

[Victoria Bampton]

Hi Karl, welcome to the forum! That sounds very frustrating, and I don't think that's one we can fix, as it sounds like you're doing everything right. The best I can suggest right now is reporting it to the engineers at the Official Feature Request/Bug Report Forum at Photoshop Family Customer Community

That said, if you want to zip up the catalog (just the lrcat) and use Dropbox or www.wetransfer.com to send it to me at [email protected], I don't mind having a quick rummage inside the database to see if I can fix the links manually for you in the meantime. I can't promise it'll work, but it's worth a shot.

 

[Wernfried]

I regret your problem but for me this is another reason why one should not organize photos in folder and file names. For that we have keywords, metadata and collections in Lightroom.

 

[clee01l]

Welcome to the forum. I agree with Victoria. But I would also. like to add that I think the issues is probably related to the Windows 10 filesystem (NTFS) incompatibilities with the MacOS filesystem (HFS+). I think the translation from HFS+ to NTFS came up with incompatible characters and the characters used in the NTFS file and folder names no longer agree with the characters stored in the LR catalog for those same files and folders. I've seen this problem before with similar situations and ASCII characters greater than &#127. (0-128)

 

[KMagnus]

Hi Karl, welcome to the forum! That sounds very frustrating, and I don't think that's one we can fix, as it sounds like you're doing everything right. The best I can suggest right now is reporting it to the engineers at the Official Feature Request/Bug Report Forum at Photoshop Family Customer Community

That said, if you want to zip up the catalog (just the lrcat) and use Dropbox or www.wetransfer.com to send it to me at [email protected], I don't mind having a quick rummage inside the database to see if I can fix the links manually for you in the meantime. I can't promise it'll work, but it's worth a shot.

Thanks Victoria!

I will definitely send you a zip shortly. And if you want to share any tips about how I can access the content of the catalogue database manually I would love to learn this. I'm pretty certain, which Cletus just pointed out, that this is related to the move between file systems. I took a chance and bulk renamed all ÅÄÖ characters to AAO in the Command Prompt interface , hoping Lightroom's "Find Nearby Photos" perhaps would trigger but no luck. So a pretty sure bet is that how Lightroom have ÅÄÖ represented in the database doesn't match the current file system. And the hope would be that looking in the .lrcat this could be fixed manually.

Best regards,
KM

 

[KMagnus]

Welcome to the forum. I agree with Victoria. But I would also. like to add that I think the issues is probably related to the Windows 10 filesystem (NTFS) incompatibilities with the MacOS filesystem (HFS+). I think the translation from HFS+ to NTFS came up with incompatible characters and the characters used in the NTFS file and folder names no longer agree with the characters stored in the LR catalog for those same files and folders. I've seen this problem before with similar situations and ASCII characters greater than &#127. (0-128)

Thanks for the advice Cletus! I definitely share your thoughts on why this error probably has occurred. I can't however come up with a smart way to solve it!

 

[clee01l]

I can't however come up with a smart way to solve it!

. Take a look at the Windows file/folder names as shown in Online Exif (Image Data) Viewer. Compare with the file/folder name as stored in the LR catalog. Look for a pattern. Search and replace might then be used to rename the files/folders in Windows explorer or John Beardsworth's Search Replace Transfer – John Beardsworth might be a solution to fix your LR catalog (Complex file renaming in Lightroom made simple(r) – Lightroom Solutions)

 

[KMagnus]

Made some progress with this and thought maybe you wanted some insight.

I used the Online Exif Viewer Cletus linked me to (thanks!) but couldn't find any clues there. Great tool though for learning more about how the actual photo and how it was taken.

But after some detective work I figured out Lightroom is using a SQLite format for it's database, and using a DB Browser (SQLite Browser) I started rummaging through it. WARNING: Wouldn't recommend this to anyone unless you've seen the inside of a large database before - if you're new to databases and just want to have a look then open your catalog in Read Only (and of course experiment on a copy of it - after making back-ups).

Digging into the AgLibraryFile table of the DB I targeted two files, both with "åäöÅÄÖ" in them. One I had fixed inside Lightroom by locating it and one still missing. At a glance, all entries concerning the file names of the two files looked exactly the same in the table. The "åäöÅÄÖ" were represented exactly the same. But looking deeper a discrepancy showed up. The working file had a data entry (baseName) with 21 characters, the broken one had 22. The difference was really visible when viewing the entry in binary format.

So, testing the theory I simply edited the entry and replaced the "å" with the same character "å", banking on that the correct ASCII would be entered. As soon as this was done, the amount of characters where the same. Saved the DB and loaded it up in Lightroom and voilá, the image popped up.

Now I just need to figure out a way to fix this problem in bulk in the database since doing this manually in here, fixing it for each DB entry, will take longer than just using Locate Photo in Lightroom.

// KM

 

[Wernfried]

You could try

update AgLibraryFile SET baseName = baseName;
or
REINDEX;
and/or
VACUUM;

All these command do not change any data content.

 

[KMagnus]

Thanks Wernfried. Didn't get the needed result with any of those though.

So I've been trying to make a simple code snippet in SQLite which will target the incorrect "å" and replace it with the correct "å". After a lot of trial and error (I'm no coder, just know some basics) I managed to figure it out. Since the database is encoded in UTF-8, I extracted the binary numbers for the bad character from the binary field in the DB editor, used an online tool (Unicode code converter) to convert this into Integers and used this in the Char() statement. Code looked like this:

UPDATE AgLibraryFile SET baseName = REPLACE( baseName, CHAR(97, 778), 'å');

And now it looks good. Loaded up the database and all images are linked!

There are a couple of more fields in AgLibraryFile which contains the file name so will replace here as well just to be sure. Then do the same for all special Swedish characters "åäöÅÄÖ".

Thanks for the help everyone!

 

[Wernfried]

CHAR(97, 778) means

• U+0061 -> LATIN SMALL LETTER A
• U+030A -> COMBINING RING ABOVE

In combination this gives "å", however there is a single Unicode character

• U+00E5 -> LATIN SMALL LETTER A WITH RING ABOVE

According to Unicode standard both representations are canonically equivalent. Usually an application should perform Normalization, i.e. transform from U+0061 U+030A to U+00E5 automatically. Actually that's what you did manually. Looks like under Mac there is a lack of functions regarding Unicode.

 

[KMagnus]

Thanks for that clarification Wernfried, really helps! Feels like this is something Lightroom should be able to handle on the fly TBH.