Import Why is "import new photographs" so hard?

[closeupandfaraway]

Lightroom Version Number

14.2

Operating System

1. macOS 15 Sequoia

LrC 14.2 MacOS Sequoia 15.4

Import (from SD card, or tethered camera, or tethered iPhone) is always such a chore!

I plug in the flash card, and choose the subfolder (I tend to start a new one monthly or so), and wait and wait (today, it's still updating the import preview 10 minutes later). The preview setting is set to "New Photographs", and the "Don't import suspected duplicates" is selected. Today, it shows me about 1400 photos that I already imported, but mysteriously doesn't show me the last 5 that were imported, nor the first 4 that haven't yet been imported. Why? I have to go manually pull the files of the CF card, or choose "Show All Photographs" and wait another 10 minutes to get the missing 4! Characteristic of the 1400 is that they were shot in Japan, and I'm now in California. The 5 that it decided it had already seen - captured without camera GPS, but otherwise no different. The 4 new ones that it failed to show for import - also captured without GPS, and before I had reset the camera clock to the PT zone, but NOT in the catalog already. Even changing the camera DST flag is enough to cause all previously imported pics to reappear on the import dialog.

And it's far worse importing from iPhone. It takes an hour to sort through all the previously imported pictures, and is kind of random on which ones it eventually shows. I end up using ImageCapture (which solves the same problem rather well), and manually moving the DNG files into my LR folder and syncing.

I've diagnosed one failure mode - if the pictures on the card were shot in a different timezone than the importing computer, they are intrepreted as new pictures, not previously seen, and LrC attempts to import them again (naming them with a -1 or -2 suffix). But I can't figure out why it's so slow, or why it's ability to identify existing and new is so poor. If I were writing the code, I would do a quick exclusion based upon a hash of filename and file DateTime field (that would quickly eliminate most previously imported from even being looked at, and then further qualify with a hash of all metadata. If the metadata hash were calculated once at import, and stored with the other metadata, then the whole "wrong timezone" thing would go away. And I have to suspect that whatever other heuristics are being used to determined whether it's been seen before would be improved too!

Am I the only one frustrated by this? It's been a thing for about a decade. It is minor, compared to the other performance challenges, but it's a stupid, easily fixable frustration!

 

[Gnits]

I regard the LrC import module as seriously constrained. I wish they put .01% of billions spend on Ai to adding real world intelligence to the Import Module.

I totally accept that it does an awful lot of stuff really well… but it was designed when LrC was at beta stage. It suffered a horrendous upgrade a few years later…. which had to be pulled within a few days and replaced by the original.

I have written my own ingest app so far in Python and MatLab . It ingests data from cards (or backed up from cards to disk) and copies intelligently to a user selected target folder. It is not a general purpose tool, not fit for public consumption, but it is my dream machine.

I rewrote it in the last 2 weeks in VB / Winforms. It took me approx 6x 30 min coding sessions… complemented by maybe 10 or so 15 min debugging / testing sessions. Most of the time I spent figuring out the syntax of the required vb statements.

I do not want to bore people with all the good stuff it does, but it does a lot. I am now using it for production jobs. It has a significant level of error checking built in and this will be reviewed further in the next few weeks. So it is not just a basic script.

My plan is to use it for a few weeks and then refine it to work under Parallels on a Mac (so I can use it when travelling).

I have tried so hard to get Adobe to spend a tiny portion of their LrC mtce budget to adding real world specific features… to no avail… so I built my own.

They key point I wish to make is if I can start with an empty app and get this working in such a short time, Adobe with their expert resources could upgrade this in a very short timescale. I understand testing large systems is not trivial ( I have designed, built and deployed such systems to very large organisations, so I know the effort involved).

And.. my frustration has lasted more than a decade. I identified the shortcomings in Lr Beta, but accepted then that a more rounded feature set would evolve in time. Unfortunately … not to be.

 

[Califdan]

If directly importing from the SD card is slow, the problem may not be with LrC. it may be a faulty card reader, a bad cable or a failing memory card (or it may be LrC).

Try this. Using Finder, copy the images from the SD card to a folder on your interhal HD. If this experiences the same slowness then the problem is in the card, card reader, card or cable. If it goes pretty quickly then try importing from this folder on your HD and see how that goes. If it moves right along then I'd just use this method of getting images into LrC and not waste time chasing down the problem with importing directly from the card.

However, if the Finder copy from card to disk goes well but the import from the HD does not, then there is a problem in LrC tht needs to be addressed. I had a similar issue a year or so ago where my import times had doubled. I finally realized that the RAW images from my new camera were over twice the size as images from my old camera (went from 20.2mpx camera to a 50+ mpx camera) resulting in over twice the time required to import. But, other than silly cuases like that, low amount of computer memory can cause such things, importing to an External drive on a slow port can do it as well (e.g. a USB 2.0 port rather than a USB 3.2 port). A nearly full destination drive can also slow things down quite a bit.

 

[Gnits]

I agree with all of the above.

Also if the catalog (and therefore the previews are on a slow disk, maybe external, maybe dodgy cable or port) then i/o building previews will suffer.

I built a high end machine to process 40MB raw images. When I upgraded to a 60MB camera my machine was literally reduced to spinning ball beach balls (windows).

I picked up the vibe on the post above that it was features / functionality of the Import module.

 

[closeupandfaraway]

If directly importing from the SD card is slow, the problem may not be with LrC. it may be a faulty card reader, a bad cable or a failing memory card (or it may be LrC).

Try this. Using Finder, copy the images from the SD card to a folder on your interhal HD. If this experiences the same slowness then the problem is in the card, card reader, card or cable. If it goes pretty quickly then try importing from this folder on your HD and see how that goes. If it moves right along then I'd just use this method of getting images into LrC and not waste time chasing down the problem with importing directly from the card.

However, if the Finder copy from card to disk goes well but the import from the HD does not, then there is a problem in LrC tht needs to be addressed. I had a similar issue a year or so ago where my import times had doubled. I finally realized that the RAW images from my new camera were over twice the size as images from my old camera (went from 20.2mpx camera to a 50+ mpx camera) resulting in over twice the time required to import. But, other than silly cuases like that, low amount of computer memory can cause such things, importing to an External drive on a slow port can do it as well (e.g. a USB 2.0 port rather than a USB 3.2 port). A nearly full destination drive can also slow things down quite a bit.

Yeah, it's not the card or the machine. I can (and often to) access the card directly (slot in the side of the Mac, not a separate reader) and copy the files to the (onboard) SSD in a few seconds. I just wish the basic functionality promised (load new pics) would actually work! Ever!

 

[closeupandfaraway]

I regard the LrC import module as seriously constrained. I wish they put .01% of billions spend on Ai to adding real world intelligence to the Import Module.

I totally accept that it does an awful lot of stuff really well… but it was designed when LrC was at beta stage. It suffered a horrendous upgrade a few years later…. which had to be pulled within a few days and replaced by the original.

I have written my own ingest app so far in Python and MatLab . It ingests data from cards (or backed up from cards to disk) and copies intelligently to a user selected target folder. It is not a general purpose tool, not fit for public consumption, but it is my dream machine.

I rewrote it in the last 2 weeks in VB / Winforms. It took me approx 6x 30 min coding sessions… complemented by maybe 10 or so 15 min debugging / testing sessions. Most of the time I spent figuring out the syntax of the required vb statements.

I do not want to bore people with all the good stuff it does, but it does a lot. I am now using it for production jobs. It has a significant level of error checking built in and this will be reviewed further in the next few weeks. So it is not just a basic script.

My plan is to use it for a few weeks and then refine it to work under Parallels on a Mac (so I can use it when travelling).

I have tried so hard to get Adobe to spend a tiny portion of their LrC mtce budget to adding real world specific features… to no avail… so I built my own.

They key point I wish to make is if I can start with an empty app and get this working in such a short time, Adobe with their expert resources could upgrade this in a very short timescale. I understand testing large systems is not trivial ( I have designed, built and deployed such systems to very large organisations, so I know the effort involved).

And.. my frustration has lasted more than a decade. I identified the shortcomings in Lr Beta, but accepted then that a more rounded feature set would evolve in time. Unfortunately … not to be.

Yes, that makes sense. Let us know if you publish it for use!
Did you think about writing in Lua as an import plugin using the SDK? Perhaps I will have a go at that!

 

[Gnits]

I thought about.. but not sure if I have the stamina to learn Lua. I suspect there may be some catches in terms of what is possible .. and will consider if it is a feasible project for me for the darker winter nights.

I cannot find a structured course to learn Lua inside the LrC dev environment and I cannot learn a language just by following some example scripts. If I have structured material I can learn a language (to a beginner level at least) very quickly.

 

[jrobnz]

I'm not sure about a structured course but there are pretty good language reference docs at https://www.lua.org/docs.html and the LrC plug-in SDK has docs on the API's available and language limitations they impose. That said, I have no idea if what you have done can be done within an LrC plug-in

 

[Gnits]

Two steps which improved my performance of LrC generally, but especially at import.

1. Turn off Spotlight (or at least exclude your catalog, preview folders and consider excluding your image repositary).
2. Exclude the same set of folders from virus checking.

At import your system is trying to copy cards to disk… an intensive task… often with a lot of large files. At the same time your system is trying to build a Spotlight index of all of this new content and at the same time your system is trying to check all this stuff for viruses.

Meanwhile your system is trying to build all the necessary previews (which also need to be indexed, not sure if these are virus checked) and update the catalog.