LR Classic: stalled import from SD cards

[BothHemispheres]

Lightroom Version

Lightroom Classic 10.2

Operating System

1. macOS 11 Big Sur

Hi All,

tl;dr LR Classic is stalling during import from SD cards :-( I start an import and, at some random time, it just...stops (progress bar stays still, no activity from card reader or destination drive).

I'm a timelapse photographer who shoots with 3-5 cameras, so I usually have to download between 2000 and 5000 raws after every shoot. For years my workflow has been the same: I plug all SD cards from a shoot into multiple readers, and set my photos to be imported overnight (it usually takes quite a while). But lately, I've been coming back to LR in the morning to find that it has stalled importing (just now I found it had stopped at photo 419, last time is was number 735), and I have to stop the import, and start it again from where it left off. Maddening, as I now I have to import until it fails, repeat and repeat, until all images are on my working drive and in the catalog. Can take a full day :-(

I haven't changed anything that I know of: I'm using the same SD cards as usual (I've tested several individually, same stalling problem), restarted LR and the Mac, unplugged and plugged in the card readers, and I haven't installed any plugins or anything else that would alter how LR behaves. Even a new test catalog has the same problem (and I saved it on a different disk too).

LR Classic 10.2
Camera Raw 13.2
M1 Mac Mini OSX Big Sur 11.2.3

Can anyone point me to a way to diagnose and, hopefully, fix this problem? I would be very grateful.

Cheers!

Chas

 

[Replytoken]

Have you tried to move some of the files (1k?) to an internal or external drive and try importing from there? And have you tried to import them into a new catalog? These two steps will help you isolate a few potential problem areas.

--Ken

 

[Gnits]

A few obvious things you have probably already checked... just to get the obvious eliminated.
1. Power Save Settings
2. Screen Saver Settings
3. Actual monitor power settings... are you also using an external monitor/s
4. Are the card readers plugged into a usb hub.
5. Is a local hub powered.
6. Is a local hub overheating.
7. Have you a new camera with bigger files that is now hitting some technical threshold.
8. Put a power monitor on your power supply to your computer
9. Have you some external power equipment which is causing a power glitch..It may not be your equipment / building. I experienced a crane in a local factory upsetting a computer system from the building next door.
10. Consider a proper designed UPS for your computer, disk drives,card readers, etc.

 

[BothHemispheres]

Have you tried to move some of the files (1k?) to an internal or external drive and try importing from there? And have you tried to import them into a new catalog? These two steps will help you isolate a few potential problem areas.

--Ken

Hi Ken, thanks for the suggestions.

Try a new catalog: yes, but try files from SD card --> drive --> LR import: no, so good suggestion!

But I think I solved my mystery myself, and it's a bit of an esoteric issue. I run a PC in parallel to my Mac, and I have software on each machine that lets me mount drives from each on each (NTFS on OSX, HFS on Windows). Last night's post here was prompted by yet another stalled import to a brand new drive (eliminating that variable), but I realised, after I posted, that that drive was formatted NTFS, not HFS. A quick reformat, and 7000 photos were successfully downloaded last night without a stall! So, not sure what the issue is with the NTFS drives and the driver, but I'm thinking that that is the problem (I'll be more diligent in downloading to Mac-native drives from now on).

No wonder I couldn't find a problem like mine in any of the fora I checked!

Thanks again for weighing in, I really appreciate it.

 

[BothHemispheres]

A few obvious things you have probably already checked... just to get the obvious eliminated.
1. Power Save Settings
2. Screen Saver Settings
3. Actual monitor power settings... are you also using an external monitor/s
4. Are the card readers plugged into a usb hub.
5. Is a local hub powered.
6. Is a local hub overheating.
7. Have you a new camera with bigger files that is now hitting some technical threshold.
8. Put a power monitor on your power supply to your computer
9. Have you some external power equipment which is causing a power glitch..It may not be your equipment / building. I experienced a crane in a local factory upsetting a computer system from the building next door.
10. Consider a proper designed UPS for your computer, disk drives,card readers, etc.

Hi Gnits, thanks very much for the comprehensive suggestions, I really appreciate the time you took!

I used to work in IT, so I can say I did consider all your points (including use a good UPS and a power monitor!), but I think I've solved my mystery myself. I'll copy here what I just wrote for Ken above:

I run a PC in parallel to my Mac, and I have software on each machine that lets me mount drives from each on each (NTFS on OSX, HFS on Windows). Last night's post here was prompted by yet another stalled import to a brand new drive (eliminating that variable), but I realised, after I posted, that that drive was formatted NTFS, not HFS. A quick reformat, and 7000 photos were successfully downloaded last night without a stall! So, not sure what the issue is with the NTFS drives and the driver, but I'm thinking that that is the problem (I'll be more diligent in downloading to Mac-native drives from now on).

So, I'm about to import 3000 or so photos from another shoot, and if that works, I'll consider the problem solved. But I really do appreciate the insightful suggestions.

Cheers!

Chas

 

[Gnits]

Good that you have solved the problem (hopefully).

I would consider the possibility of copying from cards to a folder on a network drive as a prelim step and then import to Lightroom catalog from there.... the advantage being that your Windows or Mac machines should be able to access the network drive, without worrying about disk formats. The downside is copying a lot of files to a network drive might be slower than you are comfortable with.

 

[BothHemispheres]

Good that you have solved the problem (hopefully).

I would consider the possibility of copying from cards to a folder on a network drive as a prelim step and then import to Lightroom catalog from there.... the advantage being that your Windows or Mac machines should be able to access the network drive, without worrying about disk formats. The downside is copying a lot of files to a network drive might be slower than you are comfortable with.

That is a good suggestion, though NAS vs DAS speeds are significantly different! I'll see if I can deal with the slowdowns with my next sequences (I'm out shooting again tonight, should only be 3000 photos or so, a good test amount )

That other 3000 photos I mentioned in my last reply to you? Imported without a flaw, happy to say.

 

[Gnits]

Yes, I was really surprised at how slow it its to copy data over a 1gb/s network. Max 125 MB /second , not taking into account network overheads, ... but if running overnight .. it might be useful. In the early days of available NAS drives, I installed (a then expensive SSD) into the Nas enclosure to try and speed it up. I would say my NAS was limited by its ability to write the data to disk but the network speed was the overriding bottleneck. I reverted to internal drives and used the NAS as a backup (and still do).

My biggest shock was installing SAN storage and upgrade backup system to reduce overnight server backups from 10 hours to 30 mins. Labour savings paid for the upgrade.

 

[BothHemispheres]

Yes, I was really surprised at how slow it its to copy data over a 1gb/s network. Max 125 MB /second , not taking into account network overheads, ... but if running overnight .. it might be useful. In the early days of available NAS drives, I installed (a then expensive SSD) into the Nas enclosure to try and speed it up. I would say my NAS was limited by its ability to write the data to disk but the network speed was the overriding bottleneck. I reverted to internal drives and used the NAS as a backup (and still do).

My biggest shock was installing SAN storage and upgrade backup system to reduce overnight server backups from 10 hours to 30 mins. Labour savings paid for the upgrade.

SAN is a bit out of my price range at the moment, but it'd be really nice (what with my mixed environment). And your working setup is similar to mine: large m.2 drive for working files and previews (external, but connected via thunderbolt 3), external spinning drive for less-important working files (and overflow), NAS for edited finals and backup.

 

[Gnits]

SAN was for a commercial system with multi million dollar budget. Just the shock of the actual performance gain ..... It provided options for almost real time disaster recovery options. Different needs.

I suspect the current M2 PCIe4 drive performance will be welcome by many , as it is no longer exotically priced. My initial test of a disk to disk M2 PCIe4 copy was also impressive. Will watch also as CFexpress Type A memory cards will improve speed of getting data from camera to disk.

I am convinced that Lightroom is not making the most of current hardware. Developing architectures, such as Apple M1, should provide a technology shock and encourage more innovation.

Anyway ... hoping your stalled import will go smoothly in future.

 

[clee01l]

Importing 3000 or so photos is a lot to ask the Lightroom import dialog. Lightroom uses lots of temporary files when importing. If I had to guess, I'd say the Working storage (primary HD free space) was getting saturated and when the system runs out of free space there is no room for more temporary files I recommend at least 100GB of free space at all times on "MacIntoshHD"

 

[BothHemispheres]

SAN was for a commercial system with multi million dollar budget. Just the shock of the actual performance gain ..... It provided options for almost real time disaster recovery options. Different needs.

I suspect the current M2 PCIe4 drive performance will be welcome by many , as it is no longer exotically priced. My initial test of a disk to disk M2 PCIe4 copy was also impressive. Will watch also as CFexpress Type A memory cards will improve speed of getting data from camera to disk.

I am convinced that Lightroom is not making the most of current hardware. Developing architectures, such as Apple M1, should provide a technology shock and encourage more innovation.

Anyway ... hoping your stalled import will go smoothly in future.

The closest I'll ever get to SAN is sneakernet of 512GB SD cards in the mail ;-)

I'm certainly one of those looking forward to leveraging the performance of the latest m.2 drives. CFexpress will also be impressive, but it'll be a while before I upgrade to cameras using that standard (when I'm not shooting timelapse, I'm a commercial photographer, and I don't need that level of performance there either).

Thanks again for all the insights!

 

[BothHemispheres]

Importing 3000 or so photos is a lot to ask the Lightroom import dialog. Lightroom uses lots of temporary files when importing. If I had to guess, I'd say the Working storage (primary HD free space) was getting saturated and when the system runs out of free space there is no room for more temporary files I recommend at least 100GB of free space at all times on "MacIntoshHD"

It may not be optimised for huge imports, but I've frequently had LR import over 10,000 photos at a time, and it's handled it just as well as imports of 50 photos, so I'm not worried there. As for my internal drive, I always keep tons of space free (my catalog and previews are actually sitting on an m.2 drive connected via thunderbolt).

If you read a bit up in the thread, you'll see that I've isolated the problem (importing to NTFS drives, instead of HFS/APFS drives). An esoteric issue, and I'm glad I realised that this was one variable I hadn't isolated. I've an email out to the support team who make my NTFS-on-Mac drivers.

Thanks for the tips though, I appreciate it.

 

[Gnits]

One last point, especially as you are dealing with high volumes of images.

Consider copying the cards to disk before importing to Lr, then use the Add rather than Move feature of Lr to bring them into the catalog. This means that Lr is not moving huge volumes of files while at the same time dealing with substantial volumes of database updates. [The downside is you may have to rename the files after the import process if you want different filenames].


I have developed my own injest app to copy from card to disk. This does a validation on the cards to make sure I have raws for every jpg and jpgs for every raw and summarises the number of images by date and by file type. I can then select all dates, a range of dates or selective dates with any combination of raw or jpg or both. [I may have different projects captured on different dates and want to direct them to different project folders]. Also, while travelling I will only import the jpgs to my laptop, but when I am home I want to only copy the raw files.

This functionality should be part of the basic Lr import feature set.

 

[Gnits]

Just saw your last post..... if it is working for 10,000 photos at a time and you are happy with your workflow.... why change...

 

[clee01l]

If you read a bit up in the thread, you'll see that I've isolated the problem (importing to NTFS drives, instead of HFS/APFS drives). An esoteric issue, and I'm glad I realised that this was one variable I hadn't isolated. I've an email out to the support team who make my NTFS-on-Mac drivers.

Thanks for the tips though, I appreciate it.[/QUOTE]

I never opted for NTFS formatted drives. It seems like an extra and unnecessary piece of software between me, the app and the data. ExFAT volumes are easily used by both Windows and MacOS.

I used to use VMware to run windows in a virtual machine. I stopped using Windows altogether because I can do all of my computing entirely in MacOS.


Sent from my iPad using Tapatalk

 

[BothHemispheres]

Just saw your last post..... if it is working for 10,000 photos at a time and you are happy with your workflow.... why change...

true, but alternative workflows are always interesting to examine. In LR's defence, I've imported close to 2 million photos in the last few years (no exaggeration), with very few hiccups during the import process (which is why this problem was odd, until I realised what the issue was).

Before lightroom came along, I used Photo Mechanic, and its ability to copy files to two locations was invaluable (in the studio, and in the field). I was happy to see that LR implemented this functionality (starting with ver1? I can't remember). If I move now to a "drag raws to drive, then import to LR", I'll lose LR's dual-copy function (unless, during a "Add", it'll also copy to another drive, I should check!). Maybe I should reconsider using Photo Mechanic prior to importing to LR (especially given the legendary PM speed).

 

[BothHemispheres]

If you read a bit up in the thread, you'll see that I've isolated the problem (importing to NTFS drives, instead of HFS/APFS drives). An esoteric issue, and I'm glad I realised that this was one variable I hadn't isolated. I've an email out to the support team who make my NTFS-on-Mac drivers.

Thanks for the tips though, I appreciate it.

I never opted for NTFS formatted drives. It seems like an extra and unnecessary piece of software between me, the app and the data. ExFAT volumes are easily used by both Windows and MacOS.

I used to use VMware to run windows in a virtual machine. I stopped using Windows altogether because I can do all of my computing entirely in MacOS.


Sent from my iPad using Tapatalk
[/QUOTE]

That's interesting info. I only moved to PC because my Mac Pro (tower, pre-trashcan) was getting long in the tooth, so a friend with a PC shop built me a PC that outclassed even the most robust Mac Pro at the time (and now my M1 Mac Mini is getting its own back). But I'm no Windows guru, so I formatted drives with whatever format Windows suggested: NTFS. I'll have to look into formatting drives as exFAT going into the future.

Thanks for the tips!

 

[Gnits]

PhotoMechanic is a good option to consider for copying from Card to Disk, especially as PhotoMechanic used to be able to have multiple card ingestions running in parallel (not sure now), and especially if you have a licence. I used to use PhotoMechanic to copy to disk until I wrote my own injest app. It is super flexible in terms of renaming files and folders. Occasionally I had an issue with PhotoMechanic when it lost the Unique Sequence number. The only reason I stopped is when I started to use my own app.

One think I am cautious of right now is with the speed of current processor and multi cores... I have a question in my mind in relation to the allocation of a unique sequence number (ie that it is in the sequence of capture date and time).

One reason I selected PhotoMechanic for this job is that it works on Mac and on Windows. I used a MacAir when travelling and a custom built workstation at home.

I think Windows insists that the system drive is NTFS, but prudent now to format all external Windows drives (and maybe Mac too, if a mixed setup) as exFat for Mac/Win flexibility.

 

[BothHemispheres]

PhotoMechanic is a good option to consider for copying from Card to Disk, especially as PhotoMechanic used to be able to have multiple card ingestions running in parallel (not sure now), and especially if you have a licence. I used to use PhotoMechanic to copy to disk until I wrote my own injest app. It is super flexible in terms of renaming files and folders. Occasionally I had an issue with PhotoMechanic when it lost the Unique Sequence number. The only reason I stopped is when I started to use my own app.

One think I am cautious of right now is with the speed of current processor and multi cores... I have a question in my mind in relation to the allocation of a unique sequence number (ie that it is in the sequence of capture date and time).

One reason I selected PhotoMechanic for this job is that it works on Mac and on Windows. I used a MacAir when travelling and a custom built workstation at home.

I think Windows insists that the system drive is NTFS, but prudent now to format all external Windows drives (and maybe Mac too, if a mixed setup) as exFat for Mac/Win flexibility.

Hi Gnits, sorry to take so long to reply.

If PhotoMechanic doesn't do multiple card readers, I'd be very surprised! (I see a lot of photojournalists around London, and they always have at least two cameras, and PM was originally built for photojournalists). I'll have to dig up my long ago license, and see if I can get a discount, as it's quite pricey now.

I'll trust that you are well informed re: processor speeds and multi-cores and how they relate to unique sequence numbers, as I certainly am not (and I do consider myself reasonably technical). Could this explain the occasional flipping I have of photos in timelapse sequences (photos that should be in chronological order, which is absolutely key, but sometimes they get slightly out of order e.g. 1, 2, 4, 3, 5, 6, 7, 9, 8, 10 etc).

Since getting my M1 Mac Mini, I've retired my PC from Lightroom work. I only use it for AfterEffects renders, and those from 16-bit TIFFs rendered by LR (and I'll start saving those over the network directly to the PC). I only have the software on both machines to occasionally read each other's drives (despite the root of my very original post, and now that NTFS drive is now safely back in the PC But I'll take on your and @clee01l's advice re: exFAT formatting, with much thanks.

​

 

[Gnits]

PhotoMechanic was built to deal with injesting multiple cards from multiple drives at the same time. Because, like you, I have unexplained images out of their sequence....... I have a question mark over the possibility of parallel processing during the copy to disk process (where my filenames are generated). I am not shooting volume so not a major issue for me, but because it is still unexplained it is an itch I have to scratch at some stage.


One way of guaranteeing they will be in the sequence as captured on your card is to rename using the following elements NNNN_SSSSS as part of the filename where NNNN is the sequence number from your card and SSSSS is a unique number per image within your catalog (the combination avoids duplicate file names and makes it easier to find images and derivatives of this image if you search for SSSSS in the Lightroom text search function). I am currently using the reverse SSSSS-NNNN) as this used to give me my images in date/time shot sequence.

I am a Windows user, but I am delighted to see the developments on the M1. Disappointed there was no further developments on this front in the last few days, as I thought an M1x or M2 would give some clue as to the evolving architecture, especially regarding the relationship between cpu cores and gpu cores. I have just custom built a maxed out Windows workstation... so no plans to change over to Mac anytime soon, but can see a use for the Mac mini for lots of uses (eg a music / media server) or a real machine to deliver internet content to the TV or just an email / browsing machine, although it clearly can do a lot more than this. It just fits into a living room, bedroom, office setting.

I am thinking of getting an iPad Pro for casual travel and wondering if there is a version of PhotoMechanic for the Ipad Pro. I like the idea of the TB3 /USB C port on the iPad pro and would use the Ipad pro plus an external SSD drive for backup in the field as it is lighter, slimmer than a laptop and I do not have to carry a power brick. Having the images on a TB3 ssd drive when I get home will also speed up the ingestion of images to my main workstation and file storage systems.