Catalogs Storing RAW photos on a large/not connected drive -experiences and suggestions?

[process]

Lightroom Version Number

6.14

Operating System

1. macOS 10.13 High Sierra
2. macOS 10.14 Mojave

I understand that it's possible to store RAW photos in LR "off-line" (i.e. to a drive which isn't permanently available) while still being able to view, edit (and export?) a high resolution JPG preview copy. It's been a while since I looked into this and back then I never seriously used it, so how does this all work?

I'm asking because my LR libraries take up well over 6 TB and since they're not business related I might as well be able to rely on a removable hard drive instead of the computer's internal drive, but what are the implications of this (compared to having the actual photos on an accessible drive)? What are the limitations/downsides?
I'm on Lightroom 6.14.

 

[Johan Elzenga]

You are talking about ‘smart previews’. If you generate smart previews of your images, then you can edit them while the originals are offline. Lightroom Classic (the successor of Lightroom 6.14) can export images as well, but only the size of a smart preview (2560 pixels longest side). I don’t think Lightroom 6.14 could already do this, but I’m not sure.

 

[process]

Yes, "Smart previews" was what I was thinking about!
2560 pixels isn't bad if Ijust need to occasionally share a copy with family/friends or for uploading to web forums etc.
I just tried to search the web for info if this applies to version 6.14 but didn't find anything -does anyone know? I did however read that it's possibly to "upscale" an exported version, but of course with lower quality, so that's probably something to keep in mind and avoid if possible.

With a disconnected drive, how does it work when I import new photos (RAW) from a camera? Do I just continue with the same workflow as now, then when the external drive is again attached the RAW photos are automatically moved over to the external drive? Or do I have to manually do this every now and then in order to free space off the Mac's internal drive again?
And how about MP4 video files and images that don't have RAW counterparts -do they all stay on the Mac's internal drive?

I organize all my photos in a really simple way (Hierarchical folders/sub-folders named by year/month/subject) and rely very little on Collections. This is because I first started off with Adobe's Bridge and did it this way, and also want to have a failsafe system which isn't totally reliant on a proprietary setup. It would be great if I could continue using more or less the same workflow as now, but without filling up the internal hard drive.

 

[Rob_Cullen]

SMART PREVIEWS ULTIMATE GUIDE
Importing new photos from a camera will require the external drive connected- if that is where you want the files to be stored.
You could import the files to the internal drive on a temporary basis and move (***in Lightroom folder panel) to an external drive at a later time when the external drive is connected. It is a manual process, nothing automatic.
MP4, raw, non-raw: All files must reside somewhere permanently, and it does not have to be the internal drive - so comments apply to all file-types that Lightroom handles.

 

[clee01l]

"Lightroom Smart Previews, originally introduced in Lightroom 5,..."

 

[Johan Elzenga]

"Lightroom Smart Previews, originally introduced in Lightroom 5,..."

Correct. My reservation was that I am not sure that Lightroom 6.14 will export from a smart preview, or that it will say it cannot export because the original is offline. I vaguely remember that exporting from a smart preview was added later.

 

[process]

Thanks to everybody for the comments!
Is it correct that Smart Previews are just for RAW files, or images in general? The guide linked earlier and other info online is a little unclear on this, but in case it's for all image files -is there any significant space saving compared to a JPG or PNG file?

Maybe I'm overcomplicating all this, but I'm concerned about how to configure the local (i.e. internal) drive's video files and other files I have related to my photos (more about this below*) as opposed to the same folder hierarchy on an off-line (i.e. external/disconnected) drive with the RAW files.

* Along with my photos/videos I also have "related" sub-folders containing stuff like PDF files, audio recordings etc. Since LR can't recognize/see these files they're neither previewed, nor are their folders seen in the LR hierarchy so I've solved this by creating some PNG files (as seen below) describing what's within, and of course keyworded appropriately so I can locate things easily.

So I want to access the contents of these folders like before and be able to view my MP4 videos, in addition to viewing Smart Previews of all the other images. In other words: the same functionality as before. And my setup is built upon a folder/sub-folder configuration like this:

2021/
2021-01/
vacation_family visit to grandparents/
vacation_boating trip at local lake/
387563.cr2
662942.cr2
367903.jpg
515783.mp4

2021-02/
2021-03/
(year-month folders for the rest of 2021 continued....)

2022/
2023/

So if I now want to transition from my current setup (a 6.5 TB folder on my internal hard drive) to a setup with Smart Previews for whatever I can, plus videos or whatever which needs to be available, is this more or less what I need to do?

1) copy the entire LR photo folder to an external drive (i.e. all my photos, videos, RAW files etc.)
2) on the internal drive: delete all RAW and other files that can have a Smart Preview
3) create Smart Previews
4) disconnect external drive

 

[process]

I guess what I'm envisioning is a "dual" system which combines two drives:

• External (removable) drive: all LR images and files
• Internal Mac drive: same as drive #1 except for RAW, TIFF and other large files (these are replaced by Smart Previews)

 

[Rob_Cullen]

Is it correct that Smart Previews are just for RAW files, or images in general?
Image files only! Includes RAW varieties, JPGs, TIFs, PSDs, PNG. But NOT videos or non-image files.

is there any significant space saving compared to a JPG or PNG file?
Depends, but most Smart previews are about 2MB in size.
Creating Smart Previews in no way will mirror your folders on a disconnected drive. Consider Smart Previews to be a 'Cache' of previews controlled exclusively by the LrC Catalog, and always in a file/folder adjacent to the Catalog file.

Maybe I'm overcomplicating all this,
I am sure! Because LrC does not reference PDFs, AIFFs, etc, you need to have a drive with these files always online when they are required.
So my workflow would be to have everything on a drive always available to the OS and LrC, then have a backup of the full drive.
My system uses a D:drive with {Documents}, {Images in structured folder hierarchy} and the {Catalog files} - and then easily create full drive backups.

 

[process]

I'm a little confused.
I understand how an external drive with a photo library can be disconnected then the Smart Previews "take over" when you view a photo, but when that drive is disconnected LR will no longer see any of the photo library "on-line" (attached), right?
So what I'm looking for is a way for LR to rightly notice that the photos are unavailable, but at the same time understand that the other files (videos etc.) are still permanently there. Is there a way to do this?

Alternatively I guess the only solution is to physically (file/folder) reorganize all my photos meaning all photos would be on an external drive and the other files within another "local" folder, but this would also mess up my entire file structure (which I deliberately chose to be "universal" (i.e. not dependent on a proprietary scheme such as LR's Collections) and work equally well within something like Adobe's Bridge, and it would also involve a huge amount of work.
If all else fails I could probably just back up the entire photo library as-is to an external drive, then on the internal drive convert all RAW files to JPG so I can finally delete all the RAWs and thereby save a huge amount of space.

 

[Johan Elzenga]

I understand how an external drive with a photo library can be disconnected then the Smart Previews "take over" when you view a photo, but when that drive is disconnected LR will no longer see any of the photo library "on-line" (attached), right?
So what I'm looking for is a way for LR to rightly notice that the photos are unavailable, but at the same time understand that the other files (videos etc.) are still permanently there. Is there a way to do this?

Lightroom will do this automatically. Look under the histogram. If it says ‘Original + Smart Preview’ then the photo is online. If it just says ‘Smart Preview’ then the photo is offline so Lightroom will use the smart preview if you want to edit it.

 

[process]

I figured I just had to try this out in real life to understand it better, so I created a test folder with a variety of file-types and sub-folders along the same structure as I usually have.
Once set up on my local (permanently attached) internal drive I quit LR and copied the entire folder over to an external (removable) drive, and told LR to use the new path of the folder. And sure enough the Smart previews worked as expected when I disconnected that drive.
Then, on the local drive I deleted all RAW files but kept the other files and the remaining file/folder structure, and added the drive to LR. So now I have a LR catalog with two drives (removable with ALL the files, and the internal drives with the exact same file/folder structure but without the RAW files).

This works fine, but.... it's cumbersome in use.
I would like to continue using the same file structure/functionality as before, but WITHOUT the RAW files on the local drive. In other words: not the two drives as shown above, but rather a merging into one drive, but "under the hood" LR knows that all the RAW files are on the external (removable) drive meaning they're available if attached, but if removed the Smart Previews will take over. All while the other files (PNG, TIFF, videos etc.) are accessable all the time from the internal drive, and all without the user (me) noticing any difference from the original setup (all files availble at any time, from the internal drive).
Can this be done?

 

[Rob_Cullen]

Yes it can be done as you describe, BUT-
- When you need to Export derivatives, or Edit in Photoshop, you will need the external drive attached.
- When you import new camera files you will need the external drive connected.
- You have a nightmare of a problem every time you import files to the external drive, you will need to manually set up the same folder structure on the internal drive.

 

[process]

Hmmmm... I see what you mean. A lot of things to keep in mind in order not to create a chaos.
But if I should decide to do it this way, can you please explain how it's done.

Alternatively, what would you suggest I do if I want to keep the current file structure (see post #8) but avoid having all the RAW files on the internal drive?
I can see how much easier all this would be if I'd structure things differently: keep all RAW files in one folder, then rely on Collections for the actual organizing, but I prefer using the other way because my organizing would still work if I'd move over to some other app than LR, if the catalog failed etc.

 

[Rob_Cullen]

Sorry for a late reply- been at a Photo Conference all over a 3-day weekend!
I keep thinking of your wanted file structure, but can not provide an answer other than to simply keep everything on the external drive. ie. all image files and accompanying PDFs, etc.
LrC has to make a reference to the original OOC files somewhere, and if that be an external drive then so be it, but if you want accompanying PDFs in with the OOC files then they also must be on the external.
You can have your LrC Catalog use Smart Previews when the external drive is disconnected, but comments in my Post #13 above still apply.
If you want the OOC files together with the PDFs, etc. the they must be on the same drive 'together'.-that's it, end of story!

"Collections" have no bearing the 'where' your files are located. Collections are only constructed lists of selected images. (Like a playlist in iTunes.)
Also "Smart Previews" do not have a folder structure that you can control! SPs are simply stored in a cache folder controlled exclusively by the LrC catalog.

 

[process]

Looks like I can't have it all
I think the simplest solution is to keep the file structure as it is on the external drive, with the RAW files and all.

Then copy it all over to the internal drive where I first convert all the RAWs to high resolution JPGs, and when done -delete all the RAWs.
That should allow me to access all the videos, PDFs, PNGs etc and also view all my RAW-source images, but taking up a whole lot less disk space.
Since I mostly view images these days this would make sense.
And if I should need to edit a RAW image I would just attach the external drive to access those.

With new images I'd import them (RAW files from the camera) as before to the internal drive, organize them as usual, but now and then attach my external drive where I'd copy over the new folders, then (when transferred and backed up to another external drive) I would once again convert the RAWs on the internal drive to JPG and delete all the RAW files there.

I suppose this (although sounding complicated when reading the above) might be the simplest solution if I want to "have the cake and eat it too".
What would be the simplest way to convert all the RAW files in a folder (or selected) to suitably high resolution JPGs?

 

[Johan Elzenga]

Then copy it all over to the internal drive where I first convert all the RAWs to high resolution JPGs, and when done -delete all the RAWs.

There is no need to do that. Generate smart previews for these images! Smart previews are about 1 MB each and they even allow you to make edits to the images when the originals are offline.

 

[process]

So, (on the internal drive) select all the RAW files in my entire catalog, create Smart Previews of them, and when done I can delete all the RAW files from that drive?

 

[Johan Elzenga]

So, (on the internal drive) select all the RAW files in my entire catalog, create Smart Previews of them, and when done I can delete all the RAW files from that drive?

No, I am talking about the images on the external drive. You can generate a smart preview from any original, as long as that original is present when you generate smart previews. So connect the external drive, select those images in Lightroom that you want to be available even when that drive is not connected, and generate smart previews for those images. Now those images can be viewed and even edited when the external drive is disconnected.

 

[process]

No, I am talking about the images on the external drive. You can generate a smart preview from any original, as long as that original is present when you generate smart previews. So connect the external drive, select those images in Lightroom that you want to be available even when that drive is not connected, and generate smart previews for those images. Now those images can be viewed and even edited when the external drive is disconnected.

But after reading Rob Cullen's reply (se quote below) I decided against a dual (external and internal drives) setup as this would further complicate navigating and viewing through my photos/videos/PDFs etc. (especially if someone else besides myself was to do it).

If I was to do what you're suggesting (create Smart Previews for the external drive) -won't those Smart Previews "belong" to the external drive regardless (i.e. the topmost "New S3 & S2" drive and its folder structure) as seen here, so if I navigate on the "Users" (internal) drive I simply won't see those Smart Previews for the RAW images (which have all been deleted there to save disk space)?

Here's what Rob wrote:

You can have your LrC Catalog use Smart Previews when the external drive is disconnected, but comments in my Post #13 above still apply.
If you want the OOC files together with the PDFs, etc. the they must be on the same drive 'together'.-that's it, end of story!

So in light of that I thought the best compromise (to the best of my knowledge) would be to keep a copy of the entire library on my internal drive, and after having created JPG copies, preview files or whatever works for the RAW files there, I delete the RAW files (and free up a huge amount of space) while still being able to view all those images (also along with the videos, PDF files and what have you, all within the same "Users" drive folder structure as before.
Naturally I won't throw away all my RAWs, but keep them all on an external drive (as well as an additional backup of that).

With that said, have I still misunderstood you, or is there indeed a way to create Smart Previews of the RAW images on the external drive, while still being able to navigate through them using the internal drive's folder structure (and simultaneously access the videos, PDFs and so on also on the internal drive)?

 

[Johan Elzenga]

Smart previews are created for images, not for drives. They are small proxies for your originals and give you the option to edit the images even if the originals are offline. If you store all your images on the internal drive after all, then generating smart previews for them does not make a lot of sense. Images on your internal drive can never be offline…

 

[process]

Yes, I understand about Smart Previews -that they're for images which are not currently present (i.e. on an external drive which isn't attached).
That is why I suggested (post #16) I create JPG copies of all my RAW images on the internal drive, then delete the RAW files there.
But to that you replied:

There is no need to do that. Generate smart previews for these images! Smart previews are about 1 MB each and they even allow you to make edits to the images when the originals are offline.

So are we talking past each other, or have I misunderstood what I should do in this situation?

Unless there's a better way to do all this, how would I go about to make JPG copies of all RAW files in a folder (including all of its sub-folders)?
I'd use the Library module's "Library filter" function set to "Metadata" for this (so I can select RAW files only), and normally when converting to another format I'd go to the Export (File-Export) menu or bottom-left Export button for this, but here I want to create JPG copies of all those RAW files to stay alongside wherever the RAW files are originally placed, and I really, really don't want to do this manually!
(NOTE: I said "how would I go about to make JPG copies of all RAW files in a folder (including all of its sub-folders)?" because I like doing large tasks like this step by step (i.e. one calendar year at a time) in case the computer crashes or something, instead of selecting all my RAW files in one go).

 

[Hal P Anderson]

If you want to replace your raw files with JPEGs and then delete the raw files, why not just create lossy-compressed DNG files to replace your raws? They are better than JPEGs for further editing and take up about as much room. You can do it in one fell swoop, too.

 

[Johan Elzenga]

So are we talking past each other, or have I misunderstood what I should do in this situation?

Yes, I believe we are talking past each other. The title of this thread is “Storing RAW photos on a large/not connected drive -experiences and suggestions?”. So you were asking about storing your original photos on a not-connected drive. In that case smart previews are the way to go, not JPEG copies. Smart previews are like raw files, so they are much better than jpeg copies. And any edit you make syncs back to your raw originals.

More recently however you said “So in light of that I thought the best compromise (to the best of my knowledge) would be to keep a copy of the entire library on my internal drive”. I assumed that means you are now considering keeping the originals on your internal drive after all (it’s always vague what ‘library’ means). But if you are not, then my advice to use smart previews still stands. Don’t delete your original raw files and don’t replace them by smaller DNG files either. One day you’ll regret that (for example now that denoise arrived, which only works on original raw files).

 

[process]

Glad we cleared up the confusion
Yes, initially I was hoping for a solution where the RAW files were stored on an external drive and simultaneously allowing me the same workflow, but as the discussion went on and I learnt more I came to the understanding that I would need to have everything on the same drive (videos, PDF files and so on) for this, so the solution changed. Sorry about the confusion.

I have no plans to permanently delete all my RAW files (and nobody reading should ever consider this either -especially in this time and age of cheap mass-storage drives), but for my internal drive I want to delete them (after converting them) to save disk space (ironically not because of cost issues but because the large drives I have attached internally are too noisy for my liking!). I did at one stage, years ago, look into converting them to DNG files, but voted against it -probably for the same reasons you state. I can't remember the details. The only change I've made to them are their filenames for better organizing.

If you want to replace your raw files with JPEGs and then delete the raw files, why not just create lossy-compressed DNG files to replace your raws? They are better than JPEGs for further editing and take up about as much room. You can do it in one fell swoop, too.

I took a closer look at this with a test library I just set up. I found out that with a couple of test RAW files (Canon .CR2) at around 27 and 30 MB I only got a saving of around 1MB when converting to DNG while converting them to JPG (75% quality) I got a 90% saving in filesize!
(EDIT: having re-read your reply I believe you suggested the above in light of permanently deleting my RAW files after converting them to JPGs, but fortunately that isn't the case -I just want to delete them from the internal drive. All my RAWs will still be safely stored on an external drive and an additional backup).

I'll be keeping the original RAW files on an external drive using the exact same structure as with the internal drive so I don't think there's much reason to convert to DNG in my opinion and JPG will suffice for browsing and viewing photos on the computer's screen (I opted for full size at 75% quality -please comment on this if I should consider other numbers).

So the next step is how to convert all the RAW files to JPG without too much manual interaction (we're talking thousands of RAW files) and without risking losing files or conversion issues. I searched for help and came across an article entitled How To Convert RAW To JPEG In Lightroom (Complete Guide) where I discovered that there's actually an option for saving the converted file within the same folder as the source, or even a sub-folder of the source's folder. This'll save me a LOT of time.

Having done the above (selected the RAW files, then Exported with the above settings with/without putting them in a sub-folder) I would probably just need to locate all the selected RAW files again in order to delete them, then right-click on the highest-hierarchy folder of my library and select "Synchronize folder" in order to add the new (JPG) photos and remove the RAW files from the library.
However, I still don't get an "error checking" option where I get to ensure that all the selected RAW files have had a JPG copy made of them and put in the right folders/sub-folders. Ideas anyone?