Massive Portfolio Storage for move from Film => Digital

[Bill McClatchey]

I am recently retired, 69yo, and have been an active amateur photographer for many years, starting with an Argus C3 at age 10 and maintenance of my own darkroom for over 50 years. My slide collection includes original Kodachrome, and many later Kodachrome and Ektachrome slides as well as a massive trove of B&W as well as color negatives, most in the 6x7 format. The quantity of images exceeds 20,000. I now use my Nikon D810, and for the past decade have used RAW digital as the primary image capture methodology, and Apple Photos as the primary digital processing engine. I still maintain a darkroom, although it is increasingly difficult to obtain appropriate materials for obvious reasons.

It is time to move to Lightroom, digitize and collate the entire collection.

I have started the scanning process using a commercial scanning service over the past 6 months, and am 75% completed (all to JPEGs), although the combined volume of files is pushing 450Gb, excluding any subsequent editing or metadata information.

My question for the Lightroom community is how to best store and access for Lightroom this volume of images.

I suspect I will have to balance the virtues of online storage with ICloud, Dropbox or AWS (availability from multiple machines, access from any location, reliable third party backup maintenance) with its downsides (slower access, at least for Dropbox a mirror of the data on my local machine, modest cost, lack of local control). Local access via hard drive and thunderbolt or USB3 access likewise has it virtues (rapid access, local control, mirroring on multiple drives for backup) as well as its problems (minimal multiple machine access outside of the LAN, difficulty or impossibility of moving data to use at other locations).

My primary machine is a 6 core Mac Pro 2013 with 64Gb of ram, but I also travel frequently with my 15" mid-2015 MacBook Pro retina. This leads to further complications in Lightroom for the location of storage of the Lightroom edit files, assuming a single location for the original image files.

And so, as I begin a likely multiyear process of digitizing and collating this collection with Lightroom, I am in a quandary trying to determine the best way to store and access the images, all the while realizing that the decisions which I make now will likely be very difficult to unwind in the future if they are not made wisely.

Can anyone on the forum advise me on the best way to store this quantity of images going forward based on their experience of facing the same issues?

Thanks!

Bill

 

[babuja]

I am recently retired, 69yo, and have been an active amateur photographer for many years, starting with an Argus C3 at age 10 and maintenance of my own darkroom for over 50 years. My slide collection includes original Kodachrome, and many later Kodachrome and Ektachrome slides as well as a massive trove of B&W as well as color negatives, most in the 6x7 format. The quantity of images exceeds 20,000. I now use my Nikon D810, and for the past decade have used RAW digital as the primary image capture methodology, and Apple Photos as the primary digital processing engine. I still maintain a darkroom, although it is increasingly difficult to obtain appropriate materials for obvious reasons.

It is time to move to Lightroom, digitize and collate the entire collection.

I have started the scanning process using a commercial scanning service over the past 6 months, and am 75% completed (all to JPEGs), although the combined volume of files is pushing 450Gb, excluding any subsequent editing or metadata information.

My question for the Lightroom community is how to best store and access for Lightroom this volume of images.

I suspect I will have to balance the virtues of online storage with ICloud, Dropbox or AWS (availability from multiple machines, access from any location, reliable third party backup maintenance) with its downsides (slower access, at least for Dropbox a mirror of the data on my local machine, modest cost, lack of local control). Local access via hard drive and thunderbolt or USB3 access likewise has it virtues (rapid access, local control, mirroring on multiple drives for backup) as well as its problems (minimal multiple machine access outside of the LAN, difficulty or impossibility of moving data to use at other locations).

My primary machine is a 6 core Mac Pro 2013 with 64Gb of ram, but I also travel frequently with my 15" mid-2015 MacBook Pro retina. This leads to further complications in Lightroom for the location of storage of the Lightroom edit files, assuming a single location for the original image files.

And so, as I begin a likely multiyear process of digitizing and collating this collection with Lightroom, I am in a quandary trying to determine the best way to store and access the images, all the while realizing that the decisions which I make now will likely be very difficult to unwind in the future if they are not made wisely.

Can anyone on the forum advise me on the best way to store this quantity of images going forward based on their experience of facing the same issues?

Thanks!

Bill

Hi Bill,

20k isn't a big deal for lightroom so not a problem. I have almost 30k and no issues whatsoever.

Regarding image/photo storage you could use a fast external drive so you can use it in both laptops as you wish. You can use 1 main catalog in you mac pro and a second catalog in your 15" retina (and import/export from catalog) or you could use a single catalog, synchronized between laptops...not sure what's the best, never test it.

Apart from the external drive where you store your photos and backup lightroom catalog and settings you should find a backup solution. This is my backup workflow...i use a NAS, 2 external drives to backup the NAS and a could service.

There are so many different products, softwares, solutions available it's difficult to find the best...just start with one that suites you and learn, read and test...and with time you will evolve to new and different solutions, i believe.

Sent from my Nexus 6P using Tapatalk

 

[tspear]

A few comments.

• Skip Dropbox and AWS. They are too expensive.
• If you want to use a cloud sync between computers, Microsoft OneDrive (get as part of Office, even cheaper) or Google Drive are the best/cheapest option. Note: this is not a backup solution.
• Use crashplan, or some other backup solution. Timemachine on the Mac is a great local solution, but an offsite solution is highly recommended.
  
• If you go with a drive sync solution, you must verify the drive has finished syncing between computers before you switch. So if you normally only work on the desktop, but then go to the laptop for days/weeks at a time. This works well. If you are switching back and forth every day, this is likely a bad idea.
• If you want to constantly switch between computers, you need to think through workflow. This can be done via a travel catalog or via placing the master catalog on an external drive or in a limited manor on the web. There are pluses and minusses for each approach. The web solution is by far the most limiting in terms of the feature set. But if you are mostly doing meta-data tagging, a few people have found this sufficient (not me).
  
• Also take a look at John Beardsworth Smart Workflow. You can then create work folders for each batch of images as you move them through the workflow. This has been discussed on here multiple times, including the changes I made, Cleetus and others.
  
• It is very easy in Lr to create a publish to hard drive. And this is what you share with others.
• There have been multiple threads on here about scanning. Mostly dealing with date issues, it is worth reading up on it some.

Let us know if you have questions about any point.

Tim

 

[Linwood Ferguson]

My primary machine is a 6 core Mac Pro 2013 with 64Gb of ram, but I also travel frequently with my 15" mid-2015 MacBook Pro retina. This leads to further complications in Lightroom for the location of storage of the Lightroom edit files, assuming a single location for the original image files.

Decide how important it is to have access to your collection IN lightroom to edit when travelling. If you can just give up on that, it makes live vastly simpler.

Note that's a separate question from shooting while travelling and editing (in Lightroom) what you shot, then merging it back in when you return.

But if you keep your total collection only one one system and edit only on that system, your technology world is a lot simpler and less error prone. For some that's not practical. For others travel-with-archive is more a "that would be cool but not sure when I would use it" type thing best left unexplored.

 

[clee01l]

In support of what Linwood has said. Keeping the Master catalog on one computer with the catalog file in a location to optimize performance (i.e. on a local drive) is always the preferred workflow. I think that you will find that most people that use two computers will be using a workflow similar to mine:

• Lightroom catalog on the primary drive (i.e. in your case Mackintosh HD).
• Recent images on a fast locally attached drive (buss mounted or Thunderbolt, USB3 if internal storage is limited)
  • Recent images are active and in a current develop state.
• Once the develop part of the workflow process is complete, older images can be moved to a slower connection (ethernet or USB2) if space on the fasted drives is limited. LR does not use the original except to generate a new developed image or for publish/export or printing. These last two processes do not require a high bandwidth connection.

For traveling with a separate computer, Use the Export as Catalog function to create a subset of your permanent inventory that will actually be used during travel. For me, most of the time I travel with initially an empty copy of my master catalog to which I import new photos that I take while on the trip. Upon returning, I use the Import from another catalog to merge the travel catalog file into the master catalog file and to move the travel images to the permanent location on the main computer.

 

[LouieSherwin]

Hi Bill and welcome to the forum.

Physical storage is not going to be a problem. I currently use a single 3TB hard drive to store 80k+ raw images and have over 50% free space left. Your 20k JPG scan files are not going to take up nearly as much space as your 40MP raw files. Going forward since a 4TB drive is so in expensive you might want to consider TIFF for you scans if your prefer.

My suggestion is to follow the KISS adage, "Keep it simple S....".

A big drive for your images (2-4TB), a bigger drive for Time Machine backup (4-6TB) and a smaller drive for a bootable clone of your system drive (1TB 2.5 inch external or internal in a 3.5" carrier) . And finally some kind of offsite storage for disaster recovery, fire, flood etc.. I would not recommend any of the "free" cloud based storage solutions for anything except sharing with your friends and family. All of these services mine your data for their own purposes.

The easiest offsite storage is going to be a cloud based backup. CrashPlan has gone out of the consumer business so I think the best bet is BackBlaze. Once you subscibe to the service and install and configure the software the backup is automated so you do not have to remember to do something and it can be encrypted so that no one but you can read and use the metadata.

One idea, since your MacPro has Thunderbolt, is to consider getting a Thunderbolt external storage system that holds multiple hot swappable drives. This can dramatically simplify your desktop and cabling. A good option could be T2-QuadMini or T2-Quad from MacGurus. With one of these you could have your image data drive in one slot, Time Machine drive in the second slot and a bootable clone in the third slot with one spare slot.

For travel do you really need to take your whole catalog with you? If like me you are taking multiple short (several weeks at a time) a year taking my whole catalog is overkill. So in my case I take a copy of my most recent catalog including previews on to my laptop. This is so that I have access to my current keywords and Collections. I only take originals and/or Smart Previews if I know that I want to or need to edit them while traveling. For instance attending a workshop on Lightroom editing.

I also take a single external hard drive for backups. I recently upgraded from a 1TB FW800 to a 2 TB Thunderbolt external HD. My traveling workflow is to import all the images taken during the day into Lightroom. Backup the Lightroom catalog, backup my whole laptop to the external HD and then finally erase the previous days memory cards.

An important note regarding managing scan captures in Lightroom. Lightroom by default will use the date the scan file was created because the is no metadata to tell the scanner when the image was created. With out taking some additional steps this will mean date sorting will be confused. i.e. your scans will show as being later than your digital captures. There is a plug-in that will make this easy to correct. Installing and using Capture Time to EXIF will allow you to easily update the internal dates of the scan files so that the sort in correct chronological order in your Lightroom catalog.

One final note is that you might want to consider making your own scans of your film/slide archives. Nikon has just announced a new slide/film duplicator attachment. Combined with a macro lens and a flash you can create digital scans of slides/film as fast as you can load each frame. For a complete tutorial on how you can do this check out Digitizing Your Photos with Your Camera and Lightroom.

-louie

 

[Tom75]

One idea, since your MacPro has Thunderbolt, is to consider getting a Thunderbolt external storage system that holds multiple hot swappable drives. This can dramatically simplify your desktop and cabling. A good option could be T2-QuadMini or T2-Quad from MacGurus. With one of these you could have your image data drive in one slot, Time Machine drive in the second slot and a bootable clone in the third slot with one spare slot.

Hi Louie,

I have done quite some reading and research the last 2 months because I need to replace my late 2012 iMac in the near future and I am looking for an optimal storage and backup solution because the new iMacs dont have that big internal discs anymore my plan is to go for several external discs for image storage and backup.

I have currently an iMac (the first 27 inch model with thin screen desing as they are now) with 3 TB internal fusion drive or disc. I am storing all my images and cataloges etc on this internal iMac disc and I am backing everything up to my 3TB time capsule with time machine. I have basically been happy with this system but I noticed that the new iMacs have now only 1TB or max 2TB internal discs, not 3TB or more. I find this a little strange and it means that I will have to change my system because 2 TB is not enough if I switch to a new imac, I would need minimum 3-4TB. If I am going for external discs for image storage and backup then a 1TB iMac would be enough.

So my original plan was the following.
- Buy a new iMac with 1TB internal disc which will be used for documents and LR catalogue etc
- Buy two 4TB external hard discs where one of them will be used only as a storage disc for my image files and the other will be used as backup disc for both the image disc and the local iMac disc via time machine.
- This means that it would be nice to have at least for the image storage disc one that has quite high transfere speed since I will work with the images from this disc. I guess for the backup disc it is not so important to have high speed.

1. perfect solution:
I thought to do the above the most perfect solution would be to buy a 4TBThunderbolt 3 or USB C disc as backup disc such as Lacie D2 or G-Drive and a image storage disct a 3-4 TB SSD for best performance and speed. But unfortunately this solution is simply too expensive because I will not spend several thousand $ for such as big SSD.

2. second best solution:
So the next option is to buy two 4TBThunderbolt 3 or USB C discs, I think both USB C or Thunderbolt 3 will be fine as image storage discs transfere speed-wise?

I considered going for option 2 but now my colleague asked me why I am going for a solution without at least a Raid1 backup system, so now I was not sure anymore. Because then I would need to buy sort of a 2-disc dock for the Raid system and in addition another HDD for image storage.

Now the reason why I answered on your post above is because you mentioned these "T2-QuadMini or T2-Quad from MacGurus", I looked into this and found this actually a very interesting solution. They have now something called "T3-QuadX - Thunder3 Quad X 4 drive Thunderbolt3 Enclosure " which is a 4 drive Thunderbolt 3 system. If I understand this correctly based on the information on Mac gurus web site and your explanation, I could do the following:

3. very interesting solution
- Buy this "T3-QuadX - Thunder3 Quad X 4 drive Thunderbolt3 Enclosure " with 4 discs of 3-4 TB.
Thunder3 Quad X 4 drive Thunderbolt3 Enclosure - MacGurus
- I could use one slot as image storage disc and 2 slots for the Raid 1 backup system and would have 1 spare slot for whatever.

The thing I dont fully understand form Mac gurus description is this:
"I f you want the highest speed, stick a couple of SSDs in and RAID0 them. You can easily get up around 800MB/sec just using 2 SSDs. The trays in the Thunder 3 are built to accommodate either 3.5 inch drives or 2.5 inch drives and SSDs. "

Does the above mean that I could theoretically put an SSD into one of the slots or how does that work?

Anyway I am really glad I found this Mac gurus this based on your above post I think I might go for such a device.

Regards,
Tom

 

[LouieSherwin]

Hi Tom,

Something to keep in mind is that while they share the same cable and connectors there is a big difference in Thunderbolt 3 and USB-C devices.

• Thunderbolt is an extension of your computer's internal system bus. What this means is that what we used to think of as external (read slower) devices are now able to operate at the same speed as "internal" devices.
• Each Thunderbolt port is also able to daisy chain up to 6 devices including additional displays.
• Once you plug a USB-C device into a TB3 port you loose all the additional capabilities of TB3 and are using USB 3.1 only.

For a more complete overview take a look at the following: usb-type-c-thunderbolt-3-one-cable-to-connect-them-all

For things like hard drives I would recommend that you stick to using Thunderbolt 3 devices.

For image storage in Lightroom you probably won't be able to tell the difference between a hard drive and an SSD except in your pocket book. If it is a external TB3 drive you probably won't be able to tell the difference between it and your internal drive at least in daily usage.

I would avoid RAID configurations unless you have a specific problem that only RAID will solve. It only adds complications and another possible thing to fail. You are better off investing in setting up a robust system for backups.

-louie

 

[Tom75]

Thanks a lot Louie for the valuable information.

there is obviously a price difference between USB C and Thunderbold 3 devices ..... and as you mentioned an even bigger difference to large SSD's. Its very good to know that Thunderbolt 3 will be suitable so I dont need to feel bad for not buying an SSD.

Regarding Raid stuff:
My goal is to set up a robust system for backups, I just thought or got the recommendation that a raid system would be even more safe. However I have no intention to have my images I am working with on that raid system, this will only be for backup. I am for sure planning to use a separate disc as an image storage disc, that why I thought I will now need 3 discs, 2 for the raid 1 and 1 for for image storage.

Anyway I like the idea of this "T3-QuadX - Thunder3 Quad X 4 drive Thunderbolt3 Enclosure " I found because of your recommendation, I think this will be a safe thing for many years to come if I go for this because I wont need all the slots for now. But I wanted to ask if you have such thing? It gives the impression of being quite solid and thought through and I like such stuff. But they are offering this system with discs for all slots, meaning all or nothing but I dont need all 4 right now, only 2-3 and I was also wondering why it can only be equipped with similar capacity discs and not different capacity such as oen 4 TB and 2x 3TB?

Regards,
Tom

 

[Philippe Coudé du Foresto]

Keep in mind that (whatever type it is) is for reliability (against disk failure) and (depending of the type) speed, not for backup.

A real bakup means several copies of your data on different devices (ideally at least 3), amongst them at least one is outside.

 

[LouieSherwin]

Tom,

You can purchase individual drives of any size elsewhere on their site under the "Storage" section but you do have to purchase the empty trays as well. It s been a while since I had to purchase drives but I found that the Seagates while highly reliable always were quite noisy when placed in my office. The Toshiba and WesternDigitals on the other hand are just as reliable but nearly silent when running.

RAID 1 only protects you from failure of one drive. Implementing RAID also involves additional setup of the RAID software. I know that in the past Mac OS has supported several implementations of software RAID including RAID 1 (mirroring) for internal drives in the Mac Pro towers. I have no idea if you can do this using TB attached drives.

To echo Philippe, whether or not you have implemented RAID 1 you are still subject to catastrophic loss due to theft, fire, flood etc.. The only protection for this is some kind of offsite backup. The best option, I believe, is a cloud based backup such as BackBlaze. I personally consider that more important than mirroring.

-louie

 

[Tom75]

ok, thanks lot guys for the information.

it seems then I will be fine with 2 Thunderbold 3 drives as I had planned for the local backup and storage.
I will look more into the T3-QuadX and get in contact with them.

It is also good to hear that Raid 1 etc is not really needed after all, good to get sometimes a confirmation because it was originally not my plan to use such a system but my colleagues really wanted to push me to it.

I know I will also need an external backup, I know how important that is... you never know what happens.

Anyway thanks again, I think I have a clear plan now.

Regarding the discs, I was considering Lacie or G-drive. I know seagate discs are supposed to be very good but thanks for the tip with the noise, have to look into that. Western digital I will not go for due to bad experience.

Regards,
Tom

 

[Roelof Moorlag]

Maybe it's good to know that Peter Krogh wrote a very sound book about this: Digitize Your Photos with Your Camera and Lightroom