• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • The November 2019 Lightroom updates are now available. Lightroom Classic includes the ability to export the same photos as multiple sizes/formats in a single export, content aware fill for panorama edges and more. It does require a catalog format upgrade and there are updated system requirements, so you may want to wait a few days before upgrading.
    Lightroom (cloud-based service) has export options (such as TIFF format, renaming and color space), custom sort order on the desktop, copy/paste to multiple photos on iOS and more.
    The Missing FAQ and Edit Like a Pro eBooks are already updated for the changes.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom 6 questions. All the information you need is in Adobe Lightroom 6 - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. When you upgrade to subscription, there's also a Lightroom Classic version available.

Help Me Get Out Of Lightroom Hell. Please...

Sprocket

New Member
Joined
May 19, 2011
Messages
21
Lightroom Experience
Intermediate
OK. I have to say Lightroom wouldn't be my first choice of management software but it's what my client wants to use and I'm about at my wits' end. Here's the situation:

  1. 29 external drives from 250Gb to 3Tb - 4 are USB3 and the rest USB2 and/or Firewire
  2. 5 formatted on a Mac, 23 are NTFS (PC) format and one with The Click Of Death which is probably headed for the bin.
  3. About 9Tb of (irreplaceable) files: JPG; CR2; Mov; and various other video codecs. Mostly still images - somewhere between 400 and 500k to date.
  4. The image and video files are duplicated at random over 24 of the drives with originals, backups and backups of backups
  5. Some have been renamed in a previous attempt at cataloguing (using Aperture, I think) and there are also many dupe file names since for a few years the author shot RAW+JPG
  6. My gear: i7 Pc with 12Gb RAM. Win7 x64

I already made a catalogue of 10 of the 13 years with the recommended Master Folder/Year/YYYY-MM-DD structure and the photographer asked me to re-name files in a Date+Capture Time format. Unfortunately, some of the drives were missed and (even more unfortunately) LR's renaming proved very shaky - in particular in identifying duplicates and also where images were shot in burst mode as it appears not to read data in fractions of a second. As a result, I ended up with having to identify and remove some 50,000 duplicates but in retrospect should really have started over and kept the original file names. (Why, oh why doesn't LR store the original camera filename somewhere in the Metadata and offer a simple "revert and check for duplicates" option?) As you can imagine this has taken a huge amount of time as moving all the data around at USB2 speeds is pedestrian.

I can also tell anyone who's been wondering that catalogue sizes of 200,000 images plus are slow, difficult to manage and I suspect unreliable with supposedly simple tasks like checking catalogue integrity taking a very long time. Files often show as missing even though they are in the correct place.

Nearly finished....

So now I have a 2.6Tb catalog of part of the archive, some of the files in which have been re-named and around 12Tb of data which almost certainly contains some valuable new images that need to be added along with a bunch of dupes that don't. Starting over seems attractive as I might face less dupe issues but the downside is that I don't have an empty 3Tb drive handy - there's always something! Your thoughts, then:

  • Have I missed some ability in LR to revert to the in-camera filename?
  • Start over? If so, what size of catalogue(s) to choose this time?
  • Is there a way to avoid importing the JPG files that were shot along with the RAW, given that there are often RAW+JPG pairs, RAW only and JPG only in the same folder? I know you can stack the thumbnails but also want to avoid having LR count the pair as one because it makes a rough check of the catalogue by comparing stats from Windows or FileTree impossible.
  • I'd prefer to avoid sidecar files as my client's computer attention span is limited and I'm afraid they will get detached before long. Embed Metadata is an OK option (I see posts here suggesting sidecar in response to someone else in a mess)?
  • Optimum catalogue settings advice from anyone with a similar sized archive would be appreciated.
  • Any other useful tips?
 
Last edited:
Joined
Oct 8, 2007
Messages
5,130
Location
London
Lightroom Experience
Power User
  • Have I missed some ability in LR to revert to the in-camera filename?
  • Start over? If so, what size of catalogue(s) to choose this time?
  • Is there a way to avoid importing the JPG files that were shot along with the RAW, given that there are often RAW+JPG pairs, RAW only and JPG only in the same folder? I know you can stack the thumbnails but also want to avoid having LR count the pair as one because it makes a rough check of the catalogue by comparing stats from Windows or FileTree impossible.
  • Optimum catalogue settings advice from anyone with a similar sized archive would be appreciated.
  • Any other useful tips?
From what you say, I don't think you are too far off-track. It sounds like you're approaching it methodically, but there's only so much that a DAM app can do in such situations.

As an overall strategy, I'd try to bring every picture file into one catalogue. Once all the files are recorded in one place, it's a lot easier to see what's going on than when you have some files in LR, some not, some maybe.

Let LR look after Raw+JPEGs, then sort out duplicates manually - if at all - and do it over time rather than in a big bang. There is a setting in Preferences / General to treat JPEGs next to raw files as separate files, but I normally leave this unticked, meaning Raw+JPEG shooting shows only one item. I can then be confident any JPEGs of the same image are derivatives or duplicates.

A second overall objective is to ensure all new work is properly organized and doesn't add to the existing problem. I guess you're onto that already?

I'd only rename after the whole thing has stabilised. Easy to be wise after the event, eh? But to revert, look at Library, Rename, choose preset Restore Original Name. Basically, if you rename a file in LR, the Metadata panel's Default view shows the previous file name, and this is the one used by the restoration preset. Filenames are a pretty flimsy organisational tool though.

In terms on catalogue size, I work with someone whose catalogue is over 450k and growing at 2-3k a week. Hardware is not extreme - Win7/64, i7/930? 12gb RAM, 6 or 7 Drobos. It's kept on a fast internal drive with a lot of spare space, just in case. We back up the catalogue every evening, using True Image, and do a LR backup which includes integrity checks once a month, plus a File > Optimize Catalogue. Sure, these routines take time. But when you're dealing with such numbers and there's a sense of chaos, I wouldn't want to use more than one catalogue - why fragment whatever control I've gained?

From what you say, I doubt you should start over. But if you do go that way, get that 3Tb drive if you need it. Space is cheap, and mistakes happen when you're jumping through unnecessary hoops.

Good luck

John
 
Top