My lightroom catalogs are a disaster

[raerae3027]

I need some major advice.

As of right now I have 2 different catalogs. One is on my Mac desktop and one is on my MacBook. My catalog on my desktop is about 5 or 6 years old and my laptop is only a few years old. Each one is a mess because I made the mistake of moving things around outside of lightroom very early on. For each of my computers, when my internal hard drive would start getting full, I would move pictures off of my hard drive to an external hard drive. I would keep some pictures in lightroom and some I would remove completely. Picture locations right now are on my internal and external hard drive (for both desktop and laptop.)

I have recently been linking missing folders and pictures for both catalogs, so most are fixed.

A bunch of my pictures now are not in lightroom, and just sitting on my external hard drive. I have some free time right now so i want to get all those photos into lightroom in order to sort through, edit, delete, rate, etc. I want to eventually only have one catalog but worried about merging since each catalog is such a mess and picture locations are scattered. Plus, some pictures I know are located in both catalogs so not sure merging is a good idea. I have quite a bit of editing and metadata but I think I'd rather lose those instead of losing pictures in the shuffle of transitioning to one catalog.

I'm looking at a total of 70,000 pictures or so between the 2 so I really would like to be at just one catalog with all photos stored in one location on an external hard drive.

Any advice would be greatly appreciated!

 

[Tony Jay]

Hi welcome to Lightroom Forums.

I will tackle the salient points in an order that makes sense to me:

There is no intrinsic reason to lose any metadata, keywords, or develop settings in the exercise you are contemplating.

My suggestion is to pick a catalog that is going to become the new master catalog.
Tidy this catalog up first doing whatever rating, deleting, metadata acquisition and keywording is required.

I would suggest that it might be best to put all the images onto a single hard drive - probably an external drive that is large enough to hold the entire 70 000 image collection. It is possible to move the images already imported into the catalog to any location without losing any data as already mentioned above.
There are two ways to achieve this: one is less risky that the other.
The first method is done completely in Lightroom - just drag and drop images either individually, in groups, or in a folder from one location to another. It works, but the risk is that the process by which Lightroom does this is not a copy-and-move but just a move and so if the process is interrupted for any reason an image could be corrupted.
The second method is to actually use the OS to move images around. It is safer in that it is a copy-and-move process. The downfall is that is a bit more complicated in that once the copy-and-move has been done one needs to check that all the images have been successfully copied to the new location without being corrupted. then the images in the old location need to be deleted. After that Lightroom will complain that it has lost track of those images but is relatively simple to ask Lightroom to relink with the images in their new location.

A small diversion at this point is useful:
It is a good idea to do the process referred to above on a folder-by-folder basis.
It is also a good idea to keep the folder structure the same in the new location as it was in the old.
This makes the task easier for both a human (you) and the software (Lightroom) to keep track of where the images went.

However, it is also a very useful trick to have one's image collection in a single high level folder - call it something like 'my_images' with as many subfolders as required and whatever hierarchy makes most sense.
Most people have a folder for each calendar year that they have images for - '2015', '2016', etc - and then usually folders in each year folder that correspond to the date of a shoot '20160418'.
If one is really doing a lot of shooting in a year perhaps an intermediate folder level for each month might be helpful.
Since I am flat out doing one shoot a week on average through a year I have never found the intermediate folder level for the month to be necessary.

Once all the images in the the new master catalog have been sorted out and sit in the simple folder system described above and in one hard drive, now is the time to import images that are not in any catalog yet. Perhaps the easiest way is to copy them onto the hard drive where the main image collection is now residing and put them in date-based folders exactly according to the pattern already existing in that image collection.
Then ask Lightroom to import them using the 'Add' option in the import dialog. In this instance no images will be relocated.

Once the images are imported one can go to work rating, deleting, keywording etc according to whatever is important to your workflow.

All that should be left now is the images in the catalog that was not originally selected to be the master catalog. Go to work sorting this catalog out. Once it is in the same state as the master catalog on can import it as a catalog into the master catalog. No data should be lost in this process. It is designed to completely retain all and any metadata - nothing is lost.

By this stage all the images reside on a single hard drive and belong to a single catalog.

As an overview of the process what I have explained above works - I have done all of these interventions myself.
However, particularly with the size of the task ahead of you, and given the fact that you may not be that familiar with all the different actions required a divide-and-conquer approach will be needed.

First of all back up all files, folders, and perhaps whole drives before starting. This is your insurance if anything should go pear-shaped.
Secondly, create small test catalogs containing just a few image files in a couple of folders. If you need some images to play with take a couple of dozen shots of your big toe if there is nothing else for inspiration.
With the test catalogs check all the mechanics of the various interventions you are planning.
If you screw things up it doesn't matter.
Once you are comfortable doing things with the test catalogs then one can move on to doing these things for real.
Spend as much time experimenting as required to become really confident of the process.
Come back and ask more questions on the Forum if despite lots of experimentation things are not clear.
Do not turn your precious images into guinea pigs.
Also, it seems to me that this whole process might takes weeks - even if eight hours per day is employed - so take your time and get comfortable with what you are doing.
Do not rush the process and make mistakes.

In summary it must be stressed that this is an overview. If really required we can sharpen up the detail significantly. I don't really know whether what I have outlined is a simple recipe for you or whether it is totally incomprehensible. You will need to be the guide here.
Good luck and remember there is plenty more detail if required.

Tony Jay

 

[tspear]

Tony,

The OP will need to import the second catalog before importing the unknown images. Otherwise, the OP has no idea which images have been added already.

Tim

 

[Tony Jay]

Not necessarily - lets wait and see what the OP has to say first.

 

[raerae3027]

Thank you for such a detailed response. I am going to try to touch on each point you have made.

The way you are describing me going about this transfer does make complete sense. My worry though is because I will be running into duplicates, edited vs. un edited versions of pictures, etc.

For example, say I have a folder in lightroom, stored on my external hard drive. I have imported some pictures into lightroom, but I still have a bunch of photos in that folder that are not in lightroom yet. If I try to import that folder, it says "X amount" of pictures already exist in lightroom and cannot be imported. I do not want to overwrite the importing duplicates button because that will give me WAY too many duplicates in my catalog.
In lightroom, my external hard drive is shown with some of my folder structures exposed (the ones that of course have images in them that are in my catalog.) When I try to import the rest of the folder, it is leaving some images out of the import i think. Maybe because at one point or another those images have resided in my catalog (whether or now that have been removed since then.) Like I mentioned above, most images have been in lightroom and then I took them out and put them temporarily on the external.

Another problem I am faced with is from the same shoot (or trip) I have folders in each catalog.

I have most pictures backed up on an external drive as of last week. I also did Time Machine on each of my macs last week as well, both saved on external drives.

I am not sure what route I want to take as far as moving things around. I's like to say I want to do things outside of lightroom just to play it safe, I just worrying about the linking things back together and that being a whole other issue.

As far as my folder structure goes, right now I have year folders, and inside are folders named after event. I recently started doing it that way, because before I had only years and months and that got way too confusing.

One other thing I am curious if you know anything about is in my catalog, I have a folder named "Lightroom." Same goes for on my OS. In that folder, there is another folder "download backups." Not sure what they are or how they got there. Inside them are a bunch of subfolders name "Imported on May, July, Dec, etc. (And whatever year.) Some instances, pictures in those folders may be the only place they are stored on my OS. Not sure how that can be. Did I do something wrong at some point? Those folders were obviously not created by me.

I feel like I have a bunch more roadblocks that I haven't mentioned yet, once I think of them i will throw those out there as well. Thank you again for your advice. You have no clue how happy I am to know that my lightroom work is not completely doomed.

 

[Tony Jay]

With regards to duplicate images in a folder - why should there be duplicates?
Lightroom does not make duplicates when images are edited.
If you have made derivatives such as TIFF's and JPEG's from imported images there is nothing wrong with that per se however there is usually little benefit in having those images in the catalog unless they are the results of images round tripped through Photoshop and now represent the real 'master' image. In that instance those images will need to be imported.

If Lightroom knows that it doesn't want to import all the images from a particular folder (for the very sane reason that it already has some of them in the catalog already) I don't see that as a problem but rather as a blessing.
If there are any images that were imported into Lightroom but you have since moved the image to a different location using the OS causing Lightroom to lose track of it then looking at that image in the Library module it is easy to identify since it will have an exclamation mark on the thumbnail. In order to reunite the catalog with the lost image you will need to know where to look for it. Lightroom itself cannot help you here. I suspect that at least some of the images that Lightroom does not want to import fall in to this category. One does not want to try an re-import these images. Rather find them and re-acquaint the catalog with them - and since they are likely in the same folder from which you are trying to import images then the worst part of the process - that of finding the lost images in the first place is already solved.

I mentioned that this will be a slow process of divide-and-conquer, folder by folder. I don't see any alternative to crosschecking image by image to confirm that all that needs to be imported is actually imported. Also, take my advice and use the TEST catalog to recreate all your scenario's to allow you to gain the necessary experience and confidence to solve them.
Do this: import half a dozen images of your big toe. Take one or two and then remove them from Lightroom (this option does not delete the image on the hard drive). Then re-import them - there should be no issue doing this. Now take one or two of these images that have been imported and, using the OS move them into a folder close by. Check in the Library module. Those thumbnails should have the exclamation mark indicating that Lightroom has lost track of where the images are. L-click on the actual exclamation mark - Lightroom will generate a small dialog offering you the option of tracking down the image. Do this and lead Lightroom to the correct folder and then select the image. The exclamation mark immediately disappears. The only problem now is that the image is the wrong folder. So, now, using the OS move the image back to the correct folder. Once again Lightroom complains that the image is lost but now one knows exactly how to reconnect it.
Only you know all the strange contortions to which the catalog and image collection has been subject to. Recreate each issue in the TEST catalog and practise solving the problem. Given that you are doing this only a few images (and throwaway one's at that) it should not take too long to go through each one and figure out how to solve it. Making mistakes are part of the learning process and you should feel free to experiment liberally with those images without any fear.

As for the download back-ups the import dialog does have an option for creating a back-up of each download. I suggest that you have inadvertently checked this option without realising the implications. Unless there are image there (and nowhere else), for now you can ignore them).

With regard to folder naming - leave all your old folders named as they are for now. However, going forward try this: Use the date-based solution suggested but subsequently add other information such as the event in question like this: 20160423-sylvia_wedding. Doing this gives the best of both worlds. The date coming first will still allow each folder to list chronologically but there is also the suffix to remind one of what is actually in that folder.
(When you become more expert with Lightroom you will learn to use metadata and keywords to create collections and Smart collections instead to group images far more easily and logically than is possible using folders.)
Later, once you are a Lightroom master, if this is still important to you, go back and rename the old folders according to this pattern, but, for now, it is not important to waste your energy retrofitting folder names.

Let us know how you go.

Tony Jay

 

[raerae3027]

I am just now starting this process. I want to mention a couple things while I am thinking about it.

As far as folder structure goes, to organize to one sort of system is going to be difficult. Here's my dilemma:
On my iMac (which will be my new master), I have the folders organized some by this format "2011-10-20" and some in a 2012 folder which subfolder "7-12."
I made the huge mistake of changing my system at one point because I read somewhere one way was better....
And now of course with doing a bunch of lightroom tips/technique research online, event/trip based folder structure is best, so a couple weeks ago while in lightroom, I started renaming the folders (only the ones in lightroom, through lightroom) to the event/trip based structure.

Same goes for my mac computer.

The external hard drive has strictly only event/trip based structures.
Essentially, I have a bunch of folders with different names. Especially as I would COPY pictures onto the external hard drive from my imac, those pictures would go into a folder for example "Emily 3rd Bday" to which those pictures on my internal hard drive are still in the "2012-08-03" and that location is where lightroom knows about.

Also, last couple weeks I have been linking/moving folders around but ONLY in lightroom. So I am familiar with that process and was doing it the risky way, especially moving from one drive to the other.


I hope this all makes sense. I am going to start cleaning up my master computer now.

Thanks again for everything!

 

[Tony Jay]

I can only stress again - experiment with the TEST catalog - some of the things that you fear the most are actually easy to accomplish - you just need to gain the experience and confidence to do them!

Tony jay

 

[raerae3027]

Slowly but surely I have been consolidating my catalogs. (I actually had 4 instead of the 2, maybe 3 I initially thought I did).

I got all catalogs organized and upgraded to LR 5. I have deleted the older versions of them and they all are located in one spot on my hard drive now.

Now I have to go thru and start linking all the photos, which I am comfortable doing by this point. My process is going through each folder and looking at the file name, searching thru my OS to see where pictures/folders are now located, and re-linking. The problem with my older catalogs (maybe because of LR2), is I am unable to see the preview of the picture so it is a bit difficult to link some of the pictures. I think with the pictures I am unable to either find and/or link, I may just delete and once all 3 catalogs are gone thru, eventually merge.

Then once I have my 1 master, I will then start importing each folder individually from my external hard drive where most of my original images live.
Hope that all makes sense. Thanks again for the advice.

 

[Tony Jay]

Well done and good luck going forward.

Tony Jay

 

[raerae3027]

Just realized a key aspect that may potentially solve lots of my problems...

So i have many photos in my lightroom catalog that I also have saved another place on external (in a separate names folder.)

"Updating folder location" may be something I use to help with this transition...

I will let you know!