1.000.000 photo organise - merging 2 catalogs - LR can't find photos after big upgrad

Status
Not open for further replies.
B

bskyum

Member
Joined
Jun 21, 2011
Messages
35
Lightroom Experience
Intermediate
This may be the biggest challenge to the experts so far in this forum. However I think many photographers have the same problems - thus justifying the time consuming answers.:hail:


I have roughly a million photos on 20 hard drives, that I need to reorganise and move the most important to my MacBook Pro with 1,75 terabytes + a 2TB external mini drive.


I choose to upgrade all before starting: to hard drives as above, - replacing the former 320GB internal main drive with a Momentus XT 750GB, installing the old 320GB in the Optibay as a second internal drive - until I have extracted what I need and will replace it with my 1 TB. Upgraded leopard to lion and LR3.4 to 4.1.


After opening my Catalog, LR4.1 asked me to upgrade the the 3.4 catalog to be compatible to 4.1 which I did.


Problem 1: 3 years ago a helper accidentally made a new catalog -(should only be possible with LOUD warnings) - so I have had 2 catalogs for the same kind of images for 3 years. Did not succeed merging them. One catalog with 65.000 images and one with 40.000.


Problem 2: I moved some of the 100.000 images between hard drives -outside the catalogs UPS!


Problem 3: Now after upgrade LR4.1 seems to think my images on the moved Macintosh drive are on the new Momentus drive and all folders are with ? marks - maybe I should have named the Momentus "Macintos HD" like the old one??? When pointing out the folder path for LR - only the small thumbnail from the camera shows (no preview are generated or loaded)


Problem 4: when plugging in an external drive, the minority of images that does load does so with default setting - without editing - and history is empty in editor - no metadata either.
It seems the only way to get my work back is rebooting from my old drive in the Optibay. Not tried yet.


Question 5: The reason I did not import the full million images earlier was limitations in soft and hardware. My plan is to do that now, but can LR4.1 handle 1 million images (2/3 are backups, but for complete over view I think better import all before deleting/excluding what I don't need)?


Question 6: should I do a retry merging the two catalogs before or after importing all images to the biggest catalog? (the reason I was unsuccessful last time may be that I did not have all 20 drives hooked up?)


Question 7: how to automatically avoid importing images already among the 100.000 in the catalogs - I guess all imports of raw are as the default versions from the camera (editing lost)?


Question 8: How to keep the folder structure on my hard drives when I have to open all images on the drive before I can highlight and import them. Wont LR freeze when opening up to 100.000 images in one time? Do I have to import folder by folder? Sure hope not - will take all summer.


My 2 and 1TB drives are empty and the 750GB Momentus have 60GB new images from the last month edited in LR4.1. So 3,7 TB of room to play with.


For a solution with follow up on this I reward a free 1 week stay for 2 people (outside the peak season) in my 4 star paradise resort in phuket Thailand www.coco-palace.com. This is to the main contributor to the solution.:nod:


Warm regards and thanks from Brian in Denmark www.skyumtravelimages.com
 
Sort by date Sort by votes
This seems like a huge task and will require a lot of questions, answers and screenshots to sort out.

Your two catalogs: do they contain partialy the same photos? Or completely different photos?

Is one of the two catalogs your "main" catalog? Or do you use both?

Your 1 million photos: how much space do they require for storage?

Moving photos (that are imported to LR) outside of LR is asking for trouble. Not much to do about it now that it is done, but try to avoid that in the future.
 
Upvote 0 Downvote
Are you willing to work online with someone?
 
Upvote 0 Downvote
Thanks Erro and Kiwigeoff
I think I did not reimport any photos - so non should be in both catalogs. I may remember wrong - work only in the summer with images.
Im of course willing to work online if possible - no experience in this. However others won't sheer the insight.

I regret having purchased LR, because the time saved in the daily workflow is wasted on learning, mistakes and stuff like this Its so complicated that it takes the joy out photography and put us in the chair than on our legs. With so many images and so much work invested we are also scared of experimenting to find possible solutions. Since the unchanged images are on Venus and the work is on Mars - I knew, when I bought it, that its only a question of time before the rocket have a Houston problem and work is lost. I was much more productive in ACDsee and images were not ghosts. Hope Adope reads this.
 
Upvote 0 Downvote
Forgot to say that the small catalog is the the one I have used the last half year. The big one is the older.
 
Upvote 0 Downvote
Why can't LR find folders and images with unchanged names by itself when the drive is open. It should not matter where they are or have been moved to if it just searches all files.
Would it be an idea to make LR backups and just go ahead to start importing all images - in the waiting time for the experts? If so should I tick "don't import suspected duplicates"
I suppose I should tick "minmal previews" I have never converted to DNG - that I know of. I know way to little to do such a big project - to little time to study.
Cheers
 
Upvote 0 Downvote
I'm not sure I understand your question. Lightroom is not a file browser. It does not simply search for images. Images must be imported into Lightroom. The import process does not move the images unless you tell it to do so. Lightroom just holds a pointer to where the images are located. Don't know if this will help, but I tried.
 
Upvote 0 Downvote
The basics first. LR is based on a database. LR only knows what it has been told to know. When you import images to LR, you simply tell LR where those images are located, LR reads them, builds a preview, reads metadata, and stores everything about the images in its database (the catalog). Much like a public library full of books. It also has a catalog where you can find information about all the books, where they are, a short description, year of release, author, genre and other stuff. But the books are not in the catalog. They are on the shelves somewhere. Just like the photos are on a hard drive somewhere. If you start moving the photos around outside LR (or if you move books between shelves) the catalog will loose track of them.

Can you upload a screenshot of your Folders panel in LR for the two catalogs you have?
 
Upvote 0 Downvote
Thanks, I am aware of all this, but think that LR should be able to actively search (browse) for missing images, since Im sure that this is a very common problem for most photographers - if not after moving images outside LR, then after upgrading soft and hardware, like for me right now. Since I have not dared to reimport the images from my Macintosh HD, LR4 does not se them, although the drive has only ben moved to the former CD drive within the Mac. I don't know how to take a screenshot. Will look for it. Thanks again.
 
Upvote 0 Downvote
If a disk has only been moved, LR will show questionmarks on folders and photos, since they no longer are stored at the location where LR's database say they are. This is easily solved by rightclicking said folder or photo and selecting "Find missing folder/photo" and telling LR the new location. All other moved folders and photos will be found automatically, assuming you still have the same folder hierarchy. I told you all this in a previous e-mail, and you said you've done this but it didn't work. This indicates that the disk is not just simply moved, something else must have happened also. What?

"Searching for missing images" can be accomplished by righclicking a folder and selecting "Synchronize folder". This will get LR searcing that folder to see if there are any new (unimported) photos, or if any already imported are missing. Remember though that if you have only moved photos outside of LR (as is the case when moving drives) LR will treat these as new photos, and import them from scratch.

Another question to ask yourself, considering the size of your photo collection with 1.000.000 photos on 20 drives (is that tue by the way?), is if starting over from scratch is an option? You would obviously loose all the edits, adjustments, keywords and such that you have done in your two catalogs. Only you can judge how much work that is, and if it is woth loosing that to get a fresh start. Consolidating all your photos on a smaller number of disks (preferably just one, if possible) and importing them all into one new catalog will give you a quick (well...) and simple starting point. Though, as said, it means starting from scratch.
 
Upvote 0 Downvote
Thanks ErroBefore upgrading, I made a time machine backup to my 2 TB external. After upgrade, I plugged it in and copied the up to date LR3.4 back up catalog from it to my Momentus.
Thats when I was prompted to to upgrade this back up to be compatible with 4.1. I was waiting for the Optibay to arrive from California, why I could not copy from the Macintosh drive. Today I plugged in the 2 TB drive again and tried to retrace the folder path to the backup on this drive - it worked, but some subfolder structure is missing and a major image folder is empty because LR4.1 has with me tracked those on my Macintosh HD. It seems much is missing from my time machine backup, but I may be able to find it.
Now I want to merge the two catalogs and find that impossible as usual. When following the LR tutorial, import from another catalog causes all images to end in the import folder. What an unnecessary curse multiple catalogs are. After giving up on this again, I want to start plugging in all the 20 drives. So my next major question is how do I avoid importing images that are already among the 100.000 in the 2 catalogs - and edited?
best regards and thank you very much, Brian.
 
Upvote 0 Downvote
Can LR import History, star rating etc - later, when it failed importing first time, or do I have to import everything "from the catalog" where it was edited? It appears that I merged 2 catalogs where the one was not the latest version.
There seem to be no answer anywhere as to how to merge two overlapping catalogs. The LR tutorial avoids this question. So I will just try (having a backup).
 
Upvote 0 Downvote
I just finished importing 85.000 images from an external drive, but all the EXIF and IPTC metadata from ACDsee is missing. Cant LR read this??? or do I have to tick/untick somewhere?
Thank you, Brian
 
Upvote 0 Downvote
Merging two overlapping catalogs, where some/all photos are imported in both catalogs, and duplicate photos are edited in both catalogs, is a nightmare to sort out.

Merging two overlapping catalogs, that are overlapped in a carefully thought-throgh way is very much possible. This is for instance the case when you have a main catalog in your desktop computer at home, and a temporary working catalog on your laptop in the field or on vacation. But it is crucially important that you only edit photos on one of the catalogs. There can be only one master. "One ring to rule them all".

When you import photos, LR read all the EXIF and IPTC metadata that is stored in the photos themselves. You don't have to do anything. In fact, I don't even think there is an option to not read the metadata.
 
Upvote 0 Downvote
So if LR's not reading the data you've added in ACDsee, it may not be stored in the files, or it may be stored in a non-standard location.

If you're doing an import from catalog, rather than an import, then it won't pick up any changes to the files themselves. You'd need to go to Metadata menu > Read Metadata from Files command, which would overwrite any data stored in LR's catalog.
 
Upvote 0 Downvote
Thanks Victoria.
Later after the import, the the metadata except keywords shoved up. Then I went to "metadata" and tried "read metadata from files" and "save metadata to files". No keywords showed up, but the keyword suggestion box below had relevant suggestions to any chosen image - so the keywords must be attached to the file somewhere. Mysterious.

Today another never before mystery: Some folders with images will not be moved or copied from one external drive to my new 2TB external or any other drive (internal included). I tried in every possible way. I get "unknown error" messages and The message that "LR was unable to copy files to the chosen destination !-100, 101-200 etc. Trying moving outside LR same result. Same with 2 sets of folders on two different drives - (both imported, edited in lightroom and working fine). A few thousand images in both sets.
I optimised, closed LR and restarted Mac - same problem.
When closing LR I am not prompted to backup and the LR icon has disappeared from the desktop program bar. I made my current catalog from a backup of the biggest of two catalogs and imported my latest on desktop catalog into this.
I then Googled the problem and saw an article about how to make your current or backup catalog a "master catalog" Is the biggest and latest not automatically the master catalog? Could this be the reason for weird things happening?
Since its so difficult to import old even LR edited files without loosing metadata, keywords or history, I have chosen to move most images outside LR and hope I can re establish the file path later manually - since I keep the folder and filenames unchanged.
(only a fraction of moved images has been in LR anyway and many file paths are broken already anyway)

Having great fun:mrgreen:
 
Upvote 0 Downvote
To Adope developers. For a big organise like this with thousands of folders, colour- and multiple choices of folder labelling would be a God sent! Its really tiring and time consuming to search a folder with gray on gray text only. Imagine the folders were game in the dense bush. The hunter with colour vision was the one that found the stake and survived.
I look forward to colours in LR:angel:
 
Upvote 0 Downvote
Dear victoria Bampton

I se you are a Forum leader and other, so you might be the right one to give my wish list.

1. A suggestion box of how to make LR better - up above.
2. A much better Folder pane - with folders slightly bigger so that they can contain symbols and colours chosen from a drop down menu. A face symbol for people, body for people, ski jumper for sport, mountain for landscape, a hammer for manmade, a house for buildings, an animal for fauna, flower for flora etc - in line with camera symbols. 4 different shapes of folders - star shape for best selection, round for 2. best, square for third and triangle for maybe never use. Ability to give them minimum 5 different colours.
3. Make the folder pane extendable at lest an inch further to the right so we can read long folder names on sub sub sub sub folder.
4. same folder symbols, colours and shapes attachable to extended thumbnail views in the grid.
5. Much needed improved image renaming tool.
6. Improved keyword suggestions and easier to make presets.
7. move the camera calibration profile all the way to the top where it belong in Develop module. I manually edited brightness, contrast, colour vibrance, sharpness etc for 100.000 RAWs over 3 years, because I oversaw this - when all I had to do was click landscape - to get the nice colours I saw on the camera LCD display and all the settings chosen when photographing.
8. In the folder pane hard drives should be symbolised with a special hard drive symbol or folder (when many hard rives have been copied to one big)

More later - thanks for now.
 
Upvote 0 Downvote
Hi bskyum

That's a great wish list, and there are a few I'd like to see too. I'd love to see the profile at the top in Develop.

For the extendable folder panel, you could try using Jeffrey's configuration manager to extend it http://regex.info/blog/2007-03-13/395

For the other requests, your best option is to post them on the Official Feature Request/Bug Report Forum where the developers can track them. Make sure you post different suggestions as different 'ideas' too, so that other people can vote for the requests they like.
 
Upvote 0 Downvote
Status
Not open for further replies.
Back
Top