The one Big catalog vs smaller debate

Status
Not open for further replies.
lnicole

lnicole

Member
Joined
Aug 22, 2009
Messages
79
Location
Loire Valley, France
I've started archiving 8 years of digital photos with Lightroom and I'm still trying to decide on my strategy. I've been reading a lot on the internet of different photographer's systems. Some are quite complicated!

From my experience working in Pro photo labs and as the archivist for a product photo studio, I've always been a big fan of keeping all actual neg's (or digital files) in chronological order and using catalogs to make your classifications and keywords. I just think things start breaking down when you use the actual database to organize by subject.

If you have a chronological master archive, it seems like the only way to easily make a new catalog in Lighroom is to export the actual file to another archive. (I've written a post in the wish list section on how I wish this would change) I've read complicated systems where photographers have the same image archived in different folders, but if you have one big catalog, I've read that Lighroom can get grumpy. I currently only have around 1/1'th of my images in Lightroom and it's really slow.

I'm stopping to debate though. Should I change my system? I shoot a lot of photographs. Only a small amount are portfolio worthy, but a lot are for memories and I'd like to keyword them too. I recently put together a memorial of my mother-in-law who passed away. I had to search through 1' years of photos to find all the photos of her. Sure would have been nice if it had been key-worded!

I'm stopping to wonder if I should have a big archive of chronological images and then separate archives for my flower photographs, my dog photo's, etc.

I hate that Lightroom doesn't handle multiple catalogs better. I shouldn't have to be jumping through hoops to have multiple catalogs. Please, Lightroom, take a page from Cumulus!

For those that say filing by date is outdated because of metadata, I have had cases where the creation date was corrupted. I also like seeing a chronological progression of what I was doing in my life and how I've improved taking photographs.

Anyway, sorry it's so long-winded and meandering. I'm just sort of a bit lost grappling with all these considerations. Your thoughts and experience with Lightroom welcome as I'm just beginning to use it and learn it.
 
Sort by date Sort by votes
Why not just use keywords or collections for organizing by subject? Why go through the work of creating a new catalog and then exporting and importing? You have a chronological catalog, when you want photos for one subject only, click on the appropriate keyword or collection. Seems a lot easier.

I have no idea what you mean when you say "I just think things start breaking down when you use the actual database to organize by subject. " In fact, I would say just the opposite — using a database is the perfect way to organize by subject(s).

There are reports of catalogs with over 2''K images working well. Optimize the catalog regularly.
 
Upvote 0 Downvote
"...if you have one big catalog, I've read that Lighroom can get grumpy."

I know you don't mean it, but the facetious side of me would respond with "well, that such good technical advice!" If you want to waste 2-3 minutes of your life, listen to this advice (but I recommend doing something worthwhile at the same time).

What Paige says. I'm one who works regularly with a 21'k catalogue (not my own).

John
 
Upvote 0 Downvote
[quote author=dj_paige link=topic=8'41.msg548'7#msg548'7 date=1254914'21]
I have no idea what you mean when you say "I just think things start breaking down when you use the actual database to organize by subject. " In fact, I would say just the opposite — using a database is the perfect way to organize by subject(s).
[/quote]

Sorry, tripping over my vocabulary. I meant breaking the archive into separate subjects rather than the database. As far as the one big catalog. I have read several things by people who are quite knowledgeable about Lightroom that it will get slow or have problems if it gets too big.
 
Upvote 0 Downvote
[quote author=lnicole link=topic=8'41.msg5481'#msg5481' date=1254918338]As far as the one big catalog. I have read several things by people who are quite knowledgeable about Lightroom that it will get slow or have problems if it gets too big. [/quote]

Yes, LR catalogs can slow down the bigger they get, but as John B reports, he works with a catalog of 21'K images that works well.

I am not aware of reports that large catalogs that "have problems" that smaller catalogs don't have. Could you be more specific here about what you have read?
 
Upvote 0 Downvote
I'm actually starting to come around to Eric Scouten's way of thinking. At least in a moderate way. I started thinking that it may not be a bad idea of having SOME file based categorization. Why would I want my macro floral shots in with my sporting dog shots? Or in with my photo's of my baby nephew? I'll still name them by date, so I can easily reference the main catalog if I need to.

If someday, I were to completely lose my lightroom database (or Lightroom was dropped by Adobe) I'd at least have SOME organizational start. I know, back ups, etc. Still, I used to have a Cumulus database too that is now lost in the digital cobwebs somewhere.

{Thinking out loud}
 
Upvote 0 Downvote
Think about your file system in terms of physical storage issues - certainty of backup, ease of reconstruction after a crash, and best use of space. You will waste plenty of time allocating files to certain catalogues, find some pictures end up in more than one or none, and will soon find images don't fit one category or another. So leave them as they are. If LR ever died, you'd have your keywords, sidecar files....
 
Upvote 0 Downvote
By the way, I read Eric's article as more about wanting portability than about the desirability of fragmenting control of your pictures. His 3rd major reason was "I was spending too much time placing photos in folders based on location" which is pretty well equivalent to splitting them up based on theme.
 
Upvote 0 Downvote
Thanks for your input. I really appreciate it. OK, so maybe I'll leave everything as I've started - the one big file with keywords and collections and see how it goes.

btw, I'm converting to .dng files, so no sidecar files.
 
Upvote 0 Downvote
DNGs will be more future proof than sidecars as more apps will read embedded than sidecar-based metadata. Taking your "death of LR" scenario, if it happened right now then you could move to Aperture for example and it would read your keywords and other metadata - if they had been in sidecars, that wouldn't have been the case. It can also display your updated preview.

Maybe get Peter Krogh's The DAM Book?

John
 
Upvote 0 Downvote
[quote author=lnicole link=topic=8'41.msg54815#msg54815 date=12549261'1]
This thread was referred to me by Eric Scouten:

http://forums.adobe.com/thread/48'271

Eric was on the development team of Lightroom, so I think he probably knows it a little bit. ;)

http://blog.ericscouten.com/2''9/'9/lightroom-2-technique-how-i-organize-my-catalog-and-why-2''9-edition/
[/quote]

Regarding the first link ... performance issues are both a function of your hardware and your catalog size. I wouldn't use that thread to plan multiple catalogs unless you have a better reason than "it might be a problem in the future if the catalog gets to large..." What is slow to person A might be perfectly acceptable performance to person B.

Regarding the second link ... Eric clearly says:

“I share my current practices here not because I want to preach that this is the One True Way to Organize Your Photos, but to provide a starting point for you in setting up your own Lightroom workflow. If this fits as a whole, great! If not, pick and choose what works for you and adapt the rest to meet your needs.”

Please think about what you are planning to do and decide how similar it is to what Eric is doing. He has come up with (and then modified) a rather advanced workflow to meet rather advanced requirements, based on his rather advanced knowledge of Lightroom. If your needs aren't all that similar, then maybe you need to do it a different way.

Regarding DNG files: Eric says "I prefer DNGs over RAW files because the metadata cannot be separated from the image data." Great. I say ... if you, the user, don't go into Windows (or Mac OS) and separate the .xmp from the RAW, then they never get separated. Never! As in, it doesn't happen if you don't do that. And no one I know does that. Adobe products will never separate the .xmp from the RAW.
 
Upvote 0 Downvote
[quote author=dj_paige link=topic=8'41.msg54827#msg54827 date=1254931421]
Regarding DNG files: Eric says "I prefer DNGs over RAW files because the metadata cannot be separated from the image data." Great. I say ... if you, the user, don't go into Windows (or Mac OS) and separate the .xmp from the RAW, then they never get separated. Never! As in, it doesn't happen if you don't do that. And no one I know does that. Adobe products will never separate the .xmp from the RAW.
[/quote]

Isn't this also a matter of specific manufacturer RAW files possibly not being readable in the future? DNGs are presumed to be more likely to be a format that will be around for awhile.
 
Upvote 0 Downvote
[quote author=johnbeardy link=topic=8'41.msg54826#msg54826 date=125493129']
Maybe get Peter Krogh's The DAM Book?
[/quote]

It's on my list to get. :) I live in the boonies of France though, so any English language book order will take several weeks.
 
Upvote 0 Downvote
[quote author=lnicole link=topic=8'41.msg54833#msg54833 date=1254933'25]
[quote author=dj_paige link=topic=8'41.msg54827#msg54827 date=1254931421]
Regarding DNG files: Eric says "I prefer DNGs over RAW files because the metadata cannot be separated from the image data." Great. I say ... if you, the user, don't go into Windows (or Mac OS) and separate the .xmp from the RAW, then they never get separated. Never! As in, it doesn't happen if you don't do that. And no one I know does that. Adobe products will never separate the .xmp from the RAW.
[/quote]

Isn't this also a matter of specific manufacturer RAW files possibly not being readable in the future? DNGs are presumed to be more likely to be a format that will be around for awhile.
[/quote]

Yes, of course that is a consideration. I just wanted to point out this argument about separating .xmp from RAW is, in my opinion, a totally bogus argument, and that's the only argument given in favor of DNGs in that post.
 
Upvote 0 Downvote
Well, I see the separation argument as marginally relevant, because people do move the raws without the sidecars (and what clutter too), and this is altogether eliminated. But it's not really as good a point in DNG's favour as the fact metadata is read by far more applications when it's embedded than when it's in a sidecar, and that there's no updated preview.
 
Upvote 0 Downvote
[quote author=lnicole link=topic=8'41.msg54825#msg54825 date=125493'748]
btw, I'm converting to .dng files, so no sidecar files.
[/quote]

There goes another one !! Are you going to keep your originals as well ?
 
Upvote 0 Downvote
[quote author=lnicole link=topic=8'41.msg54833#msg54833 date=1254933'25]
Isn't this also a matter of specific manufacturer RAW files possibly not being readable in the future? DNGs are presumed to be more likely to be a format that will be around for awhile.
[/quote]

Better print them all as well so that you can scan them again when future software won't open any of these old files including your RAWs.

Trust me. You will always be able to open them. Always.
 
Upvote 0 Downvote
[quote author=MarkNicholas link=topic=8'41.msg54868#msg54868 date=1254968585]
[quote author=lnicole link=topic=8'41.msg54825#msg54825 date=125493'748]
btw, I'm converting to .dng files, so no sidecar files.
[/quote]

There goes another one !! Are you going to keep your originals as well ?
[/quote]Maybe Nicole is going to ditch the Canon and it's proprietary format and get a camera that outputs RAW files that are already in DNG format :p

FWIW that is one of the reasons I chose a Pentax over Canon or Nikon.
 
Upvote 0 Downvote
[quote author=clee'1l link=topic=8'41.msg54875#msg54875 date=12549741'1]

FWIW that is one of the reasons I chose a Pentax over Canon or Nikon.
[/quote]

I have to confess to being totally perplexed by all this DNG worshipping. Is it REALLY just because you are concerned that someday you won't be able to open your RAW files for xmp files ?
 
Upvote 0 Downvote
[quote author=MarkNicholas link=topic=8'41.msg54869#msg54869 date=1254969'69]
[quote author=lnicole link=topic=8'41.msg54833#msg54833 date=1254933'25]
Isn't this also a matter of specific manufacturer RAW files possibly not being readable in the future? DNGs are presumed to be more likely to be a format that will be around for awhile.
[/quote]

Better print them all as well so that you can scan them again when future software won't open any of these old files including your RAWs.

Trust me. You will always be able to open them. Always.
[/quote]

While I do not necessarily disagree that it could be some time before there are issues regarding the readability of RAW (or DNG) files, I am curious as to your degree of certainty. I think back to all of the hardware and file formats that are no longer readable, and that is only going back 25+ years. Granted, common formats like text files have carried forth, but RAW files are not really a common format. I know that our archivist has fits with orphaned technology and file formats. Would you care to further elaborate as to why you believe we will always be able to open RAW files?

--Ken
 
Upvote 0 Downvote
[quote author=Replytoken link=topic=8'41.msg54879#msg54879 date=1254977355]
[quote author=MarkNicholas link=topic=8'41.msg54869#msg54869 date=1254969'69]
[quote author=lnicole link=topic=8'41.msg54833#msg54833 date=1254933'25]
Isn't this also a matter of specific manufacturer RAW files possibly not being readable in the future? DNGs are presumed to be more likely to be a format that will be around for awhile.
[/quote]

Better print them all as well so that you can scan them again when future software won't open any of these old files including your RAWs.

Trust me. You will always be able to open them. Always.
[/quote]

While I do not necessarily disagree that it could be some time before there are issues regarding the readability of RAW (or DNG) files, I am curious as to your degree of certainty. I think back to all of the hardware and file formats that are no longer readable, and that is only going back 25+ years. Granted, common formats like text files have carried forth, but RAW files are not really a common format. I know that our archivist has fits with orphaned technology and file formats. Would you care to further elaborate as to why you believe we will always be able to open RAW files?

--Ken
[/quote]

I am going to keep my copy of LR2.5 buried in a little box in the garden for my great grandchildren to recover.

Seriously though, what makes you believe that the DNG format will survive any longer than a RAW file format. There will always be a demand for software that opens todays files.

My concern is that relative newbies reading this forum may be tempted to blindly convert all their RAW files to DNG files and then disgard their originals, without really knowing why.
 
Upvote 0 Downvote
I'm sure both arguments have merit. I decided to go with .dng because after researching a lot of expert's workflows - people who teach and develop the program - the vast majority choose to convert to .dng. I'm trusting their advise, but your mileage may differ! One thing they do advise is to immediately back up the RAW files on the card to CD / DVD.

Hey, I imagine either scenario has it's problems. As a graphic designer, I remember thinking I was covered saving my collect for output folders from Quark 4 using multiple master fonts. HA! System 1' and a migration to InDesign made it possible, but a big pain. Similar thing. (Should have saved high quality PDF's)
 
Upvote 0 Downvote
[quote author=MarkNicholas link=topic=8'41.msg54876#msg54876 date=1254974833]
I have to confess to being totally perplexed by all this DNG worshipping. Is it REALLY just because you are concerned that someday you won't be able to open your RAW files for xmp files ?
[/quote]
No it isn't, and by narrowing it to yes/no on future readability you are missing other important factors. I've already mentioned details such as the superiority of embedded metadata - important when you're moving between applications, which you will definitely do in the long run, or when you are using more than one application. There's also the embedded preview from which you can make a print in another application that closely matches the ACR/LR output (9'% of pros do do spot the difference).

Returning though to the readability question, the classic example is Kodak raw which is no longer supported, but you need to see readability in relative terms - how readily will you be able to read those mystery meat file formats? When it means you're limited to one or two programs, perhaps on the operating system you don't use, or using software and a workflow that only its mother could love? Or an expensive upgrade to because Nikon Capture NX9 isn't supported on Mac OS Manx Moggy? Also factor in the little variations to raw files - a firmware variation your 21XX software doesn't recognise, or maybe you've used something like BreezeBrowser or iView to add metadata directly to a raw file - sure they use the camera maker's SDK, but those have introduced bugs. When you are faced with one of these obstacles to process a file, it is effectively unreadable - just not worth the effort. Better to make your working files DNGs, and stick raws in a deep archive.

John
 
Upvote 0 Downvote
Thanks for that in-depth assessment of using .dmg over raw. Very helpful. Would you recommend also keeping the RAW versions somewhere as a "just in case"?

I have some RAW files from 7 years ago from an Olympus compact camera. Always afraid Olympus will someday change their RAW format.
 
Upvote 0 Downvote
Status
Not open for further replies.
Back
Top