Organization - File Structure vs. Catalog tools

Brad Snyder

Senior Member
Joined
Nov 20, 2007
Messages
6,293
Location
Port Deposit, Maryland USA
Lightroom Experience
Intermediate
In the Troubleshooting forum, a forum member asked for assistance with manipulating filenames and folder organization, with a set of images that she/he had previously cataloged with ratings, metadata, etc.

Indulge me for a moment, and let me wax philosophical about the question in general.

We've gone to the effort to annotate our images in numerous ways, among them ratings, labels, pick/reject, keywords, IPTC/EXIF metadata (e.g. date/time/location, and gear specific, camera/lens, aperture, etc).

With collections, including smart collections, and filter presets we should be able to put our fingers on a given image in moments, or build temporary 'associations' for further processing (e.g. Smith Wedding, Bride's Mother's Picks, 5x7 aspect, b/w treatment) which are jettisoned when processing is complete. Does hierarchal file ordering and strategic file naming add anything to the process?

What do we care how the files are arranged on disk? LR is taking care of that for us. Much in the way that powerful search engines have, to a certain degree, obviated the need for bookmarking; why remember stuff, when you can just search again instantly?

Naturally, there're holes in this philosophy, principally around off-line archival and backups, but LR is evolving in that direction.

The situation described in the original post gives me the opportunity to ask, what is to be gained from the investment in normalizing filenames and folder structures? (Believe me, I understand the need for order, and personally I'm in exactly the same boat). What would happen if the existing files remain the way they are? One could adopt a more pleasing structure moving forward; is it necessary to backfit the existing files?

I don't intend anything in the nature of a 'flame' in this whatsoever, I'd just like to start a dialog. I'm wondering if wiser heads than mine have any thoughts to share.

Opinions, anybody?
 
Joined
Oct 23, 2007
Messages
680
Location
Perth, Australia
Hey Brad

Interesting. I ask this every day when i look at, what is now a small, but exponentially growing number of images i have in LR.

Digital entered my world 2 yrs ago LR came along soon after. DAM...what's that i asked a yr ago?

When i import, i do so straight from the card and without changing the number from the camera. I organise my folders by trees.

Travel
  • USA
    • California
    • Nevada
Events
Models

I read the DAM book the other week, and now i am confused as to how to rename my images in LR.

On reflection, i realise i have not needed to think about where my images are in LR, using the current structure i have. So, do i need to change it.

Brad
 

Brad Snyder

Senior Member
Joined
Nov 20, 2007
Messages
6,293
Location
Port Deposit, Maryland USA
Lightroom Experience
Intermediate
I used to be a fanatic about renaming and "folderizing" my source files. But I began ending up in the horrible place that John Beardsworth is always threatening us with. For example, I shoot horse shows professionally, my kids participate in some of those horse shows. Are those horse show pictures, or family pictures? Makes a big difference in a folder structure, not so much in LR with its organization tools.

I'm slowly migrating to the Year/Month/Date folder structure, and doing the rest of the organizing in LR. My only real problem is somehow giving the family access to images. I used to work RAW, XMP and export JPGs to the same folder for family viewing. But I'm working in DNG now, and slowly back-converting my catalog. I have some work-arounds for family viewing, but they're just that, work-arounds. The addition of a robust network aware "reader" program, akin to Acrobat Reader, for LR browsers, much like Marc Rochkind's LRViewer, would be a welcome addition to my toolkit.
 

rcannonp

Active Member
Joined
Oct 21, 2007
Messages
467
Location
Atlanta, GA
...For example, I shoot horse shows professionally, my kids participate in some of those horse shows. Are those horse show pictures, or family pictures? Makes a big difference in a folder structure, not so much in LR with its organization tools.

I'm slowly migrating to the Year/Month/Date folder structure, and doing the rest of the organizing in LR.

It seems to me that a folder name related to the event would be the obvious way to go there, but I guess that keywording would work just as well in that case. I started trying to use a dated folder structure. I have problems when I have pictures of a single event that may have taken place over more than one day. After 14 years of organizing with folders, I have a hard time bringing myself to put related pics into different folders based on dates. I suppose that I should dive headfirst into the metadata browser and keywording and just forget about folders.

I'm not sure what I want to do with my existing images. Right now I have a mixture of dated folders and ones with event or subject specific names. I don't know if I want to bother reorganizing the whole batch or not, and if so, I 'm not really sure how to go about it. I guess that I could do it in small chunks, probably while I'm watching baseball games or something like that.
 
Joined
Dec 7, 2007
Messages
2,222
Location
Puget Sound
Lightroom Experience
Intermediate
Lightroom Version
Classic
Does hierarchal file ordering and strategic file naming add anything to the process?


Naturally, there're holes in this philosophy, principally around off-line archival and backups, but LR is evolving in that direction.

Brad,

You partially answered your own question. In keeping with good DAM principles, we care to the point that we can have a managable, easily scalable system that can be restored in the event of a total loss of the primary catalog and its images. But, our organizational systems can, and do, take many different forms based on our needs. "The DAM Book" is a good read if you have not read it. I also suggest reading the manual for Image Ingester since Marc briefly talks about "simplified" naming/folder schemes. So, I guess you could say that we care to a point, but not much beyond (evident in Peter's "Bucket" approach).

--Ken
 

wblink

Active Member
Joined
Jan 6, 2008
Messages
429
Location
Gendt, Netherlands
Lightroom Experience
Beginner
Lightroom Version
I'm slowly migrating to the Year/Month/Date folder structure, and doing the rest of the organizing in LR.

I started with the YMD structure and am on "Events, Locations, People" now.

My reason is simple: I am an amateur, so I don't shoot every day, more on occasions (that I create for myself sometimes, like "birdshooting on Texel" (Nice!)). I had a problem collecting my shots from "Mexico Holiday". Of course I could have found them with all metadata and keywords filled in.
What I try to say: I think the "tree" should be like you remember your shots at first, the rest should be in the metadata and keywords.

In other words: the structure depents on what you want to do with it and thatis differen for everyone. Most professional (shooting every day, keeping an administration) would like the YMD struture others (like me) prefer another.

Hmm, I think.
 

rcannonp

Active Member
Joined
Oct 21, 2007
Messages
467
Location
Atlanta, GA
Outside of journalists, I'm not sure how much a dated folder system really makes sense for professionals either. Most professional photo shoots are related to a particular job. It makes the most sense to me to keep all of the photos related to a job or client in an appropriately named folder and then use keywording and metadata for other purposes.

I have a struggle going on in my head about how much I want to rely on LR specific organizational tools. Who knows what software I'll be using in ten years and if it will understand what a Lightroom collection is. Serious digital imaging is still less than a decade old. A lot of technology and working methods could get left behind in the coming decades. Many of us are placing bets on what we think will endure, but who really knows. Some new tech may come out next year and totally change the game.
 
Joined
Sep 29, 2007
Messages
22,764
Location
Isle of Wight, UK
Lightroom Experience
Power User
Lightroom Version
Cloud Service
My personal photographs are organized by year, and by month within that. Clean, simple, easy to backup. For those, I do use collections heavily to define events etc.

The studio photographs (raw originals) are organized by year, week number within that, and customer id within that. The chosen files are just organized by customer id, with 'variations' (sizes, crops, etc) within that.

Different things will work for different people - I guess it's just important to be able to easily keep track of what's backed up, and easy to rebuild from backups if it should all go wrong.
 
B

BobH

Guest
Initially I tried the systems that others have mentioned, organizing files by client and/or event etc. The problem with that approach is what to do with files that fit multiple events. For example, I shoot photos for both a musician and a local events center. Do I put photos of that musician in his folder or the events center folder or make duplicates or... The more photos you have, and the more events you work, the uglier it gets.

I used the folder by subject system with Iview, since it seemed to suit that program well. Now, after a couple of years of working with images, I use a very simple system that works well for me. I simply create numerical folders for the images and store them in batches of roughly 1'''. i.e. '''1 contains images '''1 to '999. Folder 1''' contains 1''' to 1999, etc.

Since the files are named sequentially by the camera they are also somewhat organized by date. It takes nearly no time at all to organize them this way, while also avoiding a single folder with 1',''' images.

Organizing stuff numerically simplifies backups too. Along with my automatic backups to an external HD, I occasionally dump the images to DVD. It's easy to see where I left off, based on image numbers, and when I start at file xxxx, the rest of the new images are very easy to find. The folder structure is easy to navigate, and the image numbers make it clear which ones are new.

rcannonp;1373' said:
I have a struggle going on in my head about how much I want to rely on LR specific organizational tools. Who knows what software I'll be using in ten years and if it will understand what a Lightroom collection is.

Does LR write any of this stuff to the exif? The problem you mention is a real issue, and one I've already dealt with. When moving from Iview to Lightroom, I lost some of the organization I'd set up using the folder arrangement. Since it was obvious it wasn't going to work long term, I didn't mind too much, but I certainly would rather not have the same issue again.
 

Brad Snyder

Senior Member
Joined
Nov 20, 2007
Messages
6,293
Location
Port Deposit, Maryland USA
Lightroom Experience
Intermediate
Does LR write any of this stuff to the exif? The problem you mention is a real issue, and one I've already dealt with. When moving from Iview to Lightroom, I lost some of the organization I'd set up using the folder arrangement. Since it was obvious it wasn't going to work long term, I didn't mind too much, but I certainly would rather not have the same issue again.

In general, no, the EXIF is produced by the camera. LR doesn't permit changes.

Without trying to be pedantic, I'm assuming you meant metadata in general, (terminology is extremely important here, that's why I'm picking nits.)
The answer to that, is "It depends". LR can be configured to embed metadata in both source and derivative files, but the method and datatypes vary between file types. No metadata is embedded in RAW files types, hence the XMP sidecar files. And by default, the metadata in not embedded in source files of other types. The embedment of metadata in export files is configured as an export option. Unfortunately, the global standards for metadata are currently in a state of flux, have problems with multi-language/culture, and depend a great deal on software vendor specific implementation.

Bob, if you'd like to talk more about the specifics, we can fire off another thread.

When I first used LR, I tended to maximize my use of XMP sidecars to maximize future compatibility with unknown systems. At present, I'm currently using DNG, with metadata embedded. While not 1''% certain, I have enough faith in Adobe's weight in the market place, that I won't be stranded here, should there be a paradgim shift in the technology.

Personally, I think the future gold-standard of data storage will be a model of content-addressable memory. This was an oxymoron, when I was a wee engineering student, how could you address memory without an address, and how could you get the address without knowing the content?

But that's what crawler, trawler, index and search engine technology is all about. Who cares where it actually lives, if it comes when I call it?

And that's why I'm spending quality time with metadata. I think it's my best shot at future-proofing my catalog.
 

stasber

Active Member
Joined
May 6, 2008
Messages
250
Location
Cork, Ireland
Bed time reading

Good thread this, helps with some of the crunchy questions that float about my own head from time to time, desperately avoiding themselves getting crunched! :shock:

I've read and applied parts of the DAM book and I also work in Data Management (and I still can't manage my photos! :mrgreen: ).

"It depends" is a common enough predicament, which is why at the core of a system there needs to be a structure that is as neutral and as transparent - and ultimately scalable and reconfigurable - as possible. It needs to conform to a norm of some kind (not Norm from Cheers, either!). (I could have said conform to a norm of some form but decided that it's just too much faux humour for one evening.)

One of the "depends" is how important or significant is your photography to you. Clearly when you accumulate a large amount of pictures, how you organise them becomes important so that you may easily locate them again "if the need arises for any purpose in the future". Maybe it's not that important to you?

It needs to be not only crystal clear to you but also automatic (don't spend time deciding where new pics should go - you should know already). Will other people need to navigate your system?

Another "depends" is how you remember things by. Think long-term. Heading up a folder for your 'Trip to Spain' might be great at the time but 5 years later you're looking for a picture you're sure you took a few years ago of a certain kind of palm tree but was it in Spain, Morocco, Malta, or in the botanical gardens because you went there too? Too bad you didn't keyword it (it was only a trip after all and you were out on the razz or in recovering from it).

Personally I use Folders in LR for the core library and then Collections for contextual organizing. I may have the same image - I have rules for where I put the master and where I make a virtual copy - in several places. But the initial image is located once in one place only.

The core library has the same structure for Originals, Masters, Derivatives, Archive, and Vault, except that the vault groups the folders into sequential dvd sized buckets rather than just date-based folders.

I shoot mostly live performance, so it is easy to group into date-based 'shoots' (not to be confused with LR's first beta concept of Shoots for any oldies out there). I use a 5 character code to help me remember the event (Circus Festival would be CFEST, or The Coronas would be CORON). This is appended to a 6 digit date.

So we see Master > 2''8 > '4 > '8'4'2-OCEAN
(That's he jist of it... not telling you all my secrets :twisted: )

Luckily for me the date based system works quite well.

Neutral - it does not rely on the subject matter (that's where Collections come in), it just records the fact that some pics were taken, with a brief standardised descriptive element for content.

Transparent - my folders correspond across the board. I don't re-import my derivatives back into the same folder of masters (though for some this would undoubtedly be a boon hence the feature in LR2) and finding 'photos from our gig last Aug' is pretty quick as the folder structure is like a template across the board.

Scalable - The same (single) system is used across 3 applications (LR, Canon DPP and LightZone), and the core folder structure used for all categories of storage per above - Original, Master, Derivative etc. I also differentiate between RAW, DNG, JPG, DRV (derivative such as scanned negs or derivatives from one application used in another as a master).

Reconfigurable - After all that, I bloody hope so :shock: :cheesy: My 'norm' is based around date but also includes a descriptive element. The images themselves are renamed also based on date but retain the image number (which would help me locate the real Original if needed). That gives me some latitude for a complete rehash if for any reason I needed to. My 'core unit' is the folder format of '8'4'2-OCEAN which I doubt I'll change for a while yet as that's how I remember things but everything above it can be changed or moved about.

The important element here is that it is solid with a single thread making it consistent. At the same time, being heavily reliant on date this would be a potential single point of failure hence the never ending rattlings-on in my head. I'm not ready to relinquish to a truly number-neutral system ala BobH.

Smart Collections in LR2 will enhance the Collections (and Sub-Collections) that I currently use. Hierarchical Keywording is also important in all this. It also makes it far easier for periodic maintenance, such as spelling variations, where there should only be one version. Again, this is by now automatic - I know where things go, don't need to think about it. Usually. :mrgreen:

Sleep well.
 

Brad Snyder

Senior Member
Joined
Nov 20, 2007
Messages
6,293
Location
Port Deposit, Maryland USA
Lightroom Experience
Intermediate
Stas, spot-quoting your post:

The core library has the same structure for Originals, Masters, Derivatives, Archive, and Vault, except that the vault groups the folders into sequential dvd sized buckets rather than just date-based folders.

.......

So we see Master > 2''8 > '4 > '8'4'2-OCEAN
(That's the jist of it... not telling you all my secrets :twisted: )

............

Transparent - my folders correspond across the board. I don't re-import my derivatives back into the same folder of masters (though for some this would undoubtedly be a boon hence the feature in LR2)

I've considered this sort of parallel Master / Derivative relationship. One of my problems is handling family photos, in conjunction with 'professional' photos. I'd rather not keep RGB derivative files unnecessarily, in most cases, post-production, I delete them, they're easy enough to recreate. Unfortunately in the family environment, their level of comfort and the tools available are jpg/tiff oriented. I'm considering giving them DNG 'capable' tools such as XnView, etc. with read-only access to my family files. But there's a learning curve, which then gives me a teaching curve.

My main question is how do you construct the parallel directory structures? Is it automated in some way? Or is that part of the secret? :cheesy: In other fora, I've seen feature requests for an export option to a parallel structure, i.e. using the source folder tree, export to and identical folder tree, at a different root/parent.

I'm still chewing on the idea, one of the many conversations I hoped to stir up in this thread.
 

stasber

Active Member
Joined
May 6, 2008
Messages
250
Location
Cork, Ireland
I've considered this sort of parallel Master / Derivative relationship. One of my problems is handling family photos, in conjunction with 'professional' photos. I'd rather not keep RGB derivative files unnecessarily, in most cases, post-production, I delete them, they're easy enough to recreate. Unfortunately in the family environment, their level of comfort and the tools available are jpg/tiff oriented. I'm considering giving them DNG 'capable' tools such as XnView, etc. with read-only access to my family files. But there's a learning curve, which then gives me a teaching curve.

I had a similar quandary, though not as intimate as yours, in that I had heaps of miscellaneous personal stuff that was distinct from my main work.

In the end I determined that a single reference was the way to go for everything, and to apply the same (organisational) principle across all. That way I have less to think about and don't need to get into one particular way of thinking in order to find my way around, and that's why it should be as transparent as possible.

If it makes sense to you, that 'Family' and 'Professional' are carved out and never the twain shall meet, then building two libraries, one for each environment, seems fit for purpose. You would want them to mirror each other in their structure because it makes management & maintenance more automatic as you have the one system to think about, not two.

Similarly it will make things so much easier to change or rehash (i.e. reconfigure) in a few years when you decide to migrate to another tool or system; you want as few surprises as possible.

If the family files are JPG then leave them be and let your family work on them as they wish, saving the changes to a different folder. Use LR purely to catalog them, not necessarily as the tool of choice for processing. Non-destructive tools are preferable obviously, but then you'd also have your original files stored & backed up separately that should never be manipulated, as your datum. This is one of the "depends" (depends how important this is to you).

My main question is how do you construct the parallel directory structures? Is it automated in some way? Or is that part of the secret? :cheesy: In other fora, I've seen feature requests for an export option to a parallel structure, i.e. using the source folder tree, export to and identical folder tree, at a different root/parent.

I created the parallel directory structures manually myself. Once I had figured out the format I was able to partially automate the renaming (i.e. the date bit) but still it took some time to rename the descriptive element, which I did folder by folder. New pics fall into the new structure easily, though it still requires manual folder renaming (LR will drop the import into a date based folder, I then rename it at the point).

Derivatives only get a folder when they need it. If I don't have any output for some shoots then there's little point in setting up a folder with nothing in it, for the sake of having a folder. You may benefit from a temporary derivatives bucket, where you output to, then delete from (and could probably set up a script or something to empty that folder periodically).

However when I do have an output, I use LR's ability to export to a subfolder, and every variation or set of pics gets it's own subfolder so that I know what it was for, or who it went to, or which application it was exported from (hence which app it was processed in). Synchronising folders from time to time is great for this.

Depending on how the re-import feature will work in the final release of LR2, I might find myself using it.

Lightroom essentially does two things; it catalogs images and allows them to be manipulated. Try to keep both distinct in your mind and be impartial about that. Use LR to organize your library (or libraries) in one place.

A good system has a singular input source (Reference), a method for staying organized (Management), allow changes to be applied (Context) which may require periodic review (Maintenance) and have a defined output area (Result).

Your Reference should never change and your Management of the system should ensure it stays organized (which you may refine as you progress); if you do make changes then you change their Context which may need Maintenance from time to time to prune or check that it still conforms (if you don't make any changes to your Reference then there's nothing to Maintain); then what you get out at the other end, i.e. the Result benefits from it's own area, distinct from the Reference.
 
Top