Managing derived files -- best practices?

[Alan Harper]

There are so many places where I want to manage "derived files" — raw + edited TIFFs, edited TIFFs + jpegs for posting, edited TIFFs + reduced TIFFs for faster previewing (or previewing in different color spaces). And this is just the 1:1 case. There are also the many:1 cases, like raw files used to make a panorama or an HDR.

I have lots of tricks (keywords, naming conventions, using the Title metadatum) for managing this, but none of them actually work. If I forget to set the keyword, or if I delete the derived file but forget to unset the keyword, I'm hosed. One can use John Beardsworth's syncromatic plugin to help with this (e.g., set all the TIFFs to blue, set all the raw files to no color, then sync color label from TIFF to raw based on date/time -- after the sync the colored raw files are exactly the files with derived TIFFs). But it can take 2-3 hours to execute for 10,000 files, and if you change a file while it is running, it fails. Don't even think about running this on a collection of 100,000 files!

And if you do have a derived file (e.g., a TIFF), there is no simple way to get to the base file (e.g., the raw). I use the Title metadata to link them, but it requires a multistep process every time. And then if you want to know if the base file has been changed since the derived file was saved, it again requires a lot of hunting and remembering. There is no way to say "show me all the out-of-date derived files."

Has anyone figured out a best practices for these cases and care to share their ideas? I can explain my conventions if you like, but perhaps someone already has a really good protocol.

 

[Johan Elzenga]

Why don't you use stacks?

 

[johnbeardy]

Because they aren't global? Thankfully or sadly, depending on the point of view.

I think you're doing most of the things you can do, Alan. Stacks can be part of a solution, but they don't help much if you store derivatives separately from raw files (a classic DAM Book concept) or when you have "competing" stacks in different collections. But they can be part of the solution.

In my view the strongest is a naming convention with a core filename that persists across all derivatives. I don't follow it, but I quite like the idea of using a suffix for multiple frame techniques (HDR, stitching etc) where the originals are named YYMMDD_SEQ-1, YYMMDD_SEQ-2, YYMMDD_SEQ-3 etc. For me, it's too much work, but afterwards it's convenient.

You could also use a field such as Source which LR's Library Filter or Smart Collections can interrogate via the Searchable IPTC Metadata criterion. My Search and Replace plugin would allow you to populate this field automatically with the file name or whatever - do this early in the workflow and that data will always be present in any derivatives.

John

 

[Roelof Moorlag]

Do the derivative files are in the same folder? Than indeed stacking could be an option.
Did you take a look at the workflow smart collection set from John Beardsworth?

 

[Gnits]

This works for me.

I keep my raw files in a project folder, within a year folder.

I have 3 (most used) export presets (ie these create derived files) which
1. Create a jpg file suitable for web/email use with watermark
2. Create a jpg file suitable for web/email use with no watermark
3. Create a tiff suitable for high quality printing.

Each of these exports are configured to go to default sub folders.

If I am creating a bunch of files for a specific client then I put a short text version of the client 's name as the subfolder name(eg JoeBloggs).

In this way I can always see exactly what I have produced per client per project.

Each of my raw files have a unique number. This number always travels in the name of any derived files. In this way I can search for all files in my catelog which may have come from an original raw image.

The amount of storage consumed by my derived files is mostly a small percentage of my original raws and I regard these as the final products from a particular project that I can always go back to if needed.

 

[clee01l]

Consider this. Ignore all derivatives except for 1st order derivatives. Let everything originate from the RAW image or the one 1st order derivative. A 1st order derivative might be one that was created during the Edit In process. If you need to use the Edit In function again, either start with the original RAW and repeat the 1st Edit in results during the second Edit In trip. If necessary use Virtual copies to maintain different processing paths.
Do not attempt to manage JPEGs or final TIFFs in LR. Instead use the LR Publish services to track the originals and republish if a second or third final file is needed

 

[Alan Harper]

All good thoughts, and I thank each of you.

I find it ironic that John Beardsworth finds it difficult to enforce naming conventions when his Syncomatic plugin uses naming conventions for some of its operations.

I don't use stacks because (a) I am always confused the moment I try to use them, (b) I do keep source and derived files in different folders, and (c) it is just another thing to forget to do when you add a file to your catalog.

At one time I thought that keywords and smart collections would obviate the need to use folders to organize my files but this isn't true because of at least three problems: derived files inherit keywords which often puts them in the wrong collection by default, Lightroom doesn't have a "contains keyword XXXX" criterion for a smart collection, and it is often easier to go the Finder to look for things, and if they aren't in a folder, you are hosed. Also, I use lots of Photoshop droplets, and it is impossible to use them if your files are scattered across your drive(s).

Cletus makes a good point which I shall try to keep in mind.

I might think that there is an opportunity for someone to build a plugin that helps enforce a workflow -- but convincing bone-headed photographers to follow someone else's workflow is probably futile.

 

[johnbeardy]

I find it ironic that John Beardsworth finds it difficult to enforce naming conventions when his Syncomatic plugin uses naming conventions for some of its operations.

Not ironic, just pragmatic. For images that remain completely within my own environment, it's no problem and the plugin is designed for automating routine tasks. The problem is non-routine stuff, like when images go to others with different naming conventions or requirements.

I'll give some thought to the "herding cats" plugin.....

 

[Michael Naylor]

And what if I composite lots of source files into one masterpiece, and then create another masterpiece using some of the source files used in the first - and so on and so forth?
This is what I do regularly. Some would say this is not photography.
Tracking what source was used where is next to impossible. I'm beginning to wonder if it even matters.

EDIT: Oops! This post has gone into the LR 6 forum and I'm now on the latest Classic. Another example of computers being incapable of being logical.

 

[Dale Strumpell]

For what it is worth, here is my approach:

1. raw + edited TIFF/PSDs: stacked together with original
2. edited TIFFs + jpegs for posting: delete jpeg after posting.
3. edited TIFFs + reduced TIFFs for faster previewing (or previewing in different color spaces): delete after using.
4. raw files used to make a panorama or an HDR: rate at level 1. My import preset sets all new photos at level 2, my view filters show level 2 and above.
5. all files created for clients: saved in a "Derivatives" folder hierarchy not in LR.

Keep as few derivative files as possible. Simply regenerate them when and if they are needed again.

 

[clee01l]

Keep as few derivative files as possible. Simply regenerate them when and if they are needed again

This is IMO perhaps the most important part of any workflow. Derivative files are created for a set purpose. Rarely does that purpose require a permanent local copy.

 

[Victoria Bampton]

EDIT: Oops! This post has gone into the LR 6 forum and I'm now on the latest Classic. Another example of computers being incapable of being logical.

We're not that fussy!

 

[Linwood Ferguson]

For what it is worth, here is my approach:

1. raw + edited TIFF/PSDs: stacked together with original
2. edited TIFFs + jpegs for posting: delete jpeg after posting.
3. edited TIFFs + reduced TIFFs for faster previewing (or previewing in different color spaces): delete after using.
....
Keep as few derivative files as possible. Simply regenerate them when and if they are needed again.

I do the above. Especially deleting temporary ones. My biggest problems are HDR and panorama inputs, where I really do not want to delete them as they are not derived but they are also not for normal use.

I have a love/hate relationship with stacks for one reason -- if collapsed they work great for allowing you to see the one key file and not be bothered by others, but if stacked mass operations DO NOT WORK. So for example if I select-all and apply keywords, any stacked items only get the keyword applied to the top image.

I really wish there was an option that said "select includes collapsed images" one could set globally. I'd much rather occasionally have to expand and select manually for the alternative.

 

[sgbotsford]

1. For each camera body change the embedded copyright notice to something like

Copyright J. Random Shutterbug -- Nikon D7200 Ser. 124567 -- XXXXX-XXXXX-XXXXX-XXXXX-XXXX

Exiftool understands enough makernote format for most camera formats that it can safely edit raw files (Test anyway)

Using exiftool, rewrite the XXXXX sequence with date, time, and hundreths, and the shuttercount. This will require learning scripting for exif tool.

You can write this same data into other exif/iptc fields that your software leaves unmangled.

Now your original has a unique ID. Much of the rest of this involves keeping that ID in the file.

2. fswatch (mac, unix, linux, bsd) or similar programs on windows can watch a file system or directrory, and note when a file is opened, closed, changed.

So now all operations have to be on a watched directory tree. Done right, a program can use the output to figure out what file is derived from what. This won't handle data transfers by cut and paste, but the typical sequence of Raw => PS -> tiff ; jpeg; resized jpeg; watermarked jpeg should be amenable.

3. Exiftool can make a full dump of meta data, including lots of stuff you aren't interested in. You can set it up to write a side car file for each image. This gives you a backup of your image meta database. Exiftool can also update an image using data in a sidecar file.

Tracking Compositing several images together is still difficult: Possibly using the unique ID as part of the layer name can help.

 

[sgbotsford]

This is IMO perhaps the most important part of any workflow. Derivative files are created for a set purpose. Rarely does that purpose require a permanent local copy.

Here's the problem: You cropped an image, changed it to mono, posterized it. Now the client wants a different set of transforms to the same image.

How do you find the original?

You provided a client with a watermarked 2048 x 3072 pixel copy three years ago. Now they want the rights to the original. Can you find it? I have a hard time with only 20,000 images. A pro may have a million or more.

The key problem: Given a derivative, where is the original?

 

[Johan Elzenga]

The key problem: Given a derivative, where is the original?

The easy solution is not to rename images on export. What you send to the client should have the same file name as the original in your catalog. You can add something, like 'B&W', but the original file name should be the base.

 

[clee01l]

Here's the problem: You cropped an image, changed it to mono, posterized it. Now the client wants a different set of transforms to the same image.

How do you find the original?

You provided a client with a watermarked 2048 x 3072 pixel copy three years ago. Now they want the rights to the original. Can you find it? I have a hard time with only 20,000 images. A pro may have a million or more.

The key problem: Given a derivative, where is the original?

As I mentioned in an earlier post, All derivatives are created in the Publish Service. My original has been imported using only the file name generated in the camera. The file in the Publish Service is the master original or a virtual copy. The derivative file (not stored in LR) is named by the publisher service as “{file name}_{publishservice descriptive}.jpg”. I can always find the original master because that file name is a persistent part of every derivative file name.

 

[sgbotsford]

I'm not familiar with the publish service. Here's an example of a real situation I had, when Lightroom was a young pup. (I didn't use lightroom to do this, but instead wrote a script using imageMagick)

184 originals at 2048 x 3072 pixels.

For each original create images with the following dimensions.

2048	3072
1625	2438
1290	1935
1024	1536
813	1219
645	968
512	768
406	610
323	484
256	384
203	305
161	242
128	192
102	152
81	121
64	96

Even scripting it took a couple days.

This sort of thing comes up. How often depends on who you are, and how you work.

I want a system that is robust.

I want a system that even if I rename files, (or someone else renames files) I can track it back.
I want a system that compensates for applications that mangle metadata. (I'm looking at YOU photoshop)
I want a system that doesn't lock me in to a propriatary data format (hence sidecar files)

 

[Johan Elzenga]

You can now do that completely from within Lightroom. Create export presets for each dimension. In Lightroom 9 you can select all these presets at once to export an image in all these different dimensions all at once. You can even select all 184 originals and then get some coffee while Lightroom is exporting them all in all these different dimensions.

 

[clee01l]

I'm not familiar with the publish service. Here's an example of a real situation I had, when Lightroom was a young pup. (I didn't use lightroom to do this, but instead wrote a script using imageMagick)

184 originals at 2048 x 3072 pixels.

For each original create images with the following dimensions.

2048	3072
1625	2438
1290	1935
1024	1536
813	1219
645	968
512	768
406	610
323	484
256	384
203	305
161	242
128	192
102	152
81	121
64	96

Even scripting it took a couple days.

This sort of thing comes up. How often depends on who you are, and how you work.

I want a system that is robust.

I want a system that even if I rename files, (or someone else renames files) I can track it back.
I want a system that compensates for applications that mangle metadata. (I'm looking at YOU photoshop)
I want a system that doesn't lock me in to a propriatary data format (hence sidecar files)

Take a look at the Hard Drive Publish Service. It is Export with a memory. Files that change after being published are marked to be republished. And since the “Exported” original is in a Publish Collection, you know everything that has been exported with a given export settings. You can create more than one HD Publish Service in LR and Each Publish Service can have individual Publish Service Folders with different criteria for selection and (I can’t remember, I’m on my iPadPro) different publish export dimensions. File naming is a part of Publish Export settings just like resizing. Jeffrey Freidl Has a couple of Plugins (Collection Publisher and Folder Publisher) that have additional features not found in HD Publish. File renaming can be tokenized using data found in Keywords and other metadata fields. There is even an If, Then, Else logic in the file name creation.

http://regex.info/blog/lightroom-goodies

Sent from my iPad using Tapatalk

 

[sgbotsford]

As I mentioned in an earlier post, All derivatives are created in the Publish Service. My original has been imported using only the file name generated in the camera. The file in the Publish Service is the master original or a virtual copy. The derivative file (not stored in LR) is named by the publisher service as “{file name}_{publishservice descriptive}.jpg”. I can always find the original master because that file name is a persistent part of every derivative file name.

And when someone mails you back the file, renamed?

Or when you/someone in your organization 'rationalizes' the picture archive?

 

[Conrad Chavez]

And when someone mails you back the file, renamed?
Or when you/someone in your organization 'rationalizes' the picture archive?

Not necessarily a problem.

Lightroom Classic maintains a metadata field called Preserved File Name. In the example below, a raw file is exported to a JPEG file using an export file naming template for derivative files.

The bottom of the figure shows the same JPEG file after it has been renamed outside of Lightroom Classic and then imported back into Lightroom Classic, as if the client had sent back the image after renaming it. The Preserved File Name field reveals the name of the source file, regardless of how anyone changes the file system name of the derivative file.

This solution isn't always perfect because:

• It would be nice to rename the file based on the Preserved File Name field, but I can't find a rename token or menu command for it…is there one?
• The source filename can be restored, but not the derivative file name.
• A client could process the image in a way that wipes the metadata.

Still…it's way better than nothing.

 

[sgbotsford]

If it does that, it would be trivial to script that using ExifTool -- if that tool is writen as a metadata field into the file.

I'm going to have to revisit Lightroom.

 

[Jim Wilde]

• It would be nice to rename the file based on the Preserved File Name field, but I can't find a rename token or menu command for it…is there one?.

Yes Conrad, you should have that token in the second of the boxes in the Image Name section of the Filename Template Editor:

 

[Conrad Chavez]

Yes Conrad, you should have that token in the second of the boxes in the Image Name section of the Filename Template Editor:

Aha! I was wondering if I was just not seeing it. Thanks Jim!