• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom Classic questions. All the information you need is in Adobe Lightroom Classic - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. And better still, the eBooks are updated for every release, so it's always up to date.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Archiving original images?

Status
Not open for further replies.

camner

Active Member
Premium Classic Member
Joined
Sep 28, 2008
Messages
737
Location
Tacoma, WA
Lightroom Experience
Intermediate
Lightroom Version
Classic
Lightroom Version Number
9.3
Operating System
  1. macOS 10.15 Catalina
  2. iOS
I'm curious about whether folks archive original images, and if so, how they do this.

I've been a religious maker of backups (both local and in the cloud) for years, and I feel quite comfortable with what I'm doing to keep multiple copies of the current state of my photo data, to protect myself against casualties ranging from drive failure and user idiocy (e.g., deleting something I shouldn't have) to flood and fire.

But, what I have come to realize that I don't have a good system for archiving originals of my photos and videos. I have done it in a rather inconsistent and haphazard way for a while (though at the time I didn't think my method-de-jour was so bad). This means that if I discover that an original image has succumbed to bit rot, or even to my having accidentally deleted an original that I didn't intend do, but didn't notice for months or years, I won't have an easy time recovering the original. I most likely have copies of originals for the vast majority of images, but in some cases I have multiple copies, theoretically identical, but in some cases I doubt that's true. Finding the original will be in many cases the electronic version of rummaging through drawers for something that one knows one has SOMEwhere, but one just can't remember exactly where.

I don't think it's worth the time to straighten out the mess of the past, but moving forward, I'd like to develop a system that I can keep up consistently.

I'd love to hear about what others do with respecting to archiving originals.

[I should probably mention that I shoot exclusively RAW, but I also take into LR photos shot on my wife's iPhone (via syncing with Adobe Cloud). I also rename images upon import (or after import, in the case of iPhone images), using YYYY-MM-DAY-OriginalFileNameOfPhoto]
 
What do you mean when you say "I don't have a good system for archiving originals of my photos and videos"? Are you somehow wanting to move them out of your existing filesystem organization, sort of "retiring" them? Or are you not sure how to organize your entire collection of media in the first place? How do you define "archive," and specifically how is it different from a backup?

You say you've been backing up to multiple locations for years, including in the cloud, but you still aren't confident that you can find a file in your backups if you needed to. Why is that? What software are you using to create your backups? Are your backups only snapshots of your disk, or are they incremental, allowing you to choose which time period to restore from?

Sorry to ask so many questions, but your post raises more questions than answers. ;-)
 
I work with current images and import into a date named folder system . Current images are stored on my primary disk drive. After about 3 month I am usually finished except for an occasional image need for some special export or print. Rather than keep 14 years worth of master images on the primary disk, I will move the oldest folders to an EHD. At one time this was a Ethernet connected NAS, Then USB2 and more recently a TB3 connected EHD which is as fast as my internal buss mounted disk.
All images remain in my master catalog and can be accessed by Lightroom if I ever need them again. I use the Folder panel to drag and drop folders from my local disk to an external disk.
Time Machine backs up all of my critical data including any image folders that I have moved to the archive EHD.
 
Not sure if you are using the term 'archive' in the same was as I do. Archive to me means removing the file from disk to an off-line archive site. If I want to see that file again, I have to restore it.

Cletus described his approach but I think saving images depends on you volume. Hard drives are cheap.

I backup my pictures and LR catalog daily to an external drive. Weekly, I have a batch job that finds all pictures I've indicated are 'keepers' and backs those RAW files and associated XMP's to the cloud. All files stay on my drive. I never 'archive' any according to my definition above.

I print my own pictures from either the RAW or TIF created from final denoise/sharpening from Topaz. I always work from the RAW with LR adjustments. I haven't got to the point of creating file DNG or TIF versions of an image. Just my approach and what I need.
 
Hard drives are cheap.

+1

Add to that "Most of my work is crap".

Go ahead, say it, you will feel better; it's true of everyone from Bill Gates to Davinci. No matter what masterpieces we occasionally produce, 100,000 photos do not contain 100,000 masterpieces. Or even 50,001.

So when I feel disk constrained I either buy more, or if it doesn't seem worth it, I cull.

There are people who have other reasons -- commercial shoots where there is a requirement to save everything. I get that. That's not what most of us have. We have reluctance to discard what we worked hard to produce. I get it; I am that. Except in moments of rational thought.

Complex archiving and space saving schemes were good business back when big commercial hard drives held 60 megabytes. Soon your nail clippers will have Alexa and hundreds of gigabytes.

Buy more or cull. Don't overthink it.
 
Thanks to all for your replies. I can see that I didn't explain well enough.

The distinction I was trying to make between an "archive" and a "backup" is this:

To me, a backup is a snapshot of data at given point in time. If the data becomes corrupted in some way (bit rot, user error in deleting something one didn't intend to delete), any backup made after that will reflect the corruption once it occurs.

I have a good backup scheme (I think...). I backup my internal and external drives weekly to an HDD, on a rotating 4-week basis, so I can go back a month if I need to. I also specifically backup my photos folders to another external HDD, and keep those backups a long time. And, I use BackBlaze to back up my internal and always attached external drives to the cloud, to take care of flood, fire, etc.

By archive I mean a copy of data that is made when the data is fresh and is not changed thereafter. The purpose of archiving photo data is to protect against bit rot or accidental deletion. For example, just by chance in LR I noticed that one photo had a horizontal semi-transparent red band across about 1/3 of the image. Fortunately, however that happened, it happened recently enough that one of the backups I had made specifically of photo data had a "pre corrupted" version of the file. I just copied that back to my photo drive and when I restarted LR and rebuilt the previews for that file, all was well.

That led me to think about the need to change my workflow so that I make copies that are never overwritten of photo files. I can think of a number of ways to do this, but before creating my own system, I thought I'd ask here what others do, since I have learned so much and gotten many good suggestions just be reading the wisdom that others here have been willing to so freely share.
 
To me, a backup is a snapshot of data at given point in time. If the data becomes corrupted in some way (bit rot, user error in deleting something one didn't intend to delete), any backup made after that will reflect the corruption once it occurs.
...
By archive I mean a copy of data that is made when the data is fresh and is not changed thereafter. The purpose of archiving photo data is to protect against bit rot or accidental deletion.

Ah... got it.

My suggestion is go after the issue directly. Validate, periodically, your backups against your originals with a checksum (or full comparison for that matter). A file not changed (i.e. not being backed up this day due to change), that doesn't match the backup, indicates bit rot on one or the other (or some other error). Two backups and you can have a tie breaker to tell which is corrupt.

What you describe as an archive is static. That may not be a bad thing, especially with raw data, but doesn't address files that do naturally change over time, like photoshop edited TIFF's. Or many other sorts of files like financial records, etc.

I try hard not to think of photos as special cases. I want to know that all my files are protected with a similar process.
 
By archive I mean a copy of data that is made when the data is fresh and is not changed thereafter.
Following up on Linwood's advice above, you can make an archive copy, as I do when I rename and before I import files into LR, but bit rot can attack an archive as easily as any other file on a storage medium. I think that validation sounds like it will address your concerns.

Good luck,

--Ken
 
Validate, periodically, your backups against your originals with a checksum (or full comparison for that matter). A file not changed (i.e. not being backed up this day due to change), that doesn't match the backup, indicates bit rot on one or the other (or some other error). Two backups and you can have a tie breaker to tell which is corrupt.
Yes, but if there is bit rot, two different backups could well have been made after the rot set in.
What you describe as an archive is static. That may not be a bad thing, especially with raw data, but doesn't address files that do naturally change over time, like photoshop edited TIFF's.
Exactly. Those (the "files that do naturally change over time, like photoshop edited TIFFs") are the ones I haven't been able to think through to a good way of archiving. Although, I think in the end the original files are more important. Any derivative file can, with some work, be recreated, but once the original is gone, it's gone.

I try hard not to think of photos as special cases. I want to know that all my files are protected with a similar process.
As I think about it, I really don't have any electronic files that I would be completely hosed if I lost the electronic versions. Really important financial documents I also keep a physical copy of. But maybe it's just that I have a stronger emotional attachment to photos than I do to other types of documents.

I appreciate your taking the time to respond; you've given me some things to think about.
 
Yes, but if there is bit rot, two different backups could well have been made after the rot set in.

The only files that present a challenge are those that are changed (legitimately) the same day that they have bit rot set in. Those become ambiguous.

It goes like this:

- Before backup, run a checksum on all files potentially backed up (ideally on all files that will NOT be backed up because they haven't got a change indicator set, but that may be hard).

- If a file's checksum does not match AND the file is not due to be backed up, flag it as an issue.

A file that doesn't match that is being backed up is (most likely) different because you changed it.

At this point you know that either the backup or original have experienced an unexpected change (because if it was an expected change it would be due for a backup). You find your tie breaker backup and compare to each.

It's exceptionally unlikely that two backups experienced the same bit rot, so the two that match are correct.

I'm remiss in setting this up to be automated, I run it periodically manually. but in principle it could be done daily. And in theory you should do it with every backup, but it's expensive in terms of run time. I'm using Goodsync which has a client/server form, so only the checksum flows over the wire, not all the data, which makes it a LOT faster.

But I am really remiss in checking it out (and updating the backup programs, and making sure I haven't created folders not backed up... gesh... I got to pay more attention).
 
When I import images from the SD card, one copy goes to ~/Pictures/Lightroom Photos/{Year}/{Month}/{Day} and is the file is in my Lightroom Classic catalogue. At the same time, another copy is made to /Volumes/Rob-Backup/Originals/{Year}/{Month}/{Day} that I hope to never touch again.

Many years ago back when I was using Capture NX2, I made a colossal mistake when using exiftool and set all the dates in around 500 photos to 12th April 2008. When I noticed (long enough after the mistake that I had deleted exiftool's backup), I painstakingly fixed it (very slowly) using the dates of the photos in Originals to get my masters back to their correct date.

I cannot begin to descibe how happy I was that I took the time to create a pristine untouched copy of my original images!
 
I cannot begin to descibe how happy I was that I took the time to create a pristine untouched copy of my original images!
You make a very good point. This is also another reason that I don't recommend renaming original image file on import. A file named "MyTrip_20200819_0023.NEF" can't (easily) be associated with the original "IMG_1234.NEF" if it gets corrupted. In the Lightroom workflow, file names are not important (until they are needed outside of Lightroom). So naming a file with some descriptor does not benefit as long as you are accessing the image file in the DAM tool (Lightroom). Once committed to a DAM tool, you should not need to access the actual file using another tool outside of Lightroom.
 
Yes, but if there is bit rot, two different backups could well have been made after the rot set in.
I realized there could be a misunderstanding about backups and bit rot.

First, I am assuming that backups are incremental, not full. Meaning that on each day (or hour or whatever) you back up only those files which are purposely changed, e.g. that have the archive bit reset or the modified date is new or whatever mechanism is used by your backup program to do incremental (or differential) backups.

Bit rot is by definition a non-directed change, meaning that the file on disk has spontaneously changed due to degradation of the storage medium in some way that is not detected by the medium's error detection/correction processes - cosmic rays (literally), static discharge, chemical changes, etc. I guess it might include firmware bugs that change data below the file system level and that thus do not update headers. But the point is that bit rot is presumed NOT to have properly updated the modified flags that are used by backup. So Bitrot will not be backed up on incremental or differential backups EXCEPT where it occurs to a file also simultaneously (same backup interval) modified by normal processes.

So the above case will not occur for bit rot, or more precisely it is very, very, very (lots more very's) unlikely.

Now all that said, the far more likely source of corruption is software or firmware failure, or user error. These DO result in a new incremental update since they typically operate at the file system level, and so you end up backing up the corruption. This is one reason raid and sync'd disks make a lousy backup. The normal protection against this is versioned backups, but those suffer from the inherent limitation that if you have 1 year of versions, you are required to notice the corruption within a year. These sorts of corruptions also do not suffer from ambiguity -- they result from a change in the master that is propagated to the backup (vs bit rot which could have occurred on either side).

Your idea of saving a permanent archive forever DOES address cases of corruption that is not noticed beyond your window of versioned backups. So please do not take anything I said to minimize its utility. My comments were meant to address specifically bit rot. And non-bit-rot corruption is far, far, far more likely than bit rot. It is extremely unlikely (though as storage size grows faster than error correction rates it gets more likely).

While I'm on a ramble -- the true fix for bitrot are file systems that are self correcting (or at least self detecting of errors). zfs, btrfs and windows ReFS. Sadly ReFS never made it to a usable state and seems to get no love from Microsoft, and zfs/btrfs never made mainstream on mac as far as I know. An interesting application here is to use a backup to NAS that is stored on zfs (or other similar). The rationale here with respect to bit rot is that if you do detect a mis-match between backup and original, you no longer need a tie breaker, you can (statistically) assume tht the zfs backup is correct, as it has separately been checked for bit rot.
 
You make a very good point. This is also another reason that I don't recommend renaming original image file on import. A file named "MyTrip_20200819_0023.NEF" can't (easily) be associated with the original "IMG_1234.NEF" if it gets corrupted. In the Lightroom workflow, file names are not important (until they are needed outside of Lightroom). So naming a file with some descriptor does not benefit as long as you are accessing the image file in the DAM tool (Lightroom). Once committed to a DAM tool, you should not need to access the actual file using another tool outside of Lightroom.
I’m surprised this argument keeps coming up, even though it’s a non-argument. If you rename on import, then your secondary copy will be renamed too. If you use a backup utility to backup your images, then again that means that the back up images will have the same name as the renamed originals. And finally you can use the ‘original filename number suffix’ in the new name. In that case ‘IMG_0023.NEF’ will become ‘MyTrip_20200819_0023.NEF‘ and so it will still be easy to associate it to the unrenamed original, because the only part you renamed was IMG.
 
I’m surprised this argument keeps coming up, even though it’s a non-argument. If you rename on import, then your secondary copy will be renamed too. If you use a backup utility to backup your images, then again that means that the back up images will have the same name as the renamed originals. And finally you can use the ‘original filename number suffix’ in the new name. In that case ‘IMG_0023.NEF’ will become ‘MyTrip_20200819_0023.NEF‘ and so it will still be easy to associate it to the unrenamed original, because the only part you renamed was IMG.
Johan,

My own situation might be a "corner case" but I don't alway import photos into Lightroom right away. Sometimes I need to copy photos from the camera's memory card to a "holding" folder on one of my hard drives. That folder gets backed up daily (at 10 pm). If there is bit rot once imported into Lightroom, I can always retrieve a copy from the backup of my "holding" folder.

That is why I agree here with Cletus' point. I used to do renames as part of my Import preset, but no longer.

Since I do not convert my NEF files on import to DNG, I use this Lightroom plug-in to identify bit rot issues. Validator: A Lightroom Plugin for Verifying Image Files
The author does not identify the hashing algorithm he uses, so we don't know exactly how robust this hash is, in a mathematical sense.
 
I’m not saying that you should rename on import. I’m just saying that the backup argument is not valid. Or at least that it does not have to be valid.

The problem is not so much renaming image ON import, it is IMO renaming images managed in Lightroom. Let’s sat you rename images on import. Later you may rename this same images IN Lightroom. Bit rot creeps in and you have a bad original image file. What name was used in the file that was stored that you now need to replace your original. Wouldn’t you be better off not renaming image files that are managed by Lightroom. You don’t need to access these files outside of the Lightroom environment. You can always find the files that’s you need in Lightroom by using keywords, collections and other metadata. You really want to give a file a descriptive name when it is not managed by LR or any other DAM tool. Derivative files probably need a descriptive name as they are not usually managed by a DAM tool.


Sent from my iPad using Tapatalk
 
The problem is not so much renaming image ON import, it is IMO renaming images managed in Lightroom. Let’s sat you rename images on import. Later you may rename this same images IN Lightroom. Bit rot creeps in and you have a bad original image file. What name was used in the file that was stored that you now need to replace your original. Wouldn’t you be better off not renaming image files that are managed by Lightroom. You don’t need to access these files outside of the Lightroom environment. You can always find the files that’s you need in Lightroom by using keywords, collections and other metadata. You really want to give a file a descriptive name when it is not managed by LR or any other DAM tool. Derivative files probably need a descriptive name as they are not usually managed by a DAM tool.


Sent from my iPad using Tapatalk
When I "got serious" about using Ligthroom I created this custom name on import, based on the date and time of the photo (in the local time zone) plus a 4 digit sequence number. But after a while, I realized that this scheme only created problems for me. And, when I exported photos for sharing, I would have to rename them anyway to be meaningful to the recipients.

So I have come around completely on this question. Cletus is right here and so are all the others (I think Johan) that there is no need to rename photo file names. To be clear, I do organize all photos on my hard drive and in Lightroom under the top level folder E:\Photos with subdirectories based on year, then month, then date (as shot in local time zone).
 
The problem is not so much renaming image ON import, it is IMO renaming images managed in Lightroom. Let’s sat you rename images on import. Later you may rename this same images IN Lightroom.
You are changing the subject. I was only talking about renaming on import, not about renaming later. And so were you. You said: "You make a very good point. This is also another reason that I don't recommend renaming original image file on import." That is what I reacted to, nothing else.
 
Status
Not open for further replies.
Back
Top