How should I safely convert my 50,600 photos to dng files?

[JBtheLD]

I have so far always imported my digital photos into Lightroom without choosing to convert them to dng files.
I now have 5',6'' photos (some jpeg, but mainly Canon EOS 5D Raw files) and 5,''' Lightroom xmp files stored in 418 folders occupying 338GB on my nearly full 5''GB external drive.
The photos are all in a single 692MB Lightroom Catalogue.
I would like to delete the xmp files and then convert all the photos to dng files, so that I am free of the clutter of separate xmp sidecar files.
I intend to embed the original raw files into the dng files, and keep the original raw files for the time being, at least until I am sure that the conversion has worked ok.
1.) What would be the safest way to carry out this conversion?
2.) I understand from another thread that I can safely delete all the Lightroom xmp files found with Windows Explorer, and not lose any of my Lightroom edits.
Is that so, i.e. does Lightroom always keep the edits in it's database even after the metadata / edits have been saved to an xmp file?
3.) Can I then select all the photos and "Convert to DNG" all in one go, or will it all go pear shaped?
4.) The new dng files will have to be created on a second (identical) external hard drive. Will Lightroom automatically refer to these new dng files instead of the original raw files?

John

 

[Steve Holmes]

Before doing any of this, I would make sure you have a full backup of all of your images. That said, here are my thoughts:

1. To play it safe, I would suggest converting your files in several batches instead of all at once. It should all work fine, but 5',6'' images is a lot for LR to handle at once.
2. I would not delete anything, including XMP files, until you have confirmed the conversions were successful and you still have all of your edits. Things should carry over find, but again it's better to be safe.
3. I would do it several thousand at a time.
4. I believe LR will have to create them on the drive they are on. If you convert in batches, you can move them once you have checked them. If you move them in LR, LR will refer to the originals still. If you move them outside of LR, you will have to identify where they are.

Hope this helps.

 

[Brad Snyder]

John,

I've only tinkered with this personally, I'm sure someone that's done this more frequently will be along to comment, but I'll just add a coupla' general notes, while we're waiting.

First I would keep a backup of the XMPs before deleting them. I'm in the camp that doesn't completely trust the LR database, and I like the added security of the XMP files. (Like you, I want to move to DNG completely, but I'm no there yet). Yes in principal, all your edits are contained in the db, and you don't need the XMP files.

Secondly, I've always found, as a general rule, that sneaking up on any huge computer task, conversions, or printing 1''s of pages, copying 1',''' files, or what have you, is often best appoached in small bites. You might not want to just convert all in one go, just as a general principal. So, you don't end up staring at a progress indicator going, "what's it doing now, is it working?"

In general, I think you'll be OK with your proposed method, but I'd wait and see if anybody has any gotcha's that might sneak up on you.

In the meantime, it'll help everybody if you swing up to the User CP menu at the upper left and whip up a signature, with your tech details (personal too, if you want), so we can be take that into consideration .....brad s

 

[Braders]

Careful not to delete anything first.

I have had several files not convert to DNG on the initial try.

 

[Sean McCormack]

Also embedding the original RAW into the DNG will more than double the size of the DNG.

To convert all the files this way, you're looking at need another 7''GB available to store them.

 

[JBtheLD]

Thank you all for your advice. I'll wait a short while before I start the conversion process to see if there are any "gotchas", and I'll do them as you say, several thousand at a time. Embedding the Raw files seems to be the safest way to go, to be as sure as possible of future proofing the photos.

 

[johnbeardy]

"Embedding the Raw files seems to be the safest way to go, to be as sure as possible of future proofing the photos."
Alternatively, archive the raw files separately and treat them as another backup. While you can always extract embedded raws from the DNGs, this parallel archive approach also leaves the raws directly available if you want to try out some other software which doesn't support DNG.

Apart from the good advice you've already received, I'd also advise you to work with a DNG workflow for a short while before you make the big step. I switched when CS2 came out and initially did so for a month, thinking that going back over a month's new shooting wasn't too long if I decided to abort the test and go back to a non-DNG workflow.

You need to review aspects other than escaping from sidecar hell. One is backup. There's a point of view that xmp sidecars allow people to backup incremental changes by taking advantage of LR's auto xmp writing - in DNG the xmp is written into the files so their backup programs would wastefully backup the image data every time there's a metadata change. Alternatively - and this is where I am - others backup the DNGs only upon their creation and then only backup the database file. Work through this area too.

John

 

[Richard Earney]

My take is similar (ish)

I use a custom script to import the Raws into a dated folder separated by dots (cos that is the way my 'library' was set up), then I run the DNG converter on the converted Raws. These I then import into Lightroom. Lightroom doesn't do dot separation of dates unless the folders already exist.

I am on the verge of swapping to Image Ingester which I think will do this in one process.

However the point being my Raws are my originals and I archive them. My DNGs are my working files. I back these up by synchronization every night. And archive them at regular intervals.

I migrate the archive onto new media as it becomes appropriate. CD > DVD > BluRay etc. So there are multiple logical backups.

I think it makes sense to me!!

 

[JBtheLD]

"Alternatively, archive the raw files separately and treat them as another backup."

Thanks for this advice John. Archiving the raw files separately would of course be a great solution to my separate need to backup.

"While you can always extract embedded raws from the DNGs, this parallel archive approach also leaves the raws directly available if you want to try out some other software which doesn't support DNG."

Am I right in understanding that I can "always extract embedded raws from the dngs" only if I choose to embed the raw file when converting to dng, or is the raw file contained in the dng "container" anyway.

Apart from the good advice you've already received, I'd also advise you to work with a DNG workflow for a short while before you make the big step. I switched when CS2 came out and initially did so for a month, thinking that going back over a month's new shooting wasn't too long if I decided to abort the test and go back to a non-DNG workflow.

This seems to really point me in the right direction. I think that what I will do is convert my future imports to dng and then, if and when I'm comfortable with using dngs, go back and convert my existing raws to dngs, keeping the original raws as my backup. In fact it seems that what I should do from here on is:
1.) In the future, copy the raw files from my CF Card to my external "archive drive" - one which contains all the Raw files exactly as created by my Canon EOS 5D and nothing else. It will be like a giant memory card or in film terms the "negatives filing cabinet". If for whatever reason I ever needed to go back to the originals I will have them to hand without any edits or alterations, i.e. just as they came out of the camera.
2.) Import the raw files into Lightroom from the "archive drive", converting to dng without embedding the raw files and copying to a new location, a different external hard drive named "Masters".
3.) In Lightroom's Library, delete all the duds from the dngs on the Masters drive; the remaining dng files become my working / master photos that I then name & number, edit & crop.
4.) When all the editing work on each import of dngs is complete, make Lightroom backup the Lightroom Database to a separate hard drive, "Backups".
5.) Finally, perhaps, use Windows explorer to backup the dngs to the separate hard drive, "Backups".

You need to review aspects other than escaping from sidecar hell. One is backup. There's a point of view that xmp sidecars allow people to backup incremental changes by taking advantage of LR's auto xmp writing - in DNG the xmp is written into the files so their backup programs would wastefully backup the image data every time there's a metadata change. Alternatively - and this is where I am - others backup the DNGs only upon their creation and then only backup the database file. Work through this area too.

Thanks for your advice, still lots for me to consider before I plunge on down a dead end.

 

[JBtheLD]

Richard, thanks for your reply.
It seems to me that what you do is what I should be doing!
I have just outlined in my reply to John Beardy, in a somewhat long-winded way, what you have put very concisely. Thank you for suggesting this way of working.
I don't know of Image Ingester so I will Google a search for more info on it.
Many thanks, John

 

[johnbeardy]

I initially work with the NEFs until I decide the duds can be deleted. At this point I save out xmp files, so they'll be with the NEFs in the archive, and I make the DNGs which become my working files.

A friend converts to DNG upon ingestion and archives his raw files. He then edits the duds out of the DNGs, not the raws. The end result is he has every single raw file
but they don't have any metadata in the xmp files and the file numbers don't correspond to those of the DNGs.

In the end, we're both happy, and I'm sure one could produce many variations on the same theme.

You understood correctly about extracting raws from DNG - that's only possible if you embedded the raw.

John

 

[Brad Snyder]

5.) Finally, perhaps, use Windows explorer to backup the dngs to the separate hard drive, "Backups".

John, if you're thinking of using explorer, you might also want to check out Microsoft's "SyncToy", one of the "PowerToys"

It'll do just what the name says, synchronize directories (or whole trees).
I have it scheduled to run every night in "contribute" mode, (this prevents accidental deletions on the source from being reflected in the target).

For me, a great time saver, and data-saver.

....brad s

 

[Replytoken]

If you use Image Ingester, you can have it rename your RAW file while it backs up the file. This same name, but with a DNG extension can also be used when it creates the DNG file with Adobe's DNG conveter. This way, both sets of files, RAW and DNG, will carry the same name. Image Ingester's forum is hosted at http://www.thedambook.com . Marc, its author, seems pretty responsive to his customers.

Also, I use Replicator from Karenware for backing up files. Its free and its easy to use.

--Ken

 

[Andrew Hayton]

I have recently converted my RAW files to DNG without a problem. I was thinking of embedding the RAW file into it but it doubled the size so I have taken the gamble to go completely DNG with my files as I am happy with what I am getting from my Fuji files. To be safe, if you are still unsure which way to go, get another HD and copy all your RAW files to it as a backup.
When I converted the files to DNG I did a month folder at a time with no problem. Then a year folder again with no problem. When I had converted the files the xmp files had been deleted by LR after conversion, I don't know if it did this because I have LR set to write to xmp rather than keep the info in LR's database.

 

[JBtheLD]

Andrew, thanks for your info. I'm reassured that your conversion worked well. I'm planning to leave all my Raw files on the existing external drive (which will therefore become the backup drive) and convert the dngs onto the second, blank external drive.
I was wondering though about the xmp files: I understand that all the edits for the photos in a collection are recorded in the Lightroom Database, so surely the xmp files are superfluous as long as the Lightroom Database file is never corrupted or deleted.
John

 

[DonRicklin]

Especially superfluous in that you are moving on to DNG and much of that Metadata will be with the DNG and be come obsolete with further changes to the images. You can safely delete all those XMPs. If you want you could keep a back up copy of the Catalog for the Raws (before the move) with them as a zipped file.

That would serve a similar purpose and have all the other stuff that sidecars don't carry.

Don

 

[Andrew Hayton]

John,

I took a gamble when I started and didn't back up the RAW files and luckily didn't have a problem. LR then deleted the xmp sidecar files when it created the DNG files. You could do as Don has suggested for a backup option.

Andrew

 

[johnbeardy]

Superfluous probably, but you never know. Going back to my post, notice how I first process the files as NEFs, save the xmps, make DNGs, and archive the NEFs. So my NEF archives have the xmp files with Adobe adjustments and metadata. I feel this is worthwhile because if I ever wanted to switch back to using them, in theory I would have all that good stuff.

I say in theory because the other program has to be able to read xmp. That's fine with Lightroom, Bridge and other modern programs - but Aperture doesn't read xmp, not even the keywords.

John

 

[JBtheLD]

Don, "much of that Metadata will be with the DNG and (will) become obsolete with further changes to the images"
If I set Lightroom to "write the metadata to file automatically", won't the dng's embedded metadata keep updating with every change I make - if I'm right that Lightroom writes edits to xmp sidecar files for Raw files and into the dngs for dng files. Isn't that one of the reasons of using dngs?
By the by, are there any noticeable differences in the speed with which the Lightroom Library module (or Quick Develop or the Develop module), displays and edits Raw files, Raw files with xmp sidecar files, dng files, or dng files with embedded Raw files? If there is a noticeable difference, maybe I should use whichever is fastest, and would certainly want to bear this in mind before I convert my Raws to to dngs.
Many thanks to you all, John

 

[DonRicklin]

I was referring to the XMP sidecars left behind (if you keep them) that would become obsolete. Sorry for the confusion. That is why I suggested saving just the single Catalog that actuakky holds more info than the Sidecars would ever have.



Don

 

[Mick Seymour]

I took a gamble when I started and didn't back up the RAW files and luckily didn't have a problem.

Oooh this always makes me cringe. Please please please go and buy another drive to keep backups on or at least archive them to DVD.

 

[Andrew Hayton]

Mick,

I have my work backed up on another drive. I used to archive to DVD but I lost a whole years worth of images as the DVD's were bad and after a year I couldn't read them. Now I use two drives at a time set as a mirrored raid and when full move them elsewhere. One onsite and one offsite.

 

[Mick Seymour]

Now I use two drives at a time set as a mirrored raid and when full move them elsewhere. One onsite and one offsite.

I was just going to add smilies but the forum says I need 1' characters or more so ...

8)8)8)

 

[Victoria Bampton]

I say in theory because the other program has to be able to read xmp. That's fine with Lightroom, Bridge and other modern programs - but Aperture doesn't read xmp, not even the keywords.

The nice thing about xmp, of course, is that, should the worst come to the worst, you can always open an xmp file in a text editor to see the settings you used.