Migrate DNG back to RAW

[thany]

In a nutshell:
How do I migrate DNG files back to RAW without losing edits?

Long story:
I have Lightroom 5 and bought a new camera, the Olympus OM-D E-M5 mark II. This isn't supported by Lightroom (although it is by Photoshop, inexplicably) and therefor I needed to convert each and every picture to DNG before being able to do anything with them in Lightroom. I converted them by embedding the original RAW.

Now that I'm updating to Lightroom 6 (because Adobe refuses to add camera support to Lightroom 5, those nasty youknowwhats) I get my camera supported. Finally. Took way too long. Now the problem arises how to migrate my existing, edited DNGs back to RAWs without losing edits.

I can convert the DNGs back to RAWs alright, but Lightroom is going to see those RAWs as new (unedited) files. There are thousands of them, so copying over all edits of all files, and all tags and metadata and whatnot, is simply not an option. Keeping DNG is also not an option, because they are horribly inefficient, doubling every file in size. I'm actually having trouble keeping them in one place because the collection is just simply getting too big.

So how do I do this? Surely, this has to be possible. It's not the first time Adobe refuses to add camera support to Lightroom, but keeps updating DNGconverter...

I've asked Adobe FOUR times about this, and they keep ignoring (only!) this question. Which makes me like them even less. I was hoping someone here has a good idea

 

[RonBoyd]

In a nutshell:
How do I migrate DNG files back to RAW without losing edits?

I don't have an answer for you and am only answering to get clarification of what you are asking. Are you asking how to divide the DNG file into two separate files -- one the Original RAW file and the other an XMP file containing your edits. (This would be the scenario if you had processed the RAW files with ACR/LR. Forgetting for the moment that some edits are stored only in the LR Catalog.)

 

[johnbeardy]

"Keeping DNG is also not an option, because they are horribly inefficient, doubling every file in size."

This seems a clue. DNGs are not "horribly inefficient, doubling every file in size". If you choose one option, they are usually the same or smaller than the raw file. However you may have chosen another option which embeds the raw file inside the DNG, and in that case they can double in size.

Anyway, DNGConverter's Extract button gets the raw files out of the DNGs. You're now left with DNGs in LR with your adjustments/metadata, and raw files without any work. If you import the raw files into LR, you can then choose each pair of pictures and sync the settings/metadata from DNG to raw, and this can be automated using my Syncomatic plugin.

John

 

[mcasan]

Why do you need to do this? You have DNG files with edits. The files are large because you told LR to embed the ORF. So of course the files are large. The good news is disk drives are dirty cheap. Keep the DNGs you have and don't do anymore.

 

[thany]

I'm not really open to things like "why'd you wanna do this?", I think I made that clear. They double in size because the RAW is still in there. And I did that because at some point I can revert to the RAWs when LR supports them, cutting the required space in half again. Storage is not "dirty cheap" when dealing with a laptop that has storage soldered on the motherboard - just don't assume anything about my environment

The biggest problem is the edits. Both in terms of metadata, and in terms of editing in the develop module. I'll have to test that Syncomatic plugin.

/edit tested it, and €9 is too expensive for what it can do. Sorry, but no-go. I'm a hobbyist, so I'm not losing money by not buying the plugin.

 

[RonBoyd]

Again, let me get clarification. So you are concerned about storage space caused by having to keep two copies of the RAW file -- the original (that you intend to use when LR supports them) and one embedded in the DNG file. And you wish to extract from the DNG any metadata (created after conversion to DNG) into a separate file (I suggested XMP earlier but that didn't resonate, I guess). This "metadata" file primary function would be readability by LR while processing the (original) RAW file in the Develop Module. Am I getting close?

 

[mcasan]

Thany, sorry to have upset you. That was not in any way the intent. I fully understand what and why you did DNGs before Adobe was ready. Now that Adobe products can handle E-M5II, you don't have to do embedded DNG again.

I looked for an option to extract raw files and the related edits as sidecars from edited DNG files that have embedded raw files. I did not find a way forward for that requirement. The only two options I could arrive at are:

• use DNG Converter to extract the raw files and reedit the images in LR as needed
• stick with the edited large DNG files from the past and only do native raw (or non embedded DNG) going forward

Sorry if none of this helps.

 

[thany]

@mcasan Sorry if I sounded blunt, but I did want to make something clear. I appreciate your options, but they're just not helping, at least for the moment.

@RonBoyd You nailed it. That is almost exactly what I'm aiming for. But as far as I understand things, some edit are only in the LR catalog, and never in the XMP sidecars, right? With that being the case, I need some way of perhaps "renaming" the files in the LR catalog to the new RAWs they'll become after converting the DNG back to their originals.

The thing I'm concerned with, is that DNG makes my entire collection double in size. All DNGs put together amounts in about 150GB. This may not sound like a lot, but when I'm forced (by various factors) to keep this entirely on a laptop with a 256GB soldered-on SSD, it can get hairy (I know I should've bought the 512GB model... me just being frugal ). The RAWs only would've taken 70-80GB, which is well within parameters of sanity, so to speak. This is the main reason I want my DNGs converted back into RAWs, and that on its turn, is why I decided to embed the RAWs.

Now, back to the XMP sidecars. I sounds promising. Can I convert DNGs to RAW+XMP without losing anything, or do I need to do something extra somewhere somehow? Obviously, have LR store all metadata in the DNGs (just hit Ctrl+S on them, right?), but then what?

 

[johnbeardy]

The thing I'm concerned with, is that DNG makes my entire collection double in size.

DNG does not make your picture collection double in size - the doubling is caused by your choice of the embed option when you made the DNGs. DNGs are usually smaller than raw files.

Now, back to the XMP sidecars. I sounds promising. Can I convert DNGs to RAW+XMP without losing anything, or do I need to do something extra somewhere somehow? Obviously, have LR store all metadata in the DNGs (just hit Ctrl+S on them, right?), but then what?

You cannot convert DNGs to raw+xmp, but you can extract the embedded raw files. So that leaves you with DNGs which have your LR work (after Ctrl S), and raw files which have none of your work. Syncomatic is one quick way to fix that, and you can run it on groups of 10 images without paying, but there are other solutions:

1. Do you have SQL skills - in which case you could point the LR database to the raw files. Yuck?
2. Have you used Exiftools - it could generate xmp files from the DNGs. Techy?
3. Have you got Bridge- it could generate xmp files from the DNGs.

2 and 3 are quite similar, but let's assume you have Bridge. First do a Ctrl S in LR, then use Explorer to make all the DNGs read only. Making them read-only is a trick which will force Bridge to create sidecars, but it won't do so until you tickle Bridge into thinking it needs to write some metadata. So select all the DNGs and apply some metadata to a field you aren't using - it might just be a colour label. Bridge should now generate sidecars which will include the adjustments and IPTC that you had entered in LR.

Now you extract the raws, put them in a folder, move those xmp files into the folder, and import the folder into LR.

How does that sound?

John

 

[johnbeardy]

Just looking into this again, I'm currently unable to force Bridge to write sidecars for read-only files. I am 99.9999% certain it used to work and am looking into it.

It's possible that someone wrote a Lightroom plugin to do so, but I'm not sure.

John

 

[Replytoken]

This may not sound like a lot, but when I'm forced (by various factors) to keep this entirely on a laptop with a 256GB soldered-on SSD, it can get hairy (I know I should've bought the 512GB model... me just being frugal ). The RAWs only would've taken 70-80GB, which is well within parameters of sanity, so to speak.

Unless you do not shoot much, and/or you have some planned hardware/storages changes in the works, it sounds like you are going to be having an impending storage issue in the not too distant future. :sad:

Good luck,

--Ken

 

[johnbeardy]

This would be the Exiftools batch file for Windows, creating sidecars for each DNG:

"C:\Users\john\Desktop\exiftools\exiftool.exe" -ext dng -o %%d%%f.xmp -r "C:\Users\john\Desktop\New folder" -k

After the path to Exiftools, it says "-ext dng" or limit to DNG extension, "-o %%d%%f.xmp" says output to same directory and filename with xmp extension, "-r" means recursive, and "-k" just leaves the command window open.

As always with Exiftools, use with great care and test on a small folder of test images. If you aren't confident you understand it, don't use it.

John

 

[RonBoyd]

The thing I'm concerned with, is that DNG makes my entire collection double in size.

DNG doesn't cause this situation. As I said above, this is caused by your decision to keep two copies of each RAW file -- the original and the embedded. Attempting to divert blame for this situation to DNG is unproductive (as far as this conversation is concerned).

 

[johnbeardy]

OK, I've now found that the Bridge method won't work. While I can force it to make sidecars (making the DNGs read only and then applying an adjustment), the sidecar won't simply apply to a raw file whose name matches.

The Exiftools method does work.

 

[thany]

I have found that when hitting Ctrl+S on a real RAW file in Lightroom and it has edits, the edits will go into XMP as well. Now the questions are:

1. Does Lightroom store edits in DNG in the same way it does in XMP?
2. Is exiftool capable of transferring every possible kind of edit from DNG to a (new) XMP sidecar?

It feels as if that would solve this issue.

 

[gregDT]

I'm thinking that if this problem is so unimportant that it isn't worth spending the cost of as couple of latte's and a muffin then it's not important enough to bother finding a solution for. If you have just spent over a thousand euros on a camera and there is a perfect solution to your problem that costs just nine I fail to see the problem.

 

[johnbeardy]

I have found that when hitting Ctrl+S on a real RAW file in Lightroom and it has edits, the edits will go into XMP as well. Now the questions are:

1. Does Lightroom store edits in DNG in the same way it does in XMP?
2. Is exiftool capable of transferring every possible kind of edit from DNG to a (new) XMP sidecar?

re 1, yes

re 2, probably. I didn't notice a difference, but you can test it yourself.

Or just use Syncomatic ten pictures at a time.

John

 

[clee01l]

I have found that when hitting Ctrl+S on a real RAW file in Lightroom and it has edits, the edits will go into XMP as well. Now the questions are:

1. Does Lightroom store edits in DNG in the same way it does in XMP?
2. Is exiftool capable of transferring every possible kind of edit from DNG to a (new) XMP sidecar?

It feels as if that would solve this issue.

Why do you want the LR edits stored in the DNG or XMP? Only LR (or PSCC or a few other Adobe products arguing to understand what these instructions are supposed to do with the master image data. All of the metadata associated with the Master image file is stored in the LR catalog. The metadata stored in the XMP section of the DNG file or a separate text based XMP file is not ALL of the metadata associated with the master image file.

Looking back at your original issue, Why are you converting to DNG in the first place? If you plan to keep the original RAW file format (and you should), The DNG is simply the same RAW data in another wrapper. That means that your files storage requirements are nearly double (The DNG is slightly smaller than most proprietary RAW formats). As you pointed out you can use the DNG wrapper to wrap a complete copy of the original RAW file as well as an additional copy of the image data block from the original proprietary RAW file format.

The Smallest storage footprint is the Original proprietary RAW file and a LR catalog contraining ALL of the metadata. Good data risk management also includes redundant back-ups of the LR catalog and the master image files. Some one is going to point out that you can get a smaller storage footprint by converting the Proprietary RAW file and discarding the original. I contend that only a fool would throw away the master original Proprietary RAW file in favor of keeping a smaller DNG

 

[RonBoyd]

I contend that only a fool would throw away the master original Proprietary RAW file in favor of keeping a smaller DNG

+1 .

 

[johnbeardy]

Looking back at your original issue, Why are you converting to DNG in the first place?

This was explained in the first post: "I have Lightroom 5 and bought a new camera, the Olympus OM-D E-M5 mark II. This isn't supported by Lightroom (although it is by Photoshop, inexplicably) and therefore I needed to convert each and every picture to DNG before being able to do anything with them in Lightroom"

 

[Hoggy]

I contend that only a fool would throw away the master original Proprietary RAW file in favor of keeping a smaller DNG

Color me a fool, then. :nod: I don't see a point in keeping the original. All I'm concerned with is the image data for LR. Of course, my Pentax cams natively save in DNG, so it's a no-brainer - but for my Canons..
But then again, I'm no pro (just yet) - and have a cognitive/memory disability, so learning LR and program X and program Y to deal with each camera would be too much for me. It's already taken me 3 years to become somewhat proficient with LR alone. I did contend with the possibility of using the manufacturer's software for a bit (a reason to keep proprietary raws) because I liked the Pentax JPG's SOOC, but then as I learned what could be done with RAW's I decided to just stick to learning one software package for all cameras and do away with JPG completely.
Anyhoo.. off topic.

As for the OP's issue, I think John's solutions would be the way to go. Otherwise there are a couple plugins from Rob Cole that might do something for you (for free), but it seems his plugins page has been down for a few weeks. If you're able to find a copy though, the relavent plugins might be XeMP, and possibly one called RAW+JPG (I know it says 'JPG', but it might still be workable somehow for your situation - Rob Cole's plugins are generally made to be altered a bit). I do have copies I could zip up from my LR Plugins directory, but without the online documentation, they might be hard to use.. Although OTOH, he likely has put a ton of comments in the script itself.

It does befuddle me though, why LR doesn't have some sort of solution for this built-in - considering it already has an option to make embedded DNG's. But to re-itterate what some other people are saying: the ONLY reason why your DNG file size was doubled was because of embedding the original raw - you're literally storing 2 copies of the data. Without embedding the original, DNG's are usually smaller than proprietary raws.. Even for my Pentax cams which natively save DNG, the 'conversion' will make them smaller still - presumably because of more efficient lossless compression. Also if you want to save on even more space, you could have LR convert to DNG with NO preview and NO fast Load Data -- the no-preview option alone could save about 5 MB or so per file, since otherwise it may store a full-size jpg within the DNG.

 

[gregDT]

I contend that only a fool would throw away the master original Proprietary RAW file in favor of keeping a smaller DNG

Shouldn't that be "I contend that only a fool would throw away the master original Proprietary RAW file in favor of keeping a smaller open source DNG"?

To Thany - to be honest if you're seriously struggling why not just leave things alone. There are numerous image file formats and in the same vein as the utterly pointless iOS vs Android, Apple vs Microsoft arguments it's not that crucial which one you're using assuming they all store your image data. While it's true that Nikon have been utterly useless in offering any sort of decent RAW handling software for years I think the idea of them suddenly dropping the NEF format is slim to the point of being non existent. Likewise Adobe are totally invested in supporting photographers and not a specific file format, so the DNG isn't about to vanish either. Yes it would have been nice if we photographers could have had a universal, open source and future proof file format for all our digital images. But it seems that is not to be.

I work full time as a photographer and my library is full of NEF, ORF, PEF, DNG, TIF and JPEG files. I don't loose any sleep about what format my images are in. I can access them all and I can live with the very, very vague possibility that one or more format owner might somehow one day lock me out of my images files. I might get struck by lightening tomorrow as well

Arguing about which format is the 'proper one to use' as if that's a universal rule for every photographer regardless of who they are or how they work is daft. So if you are having issues converting from format A to format B it might just be worth leaving well alone and stick with what you have. There's really no harm in having different formats.

 

[NJHeart2Heart]

Shouldn't that be "I contend that only a fool would throw away the master original Proprietary RAW file in favor of keeping a smaller open source DNG"?
Arguing about which format is the 'proper one to use' as if that's a universal rule for every photographer regardless of who they are or how they work is daft. So if you are having issues converting from format A to format B it might just be worth leaving well alone and stick with what you have. There's really no harm in having different formats.

Hi GregDT. The OP is not arguing about which format to use. If you take a look at the original post, his concern is about space, and the reason he's trying to change formats is to gain space back on his limited computer disk.

Hope that clarifies things, as JohnBeardy did similarly above.

 

[mcasan]

Lr 6/CC supports OM-D EM-5II. Adoble does not got back and put newer camera support into past releases of Lr and Ps. Nothing new about that. Sometimes when you get a new camera it may take a few months before Adobe includes it. Especially with something like the high res files from EM5II.

https://helpx.adobe.com/photoshop/camera-raw.html

 

[Denis de Gannes]

This was explained in the first post: "I have Lightroom 5 and bought a new camera, the Olympus OM-D E-M5 mark II. This isn't supported by Lightroom (although it is by Photoshop, inexplicably) and therefore I needed to convert each and every picture to DNG before being able to do anything with them in Lightroom"

The best advice here is to upgrade to LR 6 /ACR 9 which provides full support for the Olympus OM-D E-M5 MkII. Using the DNG "crotch" just to get support for a new camera model is only avoiding the real issue.
If you are using Lightroom as your primary raw conversion software then there is no real or perceived need for DNG or XMP. Lightroom saves all your work in the Catalog file, you just need to upgrade to the latest version of Lightroom and backup your catalog file constantly.

This recent video is interesting. https://luminous-landscape.com/single-video/lightroomcc6-17-to-dng-or-not/