Sidecars for Lightroom (and Bridge/ACR)

Status
Not open for further replies.
E

ehch

New Member
Joined
Apr 21, 2009
Messages
19
Location
CH
Lightroom Experience
Power User
Lightroom Version
Here are some reasons why LR specific sidecars are needed:

- Many photographers continue to use non-RAW files in addition to RAW (e.g. output from slide scanners).

- LR has XMP sidecars for RAW files only, but not for TIF, JPG and DNG. Lightroom stores metadata of non-raw files in its database and (optionally) in the fileheader.

-- Storing metadata in the fileheader is bad practice:

--- backups are impractical without sidecars because every time you do a "development" update of a photo the entire file must be backed again even though only the header is changed. This is impractical for any but the smallest files. Raw files are typically 1' MB, DNG files are 35 MB, TIF files from scanners are typically 12'MB). Far better to have to back up only the tiny sidecars.

--- metadata in the file header is treated differently by most packages

- it is good practice to write-protect the original file - but then LR cannot store the metadata in the file header; no problem if you have sidecars

- it is good practice to rely on backups of photofiles plus their sidecars in addition to the backup of the LR database.

- it is much easier to move or copy (parts of) a photo collection to another PC by copying a folder (with photofiles and sidecars) than having to go through LR export. Even more so when you have sidecars from other applications in the same folder.

- many photographers use several software packages. Most use sidecars (e.g. DxO Optics Pro, Capture One). Without LR sidecars it remains difficult for photographers to incorporate Lightroom in their day-to-day workflow.

- the existing xmp implementation of LR is only compatible with Bridge/ACR, but not with others (e.g.ACDSee Pro 2.5). So much for a standard... Unless and until this gets fixed it Would be much better for all packages to have their proprietary sidecar format.


Proposed naming scheme for Lightroom sidecars:
Along the lines of DxO's simple but effective naming scheme (1234.NEF.dxo) LR could use something like this:
1234.NEF.lrm
1234.tif.lrm
1234.jpg.lrm
1234.dng.lrm

but NOT:
1234.lrm
 
Sort by date Sort by votes
Edwin, well thought out and written. If you haven't already, you might want to cut and paste the whole thing to the official Adobe Feature Request; link is on the blue menu bar at the top of the thread, above the RPG banner.
 
Upvote 0 Downvote
Brad, thanks.
I prefer to wait to get more reactions to this proposal before submitting to Adobe.
 
Upvote 0 Downvote
I think there are a few flaws in the underlying argument, however! Starting from the last one about the compatibility of LR sidecars which are readable in other applications if the programmers make the effort to read them (eg iView, iMatch, idImager...). We then have assertions about storing metadata in the file header being bad practice - which may be true of mystery meat file formats but is exactly what other formats such as DNG and JPEG are supposed to permit. And then there's the old one about having to back up files just because the file header has been modified - backup the database as it contains all LR's metadata. The last thing we need is LR specific sidecars (or sidecars specific to any one app).

John
 
Upvote 0 Downvote
I don't have sidecar files at all. My camera creates non-proprietary DNG files instead of proprietary NEFs or CR2s. I do not see the need for a separate file at all.
I prefer to keep all of my metadata in the file header with the data. It takes up very little additional room compared to the compared to the Master DNG.
I disagree with the statement that "-- Storing metadata in the fileheader is bad practice" If it were such a bad practice, why are there organizations and standards committees like IPTC and EXIF.org that create and promote these standards in the first place?

I regularly back up my working image folder and my archive image folder. Usually this is done incrementally, but in every 4 backup I create a full backup. A full backup of my working image folder occured last night and my backup utility reported the ~9''' files were backed up in just under 2 hours to a network file server over a wireless (3'' m/bit) connection. This does not seem to be an obstacle for me.

Now, I am writing all my metadata changes including sometimes multiple snapshot settings to my Master RAW file inported from my camera card. When I import, I create a copy of the original onto a network backup drive, before I erase the card. So I always start with two copies of the original. At some point, I a have a backup of the working master and I feel free to delete the second copy of the original if space on the network drive is a problem. (With 3.5TB it won't be for the forseeable future).

I don't mind writing to my master file. It keeps everything consolidated and I don't have to rely on LR to access the adjustment filters. I like the DNG as a master since I do not have the faith that proprietary RAW formats will remain constant and that future software (LR v6?) will be able to read a NEF, PEF, CR2, etc. if these are long obsolete.

I am new to LR and manage about 9'% of my post processing with it now. I use it for 1''% of my catalog requirements. I even catalog derivatives where appropriate and for derivatives created with earlier post processing software. I find that I am able to track and identify post processed image versions by maintain names snapshot adjustments in bothe the master DNG and the derivative image file. If I export to websites like Flickr, I can pass that version information to the website in comments or otherwise. My version tracking abilities are limited when it comes to print media but still possible. If I have access to my version information, I can pull up and reproduce duplicate copies from the Master DNG years from now and with what ever software is available, even if I no longer have a LR catalog.

I am opposed to any proprietary formats. They are not readily accessable by other programs and must be reverse engineered to be used outside the original purpose. The reason that Sidecar files exist in the firat place is because RAW proprietary formats are not extendable. I favor Open standards and appreciate Adobe for its efforts to create an Open RAW standard in the DNG. Some day (soon I hope) Nikon and Canon will create DNG files with their camera firmware, maybe even before there is a CR3. Several other camera manufactureres already do.
It is my understanding the DNG, NEF and CR2 are all extensions of the TIFF format standard. Since I do not work with proprietary RAW formats, as I understand it, you can create a DNG standard Master file on import with Lightroom. At some point it would be nice to see other programs like Capture One Optics Pro, Bibble or LightZone be able to work natively with a DNG file that contains all the prior adjustment information for a variety of sources. It might even be possible for printing software to read DNG instead of TIFF
 
Upvote 0 Downvote
There are two camps, those that like DNG and those that don't. there have been some interesting debates on other forums. I am in the latter camp.
 
Upvote 0 Downvote
One thing confuses me about the original post. It states that "Raw files are typically 1' MB, DNG files are 35 MB". I was under the impression that DNG files were smaller than RAW files. Am I mistaken?
 
Upvote 0 Downvote
[quote author=Roy Mathers link=topic=7539.msg5169'#msg5169' date=125'37'418]
One thing confuses me about the original post. It states that "Raw files are typically 1' MB, DNG files are 35 MB". I was under the impression that DNG files were smaller than RAW files. Am I mistaken?
[/quote]
Roy, typically DNG files are smaller than Raw files. Unless the OP has embedded the Raw within the DNG,
 
Upvote 0 Downvote
[quote author=Roy Mathers link=topic=7539.msg51694#msg51694 date=125'373'94]
Thanks Geoff. (But why would you want to do that? Why would you want both>)
[/quote]

For those that don't trust the DNG convertion, they like to retain the original RAW.
 
Upvote 0 Downvote
[quote author=Kiwigeoff link=topic=7539.msg51693#msg51693 date=125'372991]
[quote author=Roy Mathers link=topic=7539.msg5169'#msg5169' date=125'37'418]
One thing confuses me about the original post. It states that "Raw files are typically 1' MB, DNG files are 35 MB". I was under the impression that DNG files were smaller than RAW files. Am I mistaken?
[/quote]
Roy, typically DNG files are smaller than Raw files. Unless the OP has embedded the Raw within the DNG,
[/quote]DNG Files are RAW files. Proprietary RAW files like NEFs and CR2s are compressed while DNG can come in two flavors Compressed and Uncompressed. DNGs, NEFs and CR2s are extensions of the TIFF format specification and can contain all of the elements in the TIFF including a JPEG of the image. A NEF, CR2 or other RAW files like the DNGs that come from my Pentax are 12-14 bit while an uncompressed DNG is likely 16 bit like the TIFF specification. My Pentax can output a PEF Raw format or a DNG Raw format, typically the PEF will be around 9 - 11 MB while a typical DNG form the camera is ~16-22 MB probably because the DNG contains a JPEG thumbnail. If I convert a DNG that came from my camera into a 16 bit TIFF, it is always 2-3 times larger that the compressed DNG master file. If I take the number of pixels on my sensor and multiply by 12 bits X3 and divide by the number of bits in a Byte (16) I get a files size that is ~32 MB not counting any EXIF header (which is trivial) The lower number of the actual is due to compression.

Now all of these compressed RAW files are supposed to be Lossless. I am not able to prove or disprove this.
 
Upvote 0 Downvote
[quote author=Roy Mathers link=topic=7539.msg5169'#msg5169' date=125'37'418]
One thing confuses me about the original post. It states that "Raw files are typically 1' MB, DNG files are 35 MB". I was under the impression that DNG files were smaller than RAW files. Am I mistaken?
[/quote]

Linear DNG's are generally much bigger because they're no longer mosaiced data. Embedded preview size and compression can also affect the DNG size.
 
Upvote 0 Downvote
DNG sizes. My mistake: the 35 MB was the output from DxO. The DNG made by LR from the same NEF file is only 7.5 MB, even smaller than the NEF (8.5 MB).
 
Upvote 0 Downvote
That makes sense, the DNG's from DXO are linear DNGs
 
Upvote 0 Downvote
Status
Not open for further replies.
Back
Top