How the Lytro Camera sees the World: Raw Data of a LightField Sensor
Lytro’s LightField Sensor consists of an ordinary CMOS imaging sensor, and a so-called microlens array mounted on top of it. Using this combination, it becomes possible to not only record a flat representation of a scene, but also the direction of individual light rays (using complex algorithms).
But what does that really mean, and what exactly does the sensor see?
The camera’s RAW data can be extracted using lfpsplitter (though it is currently lacking an update to work with the newest version of .lfp-files). Since the raw image contains 12-bit data – as opposed to the 8-bit data normally found in processed JPGs – it cannot be displayed in colour per se.
Flickr-User Corby Ziesman has taken the time to extract the RAW sensor data and converted it to a grayscale TIFF file. Then, he used a focus-stack of the JPG layers from the processed -stk.lfp file and superimposed it onto the TIFF-file, revealing what the image sensor actually records: A major picture that consists of about 100,000 sub-images that differ in minute details.
Here I extracted the JPEGs from the LFP file and focus stacked the images, then I extracted the sensor RAW data from the other LFP file and converted it to a TIFF and overlaid the color from the focus stacked image onto the grayscale sensor RAW image. This way you can see how the Lytro actually views the scene with the micro lens array over the sensor. You can even see individual hot pixels.
It is this picture that allows sophisticated algorithms to find and match individual light rays, trace them back
to their source, and create a three-dimensional representation of the scene. Following the light rays back to their original source also lets us virtually shift the focal plane of the camera (software refocus).
See the full version on Flickr: Actual Lytro micro lens array