Tesseract: Light Field Photography, Automated Subject Extraction and 3D Filters for your Smartphone

by markus · 2014-03-25

Remember Focii? The innovative light field capture method, developed by Kshitij Marwah at MIT Media Lab, allows users to convert ordinary cameras into light field cameras just by placing a coded mask film on top of the image sensor. Marwah has successfully ported Focii to a smartphone, and recently launched Tesseract Imaging to make the technology commercially available.
In a recent presentation at INKtalk, Kshitij Marwah presented the new product, and demonstrated three features on his modified Android smartphone (details below), live on stage:

Presented features:

Bokeh: software refocus and DSLR-like shallow depth of field
Chroma: “3D” color filters à la Instagram, with automatic separation of fore- and background elements
Teleport: automated extraction of foreground elements and background replacement

We’ve reached out to Marwah for some additional information:
The prototype in the videos is a modified HTC EVO 3D, where the “3D” hardware switch is used to activate light field recording. Due to the age of this particular phone, images are downsized to 1 MP before processing, but Marwah notes that current smartphones such as the Samsung Galaxy S4 or iPhone 5 should possess enough computational power to render light fields at full sensor resolution.
The printed mask itself is very cheap in production at approx. 0.10 US-Dollars per unit, excluding integration into a camera module and calibration. This is a major difference to other light field recording solutions, especially in the mobile market where space is limited. Tesseract is already in talks with manufacturers and OEMs.
When asked what future features he envisions for the technology, Marwah showed special excitement about bringing the presented photo features (Bokeh, Chroma, and Teleport) to video on a smartphone. “That would be a game changer.”