Being at the right spot at the right time to make a photograph is always challenging. Light is nonetheless one of the key components in achieving a professional look. Catching the sunset light, or aligning sun rays with some objects often require the photogrtapher to wait long hours for the best conditions. With smartphone fitted with increasingly better cameras and the advent of social networks, more and more users take and share pictures. They often use filters to beautify their pictures, but these filters cannot change the image’s content. Thanks to machine learning --- more specifically a deep neural network --- researchers have developed a method that pushes the boundary of what is possible with image filters and allows the lighting of a photo or  video to be modified after the capture.

The first results presented at the SIGGRAPH conference

Julien Philip, a Ph.D student under the supervision of Dr George Drettakis in the GraphDeco project-team at Inria Sophia Antipolis, is the main contributor to the paper describing the method. He presented their results in Los Angeles at the SIGGRAPH conference in early August. SIGGRAPH (Special Interest Group on Computer GRAPHics and Interactive Techniques) is a major international conference on computer graphics: since 1974, it has brought together every year a large number of industrial, artistic and scientific players in the world of computer graphics, whether they are animated films, the creation of special effects, video games or software for 3D modelling

The method remains experimental but already produces stunning results, says Julien. “Often neural networks can only process small images with a quality that is insufficient for photographic applications, or they focus on lower level tasks like denoising. We give back control of the lighting to the users: they can now decide whether the picture will look as if it were shot in the morning, at noon or at sunset, long after capturing it. As a matter of fact, users can let their creativity drive them and imagine lightings that are beyond reality”.

Supported by Michaël Gharbi, an Adobe researcher, Tinghui Zhou and Alexei Efros from UC Berkeley, and my his thesis advisor George Drettakis, he showed that a single photo was not sufficient to obtain convincing results with current methods. To overcome this difficulty, they use more photos (or “views”) of the same place to estimate its underlying 3D and guide the relighting. These multiple “views” can be obtained by recording a video, taking several pictures, or even downloading images of the same place from the internet.

The algorithm can then be used to alter a photo, generate a time lapse effect or edit a video. The method can also be adapted to traditional multi-views pipelines such as “Image Based Rendering (IBR)” or photogrammetry, which is particularly used in visual effects andcould lead to future industrial applications.

A light-and-shadow game

Producing a realistic modification of cast shadows is a core challenge when trying to alter the lighting of a shot. The method is able to remove and modify them to simulate another lighting. They guide their algorithm using 3D et applying methods close to the ones commonly used in video games for shadow computation. Unfortunately, these methods are not directly applicable. Julien comments that “The 3D we obtain is not sufficiently good to remove or synthesize realistic shadows, though it provides solid cues. That is where the neural network comes in, it was taught to correct the errors due to the low quality of the 3D”. To train their artificial intelligence to estimate the transformations to apply, they needed examples of places under a wide variety of lighting conditions. This type of data is hard and expensive to acquire, so they used highly realistic rendering methods that simulate the physic of light instead of real photos. This allowed them to gather enough data for the neural network to properly learn to relight images despite poor 3D reconstruction.