New camera identifies objects at speed of light, can transform robotics

Scientists have developed a new type of compact camera engineered for computer vision. Developed by scientists from the University of Washington and Princeton University, the prototype uses optics for computing and reduces power consumption. It also enables the camera to identify objects at the speed of light.

Their device also represents a new approach to the field of computer vision, a type of artificial intelligence that allows computers to recognize objects in images and video.

“This is a completely new way of thinking about optics, which is very different from traditional optics. It’s end-to-end design, where the optics are designed in conjunction with the computational block,” said Arka Majumdar, a UW professor in electrical and computer engineering and physics.

“Here, we replaced the camera lens with engineered optics, which allows us to put a lot of the computation into the optics.”

Researchers revealed that instead of using a traditional camera lens made out of glass or plastic, the optics in this camera rely on layers of 50 meta-lenses — flat, lightweight optical components that use microscopic nanostructures to manipulate light. The meta-lenses also function as an optical neural network, which is a computer system that is a form of artificial intelligence modeled on the human brain.

200 times faster than neural networks

They claimed that the unique approach has a couple of key advantages as it’s fast. Because much of the computation takes place at the speed of light, the system can identify and classify images more than 200 times faster than neural networks that use conventional computer hardware, and with comparable accuracy. It also reduces power consumption as the optics in the camera rely on incoming light to operate rather than electricity.

“There are really broad applications for this research, from self-driving cars, self-driving trucks and other robotics to medical devices and smartphones. Nowadays, every iPhone has AI or vision technology in it,” said Felix Heide, an assistant professor of computer science at Princeton University. “This work is still at a very early stage, but all of these applications could someday benefit from what we are developing.”

New computer vision system

Heide stressed that the idea was to use some of the work that Arka pioneered on meta-surfaces to bring some of those computations that are traditionally done electronically into the optics at the speed of light. “By doing so, we produced a new computer vision system that performs a lot of the computation optically.”

Published in Science Advances, the study bridged a major gap by embedding parallelized optical computation into flat camera optics that perform neural network computations during capture before recording on the sensor.

Relevant for autonomous navigation in self-driving vehicles

“We leverage large kernels and propose a spatially varying convolutional network learned through a low-dimensional reparameterization,” said researchers in the study.

“We instantiate this network inside the camera lens with a nanophotonic array with angle-dependent responses. Combined with a lightweight electronic back-end of about 2K parameters, our reconfigurable nanophotonic neural network achieves 72.76% accuracy on CIFAR-10, surpassing AlexNet (72.64%), and advancing optical neural networks into the deep learning era.”

The next steps for this research include further iterations and evolving the prototype so it is more relevant for autonomous navigation in self-driving vehicles. This is an application area they both have identified as promising. They also plan to work with more complex data sets and problems that take greater computing power to solve, such as object detection (locating specific objects within an image), which is an important feature for computer vision, according to a press release by researchers.