Camera: Sony Image Sensor— AI Initiatives — ECCV2020

Sony Semiconductor will show the unique intelligent vision sensor, which realizes high-speed edge AI processing. Sony R&D Center will introduce cutting edge algorithms and implementations for next generation entertainment. Sony AI will talk about the concept of Flagship Projects to leverage assets with AI for visionary applications.

Session 1
Intelligent Vision Sensor

Sony has developed innovative image sensor called intelligent vision sensor. They are the first image sensors in the world to be equipped with AI processing functionally. The sensor was realized by utilizing Sony’s advanced stacking technology and original DSP for AI. Including AI processing functionality on the image sensor itself enables high-speed edge AI processing and extraction of only the necessary data, reduces data transmission latency, addresses privacy concerns, and reduces power consumption and communication costs in using cloud services. These products expand the opportunities to develop AI-equipped cameras, enabling a diverse range of applications in the retail and industrial equipment industries and contributing to building optimal systems that link with the cloud.

Fig. Intelligent Vision Sensor

Session 2
Online and Offline 3D Mapping

Sony RDC is doing research in the field of Computer Vision. Among others, we have implemented state-of-the-art systems in 3D modelling; both: online and offline reconstruction. For rendering possible 360˚ reconstruction and sensing in real time with low power consumption, we have developed a processor specifically for vision sensing. This processor can connect up to 12 cameras, IMUs, etc., and performs stereo depth estimation, SLAM, occupancy mapping, etc., simultaneously. The focus of our offline 3D modelling system comprises the reconstruction of high-resolution as well as large sets of images. We have developed effective solutions on Multi-View Stereo, depth refinement and mesh reconstruction and show superior results on well-known academic benchmarks. Combining online and offline reconstruction enables a unique VR experience and extended robotic applications.

Session 3
Sony AI

Sony AI is a new organization, established in November 2019, with offices in Japan, the US and Europe. With a mission to unleash human imagination and creativity, we believe in researching and developing AI techniques that empower artists, makers and creators around the world. In this session, I will introduce Sony AI’s approach to AI research, and discuss some of our first projects.

Business use case

Case 1
Sony’s World’s First Intelligent Vision Sensors with AI Processing Functionality
Enabling High-Speed Edge AI Processing and Contributing to Building of Optimal Systems Linked with the Cloud

Sony Corporation announced two models of intelligent vision sensors, the first image sensors in the world to be equipped with AI processing functionality. Including AI processing functionality on the image sensor itself enables high-speed edge AI processing and extraction of only the necessary data, which, when using cloud services, reduces data transmission latency, addresses privacy concerns, and reduces power consumption and communication costs.

Fig. Intelligent Vision Sensor

The spread of IoT has resulted in all types of devices being connected to the cloud, making commonplace the use of information processing systems where information obtained from such devices is processed via AI on the cloud. On the other hand, the increasing volume of information handled in the cloud poses various problems: increased data transmission latency hindering real-time information processing; security concerns from users associated with storing personally identifiable data in the cloud; and other issues such as the increased power consumption and communication costs cloud services entail. The new sensor products feature a stacked configuration consisting of a pixel chip and logic chip. They are the world’s first image sensor to be equipped with AI image analysis and processing functionality on the logic chip. The signal acquired by the pixel chip is processed via AI on the sensor, eliminating the need for high-performance processors or external memory, enabling the development of edge AI systems. The sensor outputs metadata (semantic information belonging to image data) instead of image information, making for reduced data volume and addressing privacy concerns. Moreover, the AI capability makes it possible to deliver diverse functionality for versatile applications, such as real-time object tracking with high-speed AI processing. Different AI models can also be chosen by rewriting internal memory in accordance with user requirements or the conditions of the location where the system is being used.

Case 2
Using imaging information
that humans can’t see in AI applications
Polarization image sensor

1) Polarization Image Sensor with Four-Directional on-chip Polarizer and global shutter function

Sony Semiconductor Solutions has launched a polarization image sensor (polarization sensor): 3.45µm pixel size with four-directional polarizer which is formed on the photodiode of the image sensor chip*1. This polarization sensor is targeting the industrial equipment market. In addition to capturing brightness and color*2, this image sensor can also capture polarization information that cannot be detected by a normal image sensor. This polarization sensor can be used in many applications in the industrial field, such as inspection when visibility and sensing are difficult.

2) Developing high-performance AI as we develop products

In many cases, AI has been developed by applying machine learning to images visible to the human eye. Without a doubt however, using polarization sensors to take advantage of image information that we cannot see will greatly expand the horizons of AI. To put it another way, instead of this technology solving our problems for us, the technology will expand our own ability to solve problems. We are picking up the pace in wholly new AI development that on draws on our unmatched development of high-performance sensors. And in consideration of the many different products and services Sony offers, we are uniquely positioned to rapidly develop practical, high-performance AI as we develop products and launch them around the world. Unlike AI development that relies solely on image processing or machine learning, this makes our work very satisfying.
Polarsens is a CMOS Image Sensor pixel technology that has several different angle polarizer formed on chip during the semiconductor process allowing highly accurate alignment with pixel.

3) Example of Polarization Direction

Polarization direction provides the direction information of reflected plane of an object. The direction of polarization image (Fig. “Normal Image”) shows the angle of the polarization direction in color using HSV color mapping (Fig. “Polarization Direction image”). In this example, the upper side of the cube is highlighted in light blue meaning that the angle of the polarization direction is 90 degree (according to Fig. “HSV color mapping”).

Case 3
Eye AF in “Alpha” Series and “Xperia 1”
Focus on Shooting with Eye AF and Capturing the Decisive Moment of Kando

1) Certain Success through AF Technology The Strong Belief Supporting Advanced Technology

Eye AF refers to the camera’s ability to detect subjects’ eyes and focus perfectly on them. We produce all of the core elements at our company including the CMOS sensors and image processing engines and lenses, and as we optimize every individual step, we can draw out the very best AF accuracy and tracking. The Eye AF feature is highly rated for its ability to look after all focusing issues, leaving the photographer free to concentrate on composition and the subject’s expression. The evolved real-time Eye AF found on Alpha 9 II cameras can keep up with even the fastest athletes, quickly and accurately focusing and tracking their eyes. Sony’s Eye AF technology has since been used in more cameras and has been well received by professionals and many more.

Fig. Sony’s full-frame mirrorless camera, Alpha 9 II

This full-frame mirrorless camera has a game-changing image sensor capable of a new dimension of photographic speed. With high-speed subject recognition and lightning fast AI-based data processing, tracking subjects’ eyes in real time. In addition, the camera is equipped with other unique features such as silent shooting which omits shutter noise and unwanted vibration, expanding photographic potential.

2) Developing New Face Detection and Realizing the World’s First Smartphone Eye AF

Sony’s flagship Xperia 1 smartphone is the world’s first smartphone with Eye AF. Eye AF is also well known as one of the signature features of Sony’s “Alpha” series cameras. Normal cameras are held either vertically or horizontally, but smartphones are totally free. Xperia 1 introduced face and eye detection technology that would work seamlessly for faces and eyes at any angle. Getting real-time processing onto the device was a struggle, but by making use of our base technology, we succeeded. Furthermore, Xperia 1 II, the successor to the Xperia 1, is equipped with “Real-time Eye AF” which locks focus on the subject’s eye for stunning portrait shots, is now available for both humans and animals.

Case 4
Image Recognition Technology on aibo
~ Building Eyes of aibo to Live in Homes ~

aibo uses its eyes (cameras) to recognize the surroundings and modifies the behavior based on what it sees. It identifies owners and detects friends (other aibos) and toys to play with. Edge-friendly lightweight algorithm based on deep learning allows to recognize rapidly to interact with people. With the depth sensor, it can reach to the toys (ex. Pink balls/bones) while avoiding obstacles. A camera on the back realizes SLAM for building the map of the room and it can be used for go-to-charger and patrol application.

Originally published at https://www.sony.net.