Camera: Sony Image Sensor— AI Initiatives — ECCV2020

Sony Semiconductor will show the unique intelligent vision sensor, which realizes high-speed edge AI processing. Sony R&D Center will introduce cutting edge algorithms and implementations for next generation entertainment. Sony AI will talk about the concept of Flagship Projects to leverage assets with AI for visionary applications.

Session 1
Intelligent Vision Sensor

Fig. Intelligent Vision Sensor

Session 2
Online and Offline 3D Mapping

Session 3
Sony AI

Business use case

Case 1
Sony’s World’s First Intelligent Vision Sensors with AI Processing Functionality
Enabling High-Speed Edge AI Processing and Contributing to Building of Optimal Systems Linked with the Cloud

Fig. Intelligent Vision Sensor

The spread of IoT has resulted in all types of devices being connected to the cloud, making commonplace the use of information processing systems where information obtained from such devices is processed via AI on the cloud. On the other hand, the increasing volume of information handled in the cloud poses various problems: increased data transmission latency hindering real-time information processing; security concerns from users associated with storing personally identifiable data in the cloud; and other issues such as the increased power consumption and communication costs cloud services entail. The new sensor products feature a stacked configuration consisting of a pixel chip and logic chip. They are the world’s first image sensor to be equipped with AI image analysis and processing functionality on the logic chip. The signal acquired by the pixel chip is processed via AI on the sensor, eliminating the need for high-performance processors or external memory, enabling the development of edge AI systems. The sensor outputs metadata (semantic information belonging to image data) instead of image information, making for reduced data volume and addressing privacy concerns. Moreover, the AI capability makes it possible to deliver diverse functionality for versatile applications, such as real-time object tracking with high-speed AI processing. Different AI models can also be chosen by rewriting internal memory in accordance with user requirements or the conditions of the location where the system is being used.

Case 2
Using imaging information
that humans can’t see in AI applications
Polarization image sensor

Sony Semiconductor Solutions has launched a polarization image sensor (polarization sensor): 3.45µm pixel size with four-directional polarizer which is formed on the photodiode of the image sensor chip*1. This polarization sensor is targeting the industrial equipment market. In addition to capturing brightness and color*2, this image sensor can also capture polarization information that cannot be detected by a normal image sensor. This polarization sensor can be used in many applications in the industrial field, such as inspection when visibility and sensing are difficult.

2) Developing high-performance AI as we develop products

In many cases, AI has been developed by applying machine learning to images visible to the human eye. Without a doubt however, using polarization sensors to take advantage of image information that we cannot see will greatly expand the horizons of AI. To put it another way, instead of this technology solving our problems for us, the technology will expand our own ability to solve problems. We are picking up the pace in wholly new AI development that on draws on our unmatched development of high-performance sensors. And in consideration of the many different products and services Sony offers, we are uniquely positioned to rapidly develop practical, high-performance AI as we develop products and launch them around the world. Unlike AI development that relies solely on image processing or machine learning, this makes our work very satisfying.
Polarsens is a CMOS Image Sensor pixel technology that has several different angle polarizer formed on chip during the semiconductor process allowing highly accurate alignment with pixel.

3) Example of Polarization Direction

Polarization direction provides the direction information of reflected plane of an object. The direction of polarization image (Fig. “Normal Image”) shows the angle of the polarization direction in color using HSV color mapping (Fig. “Polarization Direction image”). In this example, the upper side of the cube is highlighted in light blue meaning that the angle of the polarization direction is 90 degree (according to Fig. “HSV color mapping”).

Case 3
Eye AF in “Alpha” Series and “Xperia 1”
Focus on Shooting with Eye AF and Capturing the Decisive Moment of Kando

Eye AF refers to the camera’s ability to detect subjects’ eyes and focus perfectly on them. We produce all of the core elements at our company including the CMOS sensors and image processing engines and lenses, and as we optimize every individual step, we can draw out the very best AF accuracy and tracking. The Eye AF feature is highly rated for its ability to look after all focusing issues, leaving the photographer free to concentrate on composition and the subject’s expression. The evolved real-time Eye AF found on Alpha 9 II cameras can keep up with even the fastest athletes, quickly and accurately focusing and tracking their eyes. Sony’s Eye AF technology has since been used in more cameras and has been well received by professionals and many more.

Fig. Sony’s full-frame mirrorless camera, Alpha 9 II

This full-frame mirrorless camera has a game-changing image sensor capable of a new dimension of photographic speed. With high-speed subject recognition and lightning fast AI-based data processing, tracking subjects’ eyes in real time. In addition, the camera is equipped with other unique features such as silent shooting which omits shutter noise and unwanted vibration, expanding photographic potential.

2) Developing New Face Detection and Realizing the World’s First Smartphone Eye AF

Sony’s flagship Xperia 1 smartphone is the world’s first smartphone with Eye AF. Eye AF is also well known as one of the signature features of Sony’s “Alpha” series cameras. Normal cameras are held either vertically or horizontally, but smartphones are totally free. Xperia 1 introduced face and eye detection technology that would work seamlessly for faces and eyes at any angle. Getting real-time processing onto the device was a struggle, but by making use of our base technology, we succeeded. Furthermore, Xperia 1 II, the successor to the Xperia 1, is equipped with “Real-time Eye AF” which locks focus on the subject’s eye for stunning portrait shots, is now available for both humans and animals.

Case 4
Image Recognition Technology on aibo
~ Building Eyes of aibo to Live in Homes ~

Originally published at



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
For 4D Sensing, AR, AV/Robot, and AIoT

Marc. Y. , Tech Leader. Focus on new techs such as 4D Sensing, AR, AV/Robot, and AIoT. More at