Global Shutter Image Sensors have been witnessing a high adoption rate due to the rising demand for high-definition image-capturing devices in various industries. They find wide applications not only in smart factories but in smart cities, automotive and all applications that form high-speed processing. STMicroelectronics is addressing computer-vision applications with new high-speed global shutter image sensors.
In a recent online media briefing on ST global shutter image sensor, STMicroelectronics’s Vincent LIN, Technical Marketing Manager, Imaging Division, Asia Pacific explains about global shutter sensor need, application areas and why the Global Shutter image sensor is a preferred imaging technology to realize accurate Computer Vision. He throws detailed light on the different types of cameras used in in-cabin monitoring and their features. Below are excerpts.
Can you share some use cases of computer vision related to Global Shutter Sensor?
Vincent LIN: The Global Shutter Sensor Computer vision use cases can be classified into 4 different application groups. The first is Depth Sensing in which 2 global shutter sensors are used to build a stereo camera system or a single global shutter sensor for a structured light system. Normally the depth sensing application is Face ID and other 3D scanning applications. The second one is biometrics which uses human features as input and performs certain identifications. In an automotive scenario like driver in-cabin monitoring, the input information is specifically the driver and passenger behaviour and the output will be the car’s reminder for the user to get focus or to act. The third one is about the AR/VR/MR it’s related to the Metaverse also. In this advanced application, they are used for eye tracking, hand and face tracking. The last is robotics and industrial where they are used in drones, Barcode, industry 4.0 etc.
How ST’s Global Shutter image sensors is addressing metaverse applications?
Vincent LIN: Regarding the Metaverse, ST provides customer and industrial-grade sensors. So, we have two products as we already mentioned. We have 0.4 mega product and 1.5 mega product. So, the 0.4 mega product is mainly used for eye tracking, and gaze tracking. The 1.5 mega product is used outside for wall tracking, collision avoidance or other applications. These two products are a perfect fit for Metaverse applications.
Why rolling shutter is not used in computer vision and how does it differ from a global shutter?
Vincent LIN: Rolling shutter is designed to capture the still image, and for video recording, thus it features high resolution and a distinguished colour. Smartphone’s main camera, selfie camera surveillance camera and even the digital camera are rolling shutter. The rolling shutter takes the picture line by line, so the picture capture will be finished in 1 frame time, if you wish to capture fast-moving object the image will be deformed and hence cannot be used for computer vision.
Global shutter detects pictures very fast as the camera captures them. It takes a very accurate image of the fast-moving object thus can be used for computer vision.
Tell us about ST Imaging product family and what are currently offerings in global shutter sensors for consumer and industrial?
Vincent LIN: ST Imaging products have 3 different product families, the first one is time of flight product which provides both direct ToF and indirect ToF, the second one is global shutter which caters to automotive and consumers, and the third ambient light sensor for consumer array.
ST Global Shutter product offerings perfectly fit Computer Vision requirements with ST proprietary Pixel technology, higher and square resolution, and multiple embedded Computer Vision features.
ST has 2 products under consumer and industrial first isVD55G0 which is 2.6mm x 2.5mm in size, and another one is VD56G3 which is 3.6mm x 4.3mm. Both products are designed using ST 3D technology, which means the original design has 2 layers of wafer, but ST has combined them to form one wafer hence, for customers it’s still a single wafer.
Generally, computer vision sensor is not in NIR domain but our products offer high sensitivity in NIR domain with the highest quantum efficiency at 940nm for robust Image capture. They also offer square resolution which is ideal for lens matching and rotated use cases. The sensor can provide very short image capture to ensure a very correct image and also very low system power consumption. Apart from the output image these products can also provide output data in form of movement vectors.
What are other application areas they are used in?
Vincent LIN: ST offers global shutter sensor for smart homes and smart buildings which brings assistance & comfort while preserving privacy. Unlike webcam, these specialized cameras enable new human-machine interactions. These sensors are also designed for laptops and PC cameras where they perform facial recognition, gesture and gaze detection, and comfort level detection and privacy.
Explain the role of Global shutter camera for in-cabin monitoring in automotive and ST products offered for in-cabin monitoring?
Vincent LIN: Mainly 4 types of camera sensors used in automotive, starting with in-cabin camera, ADAS camera, viewing camera and aftermarket camera i.e. dash camera, these markets are all very mature, the only in-cabin camera is a new field.
The in-cabin camera market is moving very quickly at the end of 2021 the penetration rate of in-cabin cameras was already around 10%, and in 2024 the penetration rate will become 50%. In-cabin monitoring can be divided into two major types the first one is driver monitoring also called DMS, the other is cabin occupancy monitoring also called CMS or OMS.
The driver monitoring focuses on the driver, the driver’s attention or distraction or drowsiness because these 3 behaviours will affect safe driving. While cabin or occupancy monitoring focuses on passenger monitoring, especially the children when they are left alone in the car.
In DMS camera is installed in front of the driver so the resolution can be 1 ~ 2.3MP and the Lens Field of View can be 50o – 60o but it should provide NIR (Mono) image. ST’s VD5661A with 1.6Mp Mono and VD5761A with 2.3Mp resolution are ideal for it.
In OMS, the passenger in the whole cabin has to be monitored, it should have a wider lens for monitoring the whole cabin and high resolution needs to know all the details in this field. ST’s VD6763A in Colour RGB is a good fit for it.
Another concept is all-in-one camera, which combines all the cameras including the CMS, OMS and DMS in a single camera. ST has VD1762A t which is RGB-NIR that supports both RGB Colour and NIR Mono.
Summing up automotive products, ST has totally 5 products, from 1.5Mp to 2.3Mp, covering all the demands for in-cabin monitoring. All the sensors are in AEC-Q100 grade 2 automotive qualification, HDR, provide Colour and Mono and RGB-NIR. ST sensor can control the NIR illumination, high sensitivity and very outstanding MTF in contrast.
How ST’s Pixel technology is different from its competitors and what is your go-to-market strategy?
Vincent LIN: Our pixel technology is ST’s proprietary technology developed by ST and manufactured in ST 12 inch fab. So, with our technology, we have designed sensors with very high performance, very high sensitivity and very high dynamic range. Also, we can enable an image signal processor and some computer vision IPs. these are all benefits of ST’s technology. ST can use all these technologies to choose the right technology and IPs to develop a customized sensor. For example, a customer wants ST to design a custom sensor. We can reuse all these technologies and build one for specific requirements. In comparison, our competitors are mostly using so-called foundry service. The technology is not unique, and maybe they use the same technology. If they use the same technology, the product they offer will be more similar. Regarding the go-to-market strategy, in addition to the product sample and evaluation kits, ST also provides various reference designs. We will work with camera module makers, lens makers and processor makers. So, we can work together to build a kind of reference design for customers. Our customers just use this reference design to facilitate their product and application designs.
Kindly explain how pixel size, responsible for achievable resolution, is related to sensor size/price. Will price pressure and increasing production costs challenge image sensor OEMs of machine vision cameras in the near future?
Vincent LIN: So maybe we take the example of automotive application. In the market, especially for the entry level driver monitoring cameras, customer wish that the cameras could be smaller and cheaper. What most of sensor suppliers will do is to reduce the pixel size. And when you reduce the pixel size, I think the trade-off is that sensitivity will become worse. When they reduce the pixel size, they still can provide a set of acceptable performance however they can only implement low end SW Algorithm. Some customers may agree with the product offer , because they want to very low price. This is the market. But for ST, we have a different approach. For example, with ST’s second generation automotive products, we have reduced pixel size to reduce sensor cost. But we changed the structure to 3D stacks. We have the benefit that we reduce the size while we can even increase the resolution. We can reduce the size, lower the system cost, and also provide a very good performance. From my point of view, ST is doing better to support customer reduce sot and enhance performance.