基于普通器件实现快1000倍的相机与机器视觉  

1000×Faster Camera and Machine Vision with Ordinary Devices

作　　者：Tiejun Huang Yajing Zheng Zhaofei Yu Rui Chen Yuan Li Ruiqin Xiong Lei Ma Junwei Zhao Siwei Dong Lin Zhu Jianing Li Shanshan Jia Yihua Fu Boxin Shi Si Wu Yonghong Tian 

机构地区：[1]School of Computer Science,National Engineering Research Center of Visual Technology,Peking University,Beijing 100871,China

出　　处：《Engineering》2023年第6期110-119,M0005,共11页工程（英文）

基　　金：supported by projects of the National Natural Science Foundation of China(61425025);the Beijing Municipal Science and Technology Project(Z151100000915070 and Z171100000117008)。

摘　　要：在数码相机中,我们发现了一个重大缺陷,即从胶片相机继承的图像和视频模型阻碍了相机捕捉快速变化的光子世界。我们提出了一种新的视觉形式,称为视象(vform),这是一个比特序列阵列,其中每个比特表示光子的累积是否达到了一个阈值,从而可以记录和重建任何时刻场景的光强。仅使用消费级CMOS传感器和集成电路,开发了一种比传统相机快1000倍的脉冲相机。将视象看作生物视觉中的脉冲序列,进一步开发了基于脉冲神经网络的机器视觉系统,它可以将机器的速度和生物视觉的机理结合起来,从而实现了比人类视觉快1000倍的高速目标检测和跟踪,并通过辅助裁判和目标瞄准系统证明了脉冲相机和超级视觉系统的效用。视象模型和芯片有望从根本上改变图像和视频的概念以及摄影、电影和视觉媒体等相关行业,并开启一个全新的基于脉冲神经网络的速度自由的机器视觉时代。In digital cameras,we find a major limitation:the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world.Here,we present vform,a bit sequence array where each bit represents whether the accumulation of photons has reached a threshold,to record and reconstruct the scene radiance at any moment.By employing only consumer-level complementary metal–oxide semiconductor(CMOS)sensors and integrated circuits,we have developed a spike camera that is 1000×faster than conventional cameras.By treating vform as spike trains in biological vision,we have further developed a spiking neural network(SNN)-based machine vision system that combines the speed of the machine and the mechanism of biological vision,achieving high-speed object detection and tracking 1000×faster than human vision.We demonstrate the utility of the spike camera and the super vision system in an assistant referee and target pointing system.Our study is expected to fundamentally revolutionize the image and video concepts and related industries,including photography,movies,and visual media,and to unseal a new SNN-enabled speed-free machine vision era.

关 键 词：脉冲神经网络 机器视觉 生物视觉 视觉系统 CMOS传感器 视觉媒体 胶片相机 脉冲序列 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]