Real-Time Decoding of Snapshot Compressive Imaging Using Tensor FISTA-Net

Abstract

Snapshot compressive imaging (SCI) cameras compress high-speed videos or hyperspectral images into measurement frames. However, decoding the data frames from measurement frames is compute-intensive. Existing state-of-the-art decoding algorithms suffer from low decoding quality or heavy running time or both, which are not practical for real-time applications. In this article, we exploit the powerful learning ability of deep neural networks (DNN) and propose a novel tensor fast iterative shrinkage-thresholding algorithm net (Tensor FISTA-Net) as a real-time decoder for SCI cameras. Since SCI cameras have an accurate physical model, we can trade training time for the decoding time by generating abundant synthetic data and training a decoder on the cloud. Tensor FISTA-Net not only learns a sparse representation of the frames through convolution layers but also reduces the decoding time and memory consumption significantly through tensor operations, which makes Tensor FISTA-Net an appropriate approach for a real-time decoder. Our proposed Tensor FISTA-Net obtains an average PSNR improvement of 0.79&#x2013;2.84 dB (video images) and 2.61&#x2013;4.43 dB (hyperspectral images) over the state-of-the-art algorithms, along with more clear and detailed visual results on real SCI datasets, Hammer and Wheel, respectively. Our Tensor FISTA-Net reaches 45 frames per second in video datasets and <inline-formula> <tex-math notation="LaTeX">$70$</tex-math> </inline-formula> frames per second in hyperspectral datasets, meeting the real-time requirement. Besides, the trained model occupies only a <inline-formula> <tex-math notation="LaTeX">$12$</tex-math> </inline-formula>-MB memory footprint, making it applicable to real-time Internet of Things (IoT) applications.