DCNN-GAN: Reconstructing Realistic Image from fMRI
|
|
|
|
Shanghai Jiao Tong University
For MVA 2019
|
|
|
Abstract
Visualizing the perceptual content by analyzing human functional magnetic resonance imaging (fMRI) has been an active research area. However, due to its high dimensionality, complex dimensional structure, and small number of samples available, reconstructing realistic images from fMRI remains challenging. Recently with the development of convolutional neural network (CNN) and generative adversarial network (GAN), mapping multi-voxel fMRI data to complex, realistic images has been made possible. In this paper, we propose a model, DCNN-GAN, by combining a reconstruction network and GAN. We utilize the CNN for hierarchical feature extraction and the DCNN-GAN to reconstruct more realistic images. Extensive experiments have been conducted, showing that our method outperforms previous works, regarding reconstruction quality and computational cost.
Authors
Yunfeng Lin, Jiangbei Li, Hanjing Wang
Video Demo
Code
Please refer to our github repository here.
Paper
Paper in PDF format is availale here (~2MB).
Citation in bibtex availale here.
Acknowledgements
The GAN model is based on the pytorch implementation of pix2pix.
The fMRI data is obtained using the datasets from Generic Object Decoding.