Methodology
The framework transforms fMRI BOLD signals into beta weights, maps them to a VAE latent space using a non-linear GRU network, and reconstructs the image using a Latent Diffusion Model conditioned on CLIP embeddings.
Objective. AI-based neural decoding reconstructs visual perception by leveraging generative models to map brain activity measured through functional magnetic resonance imaging (fMRI) into the observed visual stimulus.
Approach. Traditionally, ridge linear models transform fMRI into a latent space, which is then decoded using variational autoencoders (VAE) or LDMs. Owing to the complexity and noisiness of fMRI data, newer approaches split the reconstruction into two sequential stages, the first one providing a rough visual approximation using a VAE, the second one incorporating semantic information through the adoption of LDM guided by contrastive language-image pre-training (CLIP) embeddings. This work addressed some key scientific and technical gaps of the two-stage neural decoding by: (1) implementing a gated recurrent unit (GRU)-based architecture to establish a non-linear mapping between the fMRI signal and the VAE latent space, (2) optimizing the dimensionality of the VAE latent space, (3) systematically evaluating the contribution of the first reconstruction stage, and (4) analyzing the impact of different brain regions of interest (ROIs) on reconstruction quality.
Main results. Experiments on the NSD dataset, containing 73,000 unique natural images, along with fMRI of eight subjects, demonstrated that the proposed architecture maintained competitive performance while reducing the complexity of its first stage by 85%. The sensitivity analysis showcased that the first reconstruction stage is essential for preserving high structural similarity in the final reconstructions. Restricting analysis to semantic ROIs, while excluding early visual areas, diminished visual coherence, preserving semantics though. The inter-subject repeatability across ROIs was about 92% and 98% for visual and semantic metrics, respectively.
Significance. This study represents a key step toward optimized neural decoding architectures leveraging non-linear models for stimulus prediction. Sensitivity analysis highlighted the interplay between the two reconstruction stages, while ROI-based analysis provided strong evidence that the two-stage AI model reflects the brain’s hierarchical processing of visual information.
The framework transforms fMRI BOLD signals into beta weights, maps them to a VAE latent space using a non-linear GRU network, and reconstructs the image using a Latent Diffusion Model conditioned on CLIP embeddings.
Comparison against State-of-the-Art methods on the NSD dataset.
| Method | SSIM ↑ | CLIP ↑ | Parameters ↓ |
|---|---|---|---|
| Takagi et al. [15] | 0.325 | 0.770 | - |
| Ozcelik et al. [19] | 0.356 | 0.915 | 1.4B |
| Ours (15 latents) | 0.352 | 0.921 | 12.4M (-99%) |
Comparison of Ground Truth, Stage 1 (VDVAE), and Stage 2 (Final LDM).
@Article{Veronese2025,
author = {Veronese, Lorenzo and Moglia, Andrea and Pecco, Nicolò and Anthony Della Rosa, Pasquale and Scifo, Paola and Mainardi, Luca and Cerveri, Pietro},
journal = {Journal of Neural Engineering},
title = {Optimized AI-based neural decoding from BOLD fMRI signal for analyzing visual and semantic ROIs in the human visual system},
year = {2025},
issn = {1741-2552},
month = aug,
number = {4},
pages = {046048},
volume = {22},
doi = {10.1088/1741-2552/adfbc2},
publisher = {IOP Publishing},
}
Lorenzo Veronese
:
Conceived the study, developed the methodology, implemented experiments, performed analysis, and edited the manuscript.
Andrea Moglia: Assisted in paper review.
Nicolò Pecco
:
Participated in data analysis and synthesis, and reviewed the manuscript.
Pasquale Anthony Della Rosa: Provided insights on cognitive neuroscience and fMRI imaging, and reviewed the manuscript.
Paola Scifo: Provided insights on fMRI imaging.
Luca Mainardi: Assisted in the paper reviewing process.
Pietro Cerveri: Conceived the study, supervised the work, and provided financial support.