Neuralizer: General Neuroimage Analysis without Re-Training

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Neuroimage processing tasks like segmentation, reconstruction, and registration are central to the study of neuroscience. Robust deep learning strategies and architectures used to solve these tasks are often similar. Yet, when presented with a new task or a dataset with different visual characteristics, practitioners most often need to train a new model, or fine-tune an existing one. This is a time-consuming process that poses a substantial barrier for the thousands of neuroscientists and clinical researchers who often lack the resources or machine-learning expertise to train deep learning models. In practice, this leads to a lack of adoption of deep learning, and neuroscience tools being dominated by classical frameworks. We introduce Neuralizer, a single model that generalizes to previously unseen neuroimaging tasks and modalities without the need for retraining or fine-tuning. Tasks do not have to be known a priori, and generalization happens in a single forward pass during inference. The model can solve processing tasks across multiple image modalities, acquisition methods, and datasets, and generalize to tasks and modalities it has not been trained on. Our experiments on coronal slices show that when few annotated subjects are available, our multi-task network outperforms task-specific baselines without training on the task.

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
Number of pages14
PublisherIEEE Computer Society Press
Publication date2023
ISBN (Electronic)979-8-3503-0129-8
Publication statusPublished - 2023
Event2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 - Vancouver, Canada
Duration: 18 Jun 202322 Jun 2023


Conference2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
SponsorAmazon Science, Ant Research, Cruise, et al., Google, Lambda
SeriesProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

    Research areas

  • cell microscopy, Medical and biological vision

ID: 389367610