Maintaining a youthful brain structure and function throughout life may be the single most important determinant of successful cognitive aging. In this study, we addressed heterogeneity in brain aging by making image-based brain age predictions and relating the brain age prediction gap (BAPG) to cognitive change in aging. Structural, functional, and diffusion MRI scans from 351 participants were used to train and evaluate 5 single-modal and 4 multimodal prediction models, based on 7 regression methods. The models were compared on mean absolute error and whether they were related to physical fitness and cognitive ability, measured both currently and longitudinally, as well as study attrition and years of education. Multimodal prediction models performed at a similar level as single-modal models, and the choice of regression method did not significantly affect the results. Correlation with the BAPG was found for current physical fitness, current cognitive ability, and study attrition. Correlations were also found for retrospective physical fitness, measured 10 years prior to imaging, and slope for cognitive ability during a period of 15 years. The results suggest that maintaining a high physical fitness throughout life contributes to brain maintenance and preserved cognitive ability.