Background: Cognitive impairment affects many patients with bipolar disorder (BD). No pro-cognitive treatment with robust efficacy exists partly due to limited insight into underlying neurobiological abnormalities. Methods: This magnetic resonance imaging (MRI) study investigates structural neuronal correlates of cognitive impairment in BD by comparing brain measures in a large sample of cognitively impaired versus cognitively intact patients with BD or cognitively impaired patients with major depressive disorder (MDD) and healthy controls (HC). Participants underwent neuropsychological assessments and MRI scans. The cognitively impaired and – intact BD and MDD patient groups were compared with each other and HC regarding prefrontal cortex measures, hippocampus shape/volume, and total cerebral white (WM) and grey matter (GM). Results: Cognitively impaired BD patients showed lower total cerebral WM volume than HC, which scaled with poorer global cognitive performance and more childhood trauma. Cognitively impaired BD patients also showed lower adjusted GM volume and thickness in the frontopolar cortex than HC but greater adjusted GM volume in the temporal cortex than cognitively normal BD patients. Cognitively impaired BD patients showed decreased cingulate volume than cognitively impaired MDD patients. Hippocampal measures were similar across all groups. Limitations: The cross-sectional study design prevented insights into causal relationships. Conclusions: Lower total cerebral WM and regional frontopolar and temporal GM abnormalities may constitute structural neuronal correlates of cognitive impairment in BD, of which the WM deficits scale with the degree of childhood trauma. The results deepen the understanding of cognitive impairment in BD and provide a neuronal target for pro-cognitive treatment development.