Objective: Cognitive impairment has been highlighted as a core feature of bipolar disorder (BD) that often persists during remission. The specific brain correlates of cognitive impairment in BD remain unclear which impedes efficient therapeutic approaches. In a large sample of remitted BD patients, we investigated whether morphological brain abnormalities within dorsal prefrontal cortex (PFC) and hippocampus were related to cognitive deficits. Methods: Remitted BD patients (n = 153) and healthy controls (n = 52) underwent neuropsychological assessment and structural MRI. Based on hierarchical cluster analysis of neuropsychological test performance, patients were classified as either cognitively impaired (n = 91) or cognitively normal (n = 62). The neurocognitive subgroups were compared amongst each other and with healthy controls in terms of dorsal PFC cortical thickness and volume, hippocampus shape and volume, and total cerebral grey and white matter volumes. Results: Cognitively impaired patients displayed greater left dorsomedial prefrontal thickness compared to cognitively normal patients and healthy controls. Hippocampal grey matter volume and shape were similar across patient subgroups and healthy controls. At a whole-brain level, cognitively impaired patients had lower cerebral white matter volume compared to the other groups. Across all participants, lower white matter volume correlated with more impaired neuropsychological test performance. Conclusions: Our findings associate cognitive impairment in bipolar disorder with cerebral white matter deficits, factors which may relate to the observed morphological changes in dorsomedial PFC possibly due to increased neurocognitive effort to maintain symptom stability in these remitted patients.