Objectives: Cognitive impairment occurs in approximately 50% of remitted patients with bipolar disorder (BD). However, there exists no treatment with replicated and robust efficacy on cognition in BD. This is partially due to limited insight into the neuronal underpinnings of cognitive impairment in these patients. This is the first study to investigate neuronal underpinnings of cognitive impairment in a large functional magnetic resonance imaging (fMRI) dataset comparing neural activity patterns between distinct neurocognitive subgroups of partially or fully remitted patients with BD. Methods: Patients (n = 153) and healthy controls (HC) (n = 52) underwent neuropsychological assessment and fMRI, during which they performed a verbal N-back working memory (WM) task. Based on hierarchical cluster analysis of neuropsychological test performance, patients were grouped into one of two neurocognitive subgroups (cognitively impaired, n = 91; cognitively normal compared to HC, n = 62) that were compared on WM-related neural activity. Results: Cognitively impaired patients displayed WM-related hypo-activity in left dorsolateral prefrontal cortex and frontal and parietal regions within a cognitive control network (CCN) as well as hyper-activity in the default mode network (DMN) compared to cognitively normal patients. In contrast, cognitively normal patients only exhibited hypo-activity within a small cluster in the superior frontal gyrus relative to HC. Conclusions: Cognitive impairment in BD seems to originate from a failure to recruit key regions in the CCN and to suppress task-irrelevant DMN activity during cognitive performance. These results highlight modulation of aberrant dorsal prefrontal and DMN activity as a putative target for pro-cognitive treatment in BD.