Bipolar disorder (BD) and major depressive disorder (MDD) are associated with cognitive and functional impairment. Cognitive impairment is often associated with dorsal prefrontal cortex (dPFC) hypo-activity, but the neuronal correlates of functional disability is largely unknown. In this study, 91 patients with affective disorders in full or partial remission (BD, n = 67; MDD, n = 24) with objectively verified cognitive impairment and substantial functional disability underwent neuropsychological assessment and functional magnetic resonance imaging (fMRI) scan during which they completed a strategic picture-encoding task. For comparison, 36 matched healthy controls underwent an identical test protocol. Patients showed encoding-related hypo-activity in the dPFC compared to controls. In patients, lower right dlPFC activity was associated with poorer overall functioning and more antipsychotic drug use. In conclusion, memory impairments were underpinned by failure to recruit the dPFC during task performance which was associated with impaired functioning in fully or partially remitted patients with affective disorders. This aberrant neurocircuitry activity has implications for the design of future pro-cognitive interventions that aim to improve not only cognition but also real-world functioning.