Background: Patients with bipolar disorder (BD) who are presenting with cognitive impairment and associated structural brain abnormalities have generally a poorer clinical outcome. This study aims to map the early longitudinal trajectories in brain structure and cognition in patients with recently diagnosed BD. Methods: Fully or partially remitted patients with a recent diagnosis of BD and matched healthy controls (HC) underwent structural MRI and neuropsychological testing at baseline (BD n = 97; HC n = 66) and again following an average of 16 (range 6–27) months (BD n = 50; HC n = 38). We investigated the differential trajectories in BD vs. HC in cortical gray matter volume and thickness, total cerebral white matter, hippocampal and amygdala volumes, estimated brain age, and cognitive functioning using linear mixed models. Within patients, we further investigated whether brain structural abnormalities detected at baseline were associated with subsequent mood episodes. Results: Compared to HC, patients showed a decline in total white matter volume over time and they had a larger amygdala volume, both at baseline and at follow-up time. Patients further showed lower cognitive performance at both times of investigation with no significant change over time. There were no differences between patients and HC in cortical gray matter volume or thickness, hippocampal volume, or brain-aging patterns. Conclusions: Cognitive impairment and amygdala enlargement may represent stable markers of BD early in the course of illness, whereas subtle white matter decline may result from illness progression.