Increasing evidence from animal experiments suggests that physical activity (PA) promotes neuroplasticity and learning. For humans, most research on the relationship between PA, sedentary behaviour (SB), and cognitive function has relied on self-reported measures of behaviour. Office work is characterised by high durations of SB combined with high work demands. While previous studies have shown that fitter office workers outperform their less fit colleagues in cognitive tests, the importance of PA and SB remains unknown. This study investigated associations between objectively measured PA and SB, using hip-worn accelerometers, and cognitive functions in 334 office workers. Time spent in moderate-to-vigorous PA (MVPA) was not associated with any cognitive outcome. However, time spent in SB tended to be positively associated with words recalled in free recall (β = 0.125). For the least fit participants, the average length of MVPA bouts was favourably related to Stroop performance (β = −0.211), while for the fitter individuals, a longer average length of MVPA bouts was related to worse recognition (β = −0.216). While our findings indicate that the length of MVPA bouts was associated with better Stroop performance in the least fit participants, our findings do not support the notion that more time spent in MVPA or less time in SB is associated with better cognitive function.