Many sleep less than recommended without experiencing daytime sleepiness. According to prevailing views, short sleep increases risk of lower brain health and cognitive function. Chronic mild sleep deprivation could cause undetected sleep debt, negatively affecting cognitive function and brain health. However, it is possible that some have less sleep need and are more resistant to negative effects of sleep loss. We investigated this using a cross-sectional and longitudinal sample of 47,029 participants of both sexes (20-89 years) from the Lifebrain consortium, Human Connectome project and UK Biobank, with measures of self-reported sleep, including 51,295 MRIs of the brain and cognitive tests. 740 participants who reported to sleep < 6 hours did not experience daytime sleepiness or sleep problems/disturbances interfering with falling or staying asleep. These short sleepers showed significantly larger regional brain volumes than both short sleepers with daytime sleepiness and sleep problems (n = 1742) and participants sleeping the recommended 7-8 hours (n = 3886). However, both groups of short sleepers showed slightly lower general cognitive function, 0.16 and 0.19 standard deviations, respectively. Analyses using acelerometer-estimated sleep duration confirmed the findings, and the associations remained after controlling for body mass index, depression symptoms, income and education. The results suggest that some people can cope with less sleep without obvious negative associations with brain morphometry, and that sleepiness and sleep problems may be more related to brain structural differences than duration. However, the slightly lower performance on tests of general cognitive abilities warrants closer examination in natural settings. Significance statementShort habitual sleep is prevalent, with unknown consequences for brain health and cognitive performance. Here we show that daytime sleepiness and sleep problems are more strongly related to regional brain volumes than sleep duration. However, participants sleeping ≤ 6 hours had slightly lower scores on tests of general cognitive function. This indicates that sleep need is individual, and that sleep duration per se is very weakly if at all related brain health, while daytime sleepiness and sleep problems may show somewhat stronger associations. The association between habitual short sleep and lower scores on tests of general cognitive abilities must be further scrutinized in natural settings.