Background: Existing studies on intellectual consequences of alcohol-related disorders are primarily cross-sectional and compare intelligence test scores of individuals with and without alcohol-related disorders, hence mixing the influence of alcohol-related disorders and predisposing factors such as premorbid intelligence. In this large-scale study, the primary aim was to estimate associations of alcohol-related disorders with changes in intelligence test scores from early adulthood to late midlife. Methods: Data were drawn from a follow-up study on middle-aged men, which included a re-examination of the same intelligence test as completed in young adulthood at military conscription (total analytic sample = 2,499). Alcohol-related hospital diagnoses were obtained from national health registries, whereas treatment for alcohol problems was self-reported at follow-up. The analyses included adjustment for year of birth, retest interval, baseline intelligence quotient (IQ) score, education, smoking, alcohol consumption, and psychiatric and somatic comorbidity. Results: Individuals with alcohol-related hospital diagnoses (8%) had a significantly lower baseline IQ score (95.0 vs. 100.5, p < 0.001) and a larger decline in IQ scores from baseline to follow-up (−8.5 vs. −4.8, p < 0.001) than individuals without such diagnoses. The larger decline in IQ scores with alcohol-related hospital diagnoses remained statistically significant after adjustment for all the covariates. Similar results were revealed when IQ scores before and after self-reported treatment for alcohol problems (10%) were examined. Conclusions: Individuals with alcohol-related disorders have a lower intelligence test score both in young adulthood and in late midlife, and these disorders, moreover, seem to be associated with more age-related decline in intelligence test scores. Thus, low mean intellectual ability observed in individuals with alcohol-related disorders is probably a result of both lower premorbid intelligence and more intellectual decline.