To test whether genetic variants in PICALM, BIN1, CD2AP, and RIN3—suggested to be involved in blood–brain barrier amyloid-β transcytosis pathways—associate with Alzheimer’s disease, all dementia, suggested vascular dementia, and stroke, and whether such associations are independent of the strong ε4 APOE risk allele. In a prospective cohort study of 74,754 individuals from the general population we genotyped PICALM (rs10792832), BIN1 (rs6733839), CD2AP (rs10948363), and RIN3 (rs10498633), and generated a weighted and a simple allele score. Multifactorially adjusted hazard ratios for the fourth quartile versus the first quartile of the weighted allele score were 1.42 (95% confidence interval 1.22–1.64) for Alzheimer’s disease, and 1.33 (1.19–1.48) for all dementia. For suggested vascular dementia and stroke the corresponding estimates were 1.71 (1.18–2.49) and 1.12 (1.04–1.22), respectively. Hazard ratios were similar after APOE adjustment. Genetic variants in PICALM, BIN1, CD2AP, and RIN3 are associated with increased risk of Alzheimer’s disease, all dementia, and suggested vascular dementia independent of the strong APOE ε4 allele. These findings may suggest that clathrin-mediated endocytosis in clearance of amyloid-β across the blood–brain barrier is important for the integrity of both brain tissue and cerebral vessels.