Risk of colorectal cancer (CRC) increases in relatives of patients with CRC. The extent to which this is attributable to genetic predisposition or shared environment is unclear. We explored this question using nationwide cohorts from Denmark, Finland and Sweden. From 1977 to 2013, we identified 359 879 individuals with a CRC diagnosis and 2 258 870 of their relatives who we followed for CRC incidence. We calculated standardized incidence ratios (SIR) and 95% confidence intervals (CI) for CRC in individuals with an affected relative. We used nationwide household and pedigree data along with national SIR estimates to calculate risk ratios (RR) for the contribution of shared household environment, childhood environment and genetic relationship to CRC risk in those with an affected relative. SIR of CRC was increased for individuals with an affected relative, across all countries and ages. For those with an affected parent, the SIR was 1.65 (95% CI: 1.61-1.69), 1.98 (95% CI: 1.87-2.09), for those with an affected sibling and 2.14 (95% CI: 1.84-2.49) for those with an affected halfsibling. In those <65 years old, shared childhood (RR: 1.41, 95% CI: 1.26-1.57) and household (RR: 2.08, 95% CI: 1.25-3.46) environments were significantly greater contributors to familial risk of CRC than genetics (RR: 0.88, 95% CI: 0.53-1.46). This large-scale Nordic population-based study of excess risk of CRC among relatives of those with CRC addresses the difficult disentangling of shared environment from genetic predisposition in the heritability of CRC. We found shared environment to be the most important contributor to CRC risk.