Production of testosterone is under tight control by human chorion gonadotropin (hCG) during fetal life and luteinizing hormone (LH) in adulthood. Several animal and human studies have linked vitamin D status with sex steroid production although it is not clear whether there exist a direct or indirect involvement in androgen production. Few studies have investigated this crosslink in young healthy men and putative direct or synergistic effect of activated vitamin D (1,25(OH)2D3) and LH/hCG on sex steroid production in vitro. Here, we present cross-sectional data from 300 young men and 41 hCG-stimulated men with impaired Leydig cell function combined with data from an ex vivo culture of human testicular tissue exposed to 1,25(OH)2D3 alone or in combination with hCG. Serum 25-OHD was positively associated with SHBG (β:0.002; p = 0.023) and testosterone/estradiol-ratio (β:0.001; p = 0.039), and inversely associated with free testosterone (%) (free testosterone/total testosterone) (β:-0.002; p = 0.016) in young men. Vitamin D deficient men had higher total and free estradiol concentrations than men with higher vitamin D status (19% and 18%, respectively; p < 0.01). Interestingly, men with impaired Leydig cell function and vitamin D deficiency had a significantly lower hCG-mediated increase in total and free testosterone compared with vitamin D sufficient men (p < 0.05). Accordingly, testicular tissue exposed to 100 nM 1,25(OH)2D3 had a 15% higher testosterone release into the media compared with vehicle treated specimens (p = 0.030). In conclusion, vitamin D deficiency is associated with lower testosterone/estradiol ratio in young men and lower Leydig cell sensitivity after hCG-stimulation in men with impaired gonadal function. The significant effect of 1,25(OH)2D3 on testosterone production in a human testis model supports that the stimulatory effect at least in part may be direct. Larger placebo-controlled studies are needed to determine whether vitamin D supplementation can influence testosterone production.