It is crucial to understand the natural history of genital human papillomavirus (HPV) infection in men to prevent the increasing male HPV-related disease burden. We evaluated the associations between HPV infection and circumcision, smoking, and alcohol use after accounting for sexual behavior. The study included 2331 male personnel from Danish barracks. Penile swabs were tested for HPV DNA with a polymerase chain reaction assay, INNO-LiPA. All men completed a self-administered questionnaire providing data on potential risk factors for HPV such as lifestyle and sexual habits. Using multivariable logistic regression, associations between potential risk factors and HPV infection were estimated and expressed as odds ratios (ORs) with 95% confidence intervals (CI). Current cigarette smoking was associated with increased odds of HPV detection (OR = 1.2; 95% CI: 1.0–1.4), but we found no association with alcohol use in the analysis adjusted for sexual behavior. Circumcision reduced the odds of a prevalent HPV infection (OR = 0.7; 95% CI: 0.5–1.0) although not statistically significantly. Strong associations with lifetime and recent number of female sex partners were observed, but in contrast to uncircumcised men, increasing number of sex partners was not associated with higher HPV prevalence in circumcised men. In conclusion, smoking was associated with increased odds of penile HPV in men from the general population in Denmark, whereas circumcision seemed to reduce the risk. Moreover, our results indicated that there might be differences in the viral susceptibility between circumcised and uncircumcised men.