Aim: To explore pituitary–gonadal hormone concentrations and assess their association with inflammation, severe respiratory failure, and mortality in hospitalized men and women with COVID-19, and compare these to hormone concentrations in hospitalized patients with bacterial community-acquired pneumonia (CAP) and influenza virus CAP and to concentrations in a reference group of healthy individuals. Methods: Serum concentrations of testosterone, estrone sulfate, luteinizing hormone (LH), follicle-stimulating hormone (FSH), and interleukin-6 (IL-6) were measured within 4 days of admission. Associations were assessed by logistic regression analysis in patients with COVID-19, and results were reported as odds ratio with 95% CI per two-fold reduction after adjustment for age, comorbidities, days to sample collection, and IL-6 concentrations. Results: In total, 278 patients with COVID-19, 21 with influenza virus CAP, and 76 with bacterial CAP were included. Testosterone concentrations were suppressed in men hospitalized with COVID-19, bacterial and influenza virus CAP, and moderately suppressed in women. Reductions in testosterone (OR: 3.43 (1.14–10.30), P = 0.028) and LH (OR: 2.51 (1.28–4.92), P = 0.008) were associated with higher odds of mehanical ventilation (MV) in men with COVID-19. In women with COVID-19, reductions in LH (OR: 3.34 (1.02–10-90), P = 0.046) and FSH (OR: 2.52 (1.01–6.27), P = 0.047) were associated with higher odds of MV. Conclusion: Low testosterone and LH concentrations were predictive of severe respiratory failure in men with COVID-19, whereas low concentrations of LH and FSH were predictive of severe respiratory failure in women with COVID-19.