Distressing events during pregnancy that engage activity of the body's endocrine stress response have been linked with later life cognitive deficits in offspring and associated with developmental changes in cognitive-controlling neural regions. Interestingly, prenatal stress (PS)-induced alterations have shown some sex specificity. Here, we review the literature of animal studies examining sex-specific effect of physical PS on the function and structure of the hippocampus as hippocampal impairments likely underlie PS-associated deficits in learning and memory. Furthermore, the connectivity between the hypothalamic–pituitary–adrenal (HPA) axis and the hippocampus as well as the heavy presence of glucocorticoid receptors (GRs) in the hippocampus suggests this structure plays an important role in modulation of activity within stress circuitry in a sex-specific pattern. We hope that better understanding of sex-specific, PS-related hippocampal impairment will assist in uncovering the molecular mechanisms behind sex-based risk factors in PS populations across development, and perhaps contribute to greater precision in management of cognitive disturbances in this vulnerable population.