The face inversion effect (FIE) refers to the observation that presenting stimuli upside-down impairs the processing of faces disproportionally more than other mono-oriented objects. This has been taken as evidence that processing of faces and objects differ qualitatively. However, nearly all FIE studies are based on comparing individuation of upright faces, which most people are rather good at, with individuation of objects most people are much less familiar with individuating (e.g., radios and airplanes). Consequently, the FIE may mainly reflect differences between categories in how they are processed prior to inversion, with within-category discrimination of upright faces being a much more familiar task than within-category discrimination among members belonging to other object classes. We tested this hypothesis by comparing inversion effects for faces and objects using object recognition tasks that do not require within-category discrimination (object decision and old/new recognition memory tasks). In all tasks (seven with objects and two with faces) we find credible inversion effects, but in no instance were these effects significantly larger for faces than for objects. This suggests that the FIE can be a product of familiarity with the type of identification process required in the upright conditions rather than some process that is selectively affected for faces when stimuli are inverted.