Face recognition skills vary considerably both in the normal population and in various clinical groups, and understanding the cognitive mechanisms contributing to this variability is important. We investigated whether: (i) a group of good face recognizers (high performers; HPs) were better than control subjects in face and object recognition, (ii) if any dissociations between face, object, and word processing could be demonstrated in HPs, and (iii) compared the performance of the HPs to a group of poor face recognizers (developmental prosopagnosics; DPs). We found that HPs were significantly better than matched control subjects on tests of both face and object recognition including a reading task, but they did not show a disproportionally larger inversion effects on typical tests of face processing (the CFMT and the CFPT). There was also no evidence of dissociations between face and object processing in the HPs when compared to controls, indicating superior performance across visual domains. In the DP group, however, we found significant dissociations between face and object recognition performance on a group level, indicating that face processing is disproportionally affected. On this basis, we propose that superior face processing in HPs rely on more general cognitive or perceptual processes shared with object processing. Hence, while face processing in DPs seems qualitatively different from the normal population, there is no such difference between average and high performing face recognizers. What underlies superior face processing in HPs might be conceived as a general factor in the visual domain, a VG-factor, akin to the G factor in intelligence.