Knowledge about the consequences of stroke on high-level vision comes primarily from single case studies of patients selected based on their behavioural profiles, typically patients with specific stroke syndromes like pure alexia or prosopagnosia. There are, however, no systematic, detailed, large-scale evaluations of the more typical clinical behavioural and lesion profiles of impairments in high-level vision after posterior cerebral artery stroke. We present behavioural and lesion data from the Back of the Brain project, to date the largest (N = 64) and most detailed examination of patients with cortical posterior cerebral artery strokes selected based on lesion location. The aim of the current study was to relate behavioural performance with faces, objects and written words to lesion data through two complementary analyses: (i) a multivariate multiple regression analysis to establish the relationships between lesion volume, lesion laterality and the presence of a bilateral lesion with performance and (ii) a voxel-based correlational methodology analysis to establish whether there are distinct or separate regions within the posterior cerebral artery territory that underpin the visual processing of words, faces and objects. Behaviourally, most patients showed more general deficits in high-level vision (n = 22) or no deficits at all (n = 21). Category-selective deficits were rare (n = 6) and were only found for words. Overall, total lesion volume was most strongly related to performance across all three domains. While behavioural impairments in all domains were observed following unilateral left and right as well as bilateral lesions, the regions most strongly related to performance mainly confirmed the pattern reported in more selective cases. For words, these included a left hemisphere cluster extending from the occipital pole along the fusiform and lingual gyri; for objects, bilateral clusters which overlapped with the word cluster in the left occipital lobe. Face performance mainly correlated with a right hemisphere cluster within the white matter, partly overlapping with the object cluster. While the findings provide partial support for the relative laterality of posterior brain regions supporting reading and face processing, the results also suggest that both hemispheres are involved in the visual processing of faces, words and objects.