The presence and degree of category-selective responses in the human brain remains a central research question in visual neuroscience. Evidence for category-selectivity in higher-level vision primarily stems from neuro-imaging studies of healthy participants. Converging evidence also exists in patients after brain injury, however they often focus either on in-depth analysis of single case-studies or behavioural testing of one category, for example faces or words. Here we adopt a novel approach to studying higher-level vision after brain injury by exploring the largest sample of posterior cerebral artery stroke patients currently available (n = 64; 33 left hemisphere, 23 right hemisphere, 8 bilateral). Patients were tested using an in-depth behavioural battery encompassing both low-level visual tests (e.g., visual field, visual acuity, contrast sensitivity) and higher-level visual tests of word, object, and face processing. A data-driven approach (principal component analysis) was used to establish a pattern of co-occurrence within higher-level vision. The data revealed two principal components underlying patients’ performance. The first component included tests with a verbal (written word) input. The second component included tests with a non-verbal (picture) input, including face and object processing. This behavioural model was mapped onto the patients’ lesion profiles using voxel-based lesion symptom mapping. The two components had unique lesion correlates: The verbal input component with damage in the left inferior occipital and posterior temporal lobe, and the non-verbal input component with damage in the right occipital and medial temporal lobe. This approach to studying higher-level vision after brain injury using a data-driven approach suggests that patient’s behavioural performance did not reflect strict category-selective responses.