We summarize the available in vivo validation of cerebral near-infrared spectroscopy (NIRS) oximetry to inform future in vivo validation strategies. In particular, to establish a way forward in the assessment of NIRS instrumentation for future randomized trials, a systematic literature search is performed. The records are screened and abstracts are assessed to select studies fulfilling our inclusion criteria. Twenty-two pediatric and 28 adult studies are analyzed after exclusion of three articles in each group. All studies compare regional cerebral tissue oxygenation measured by cerebral NIRS to invasive measurement of central or jugular venous oxygen saturation. In studies without Bland-Altman plots, we extracted data from scatter plots enabling estimation of mean difference (MD), standard deviation (SD), and limits of agreement (LOA). To assess the agreement between rStO 2 (regional cerebral tissue oxygenation) estimated by NIRS and by blood samples, weighted averages of the MDs and SDs from each study are calculated. We found a fair agreement between the overall mean of cerebral tissue oxygenation and the mean of a reference value measured by co-oximetry whatever NIRS instrument or site of blood sampling used. Cerebral oxygenation overestimates the reference at low values, some instruments apparently more than others. Thus, a high degree of scatter and a lack of a good reference method complicate in vivo validation of NIRS. It is difficult to draw any firm conclusions despite the large number of studies, and the result of this review leaves us questioning if more of such validation studies of cerebral NIRS oximetry are really needed. Furthermore, the combination of lack of validation and poor repeatability is an important issue when designing a randomized clinical trial of implementing cerebral NIRS oximetry into clinical care.