Objective We demonstrate and evaluate the first markerless motion tracker compatible with PET, MRI, and simultaneous PET/MRI systems for motion correction (MC) of brain imaging. Methods PET and MRI compatibility is achieved by careful positioning of in-bore vision extenders and by placing all electronic components out-of-bore. The motion tracker is demonstrated in a clinical setup during a pediatric PET/MRI study including 94 pediatric patient scans. PET MC is presented for two of these scans using a customized version of the Multiple Acquisition Frame method. Prospective MC of MRI acquisition of two healthy subjects is demonstrated using a motion-aware MRI sequence. Real-time motion estimates are accompanied with a tracking validity parameter to improve tracking reliability. Results For both modalities, MC shows that motion induced artifacts are noticeably reduced and that motion estimates are sufficiently accurate to capture motion ranging from small respiratory motion to large intentional motion. In the PET/MRI study, a time-activity curve analysis shows image improvements for a patient performing head movements corresponding to a tumor motion of ±5-10 mm with a 19% maximal difference in standardized uptake value before and after MC. Conclusion The first markerless motion tracker is successfully demonstrated for prospective MC in MRI and MC in PET with good tracking validity. Significance As simultaneous PET/MRI systems have become available for clinical use, an increasing demand for accurate motion tracking and MC in PET/MRI scans has emerged. The presented markerless motion tracker facilitate this demand.