Proper specimen collection is the most important step to ensure accurate testing for the coronavirus disease 2019 (COVID-19) and other infectious diseases. Assessment of healthcare workers’ upper respiratory tract specimen collection skills is needed to ensure samples of high-quality clinical specimens for COVID-19 testing. This study explored the validity evidence for a theoretical MCQ-test and checklists developed for nasopharyngeal (NPS) and oropharyngeal (OPS) specimen collection skills assessment. We found good inter-item reliability (Cronbach’s alpha = 0.76) for the items of the MCQ-test and high inter-rater reliability using the checklist for the assessment of OPS and NPS skills on 0.86 and 0.87, respectively. The MCQ scores were significantly different between experts (mean 98%) and novices (mean 66%), p < 0.001, and a pass/fail score of 91% was established. We found a significant discrimination between checklist scores of experts (mean 95% score for OPS and 89% for NPS) and novices (mean 50% score for OPS and 36% for NPS), p < 0.001, and a pass/fail score was established of 76% for OPS and 61% for NPS. Further, the results also demonstrated that a group of non-healthcare educated workers can perform upper respiratory tract specimen collection comparably to experts after a short and focused simulation-based training session. This study, therefore, provides validity evidence for the use of a theoretical and practical test for upper respiratory specimens’ collection skills that can be used for competency-based training of the workers in the COVID-19 test centers.