Background: International guidelines suggest that trainees should perform at least 100 flexible bronchoscopies in a supervised setting, but this number is not evidence based. An objective assessment method could provide educational feedback to trainees and help supervisors decide when basic competency is established. No former assessment instrument has been able to distinguish between trainees and experts. Objectives: The aim of this study was to explore the validity and reliability of a new assessment procedure relating to testing operators across multiple tasks with increasing difficulty using a standardized scoring form. Methods: The test was administered on a virtual reality bronchoscopy simulator to a total of 42 test subjects (14 senior consultants, 14 trainees and 14 medical students). The inter-rater reliability of the test procedure was explored according to examination of test results from 10 subjects using 3 raters (2 blinded). Results: We found a high inter-rater reliability (Cronbach's a = 0.90). The assessment procedure sufficiently differentiated the performance of the 3 groups (p <0.001). Assessment of 6 procedures was necessary to secure a generalizability coefficient >0.80. Conclusions: The new assessment procedure proved both valid and reliable.