Background: Clinical lung ultrasound (LUS) is a fast bedside diagnostic tool which can assist clinicians in decisions regarding the treatment and monitoring of patients with respiratory symptoms. LUS training and education differ widely, and is often done in a clinical setting, with potential risks for patients if decisions are made based on the wrong interpretations. No clear guidelines or recommendations for objective and standardized assessment of LUS skills exist, and those that do are often based on a fixed time-frame or an arbitrary number of examinations performed; this does not ensure adequate competencies. Objectives: The study aimed to develop and gather validity evidence for a practical, simulation-based test in LUS. Methods: Nine cases were developed in collaboration with 3D Systems Healthcare, Littleton, CO, USA, representing the most common diagnosis and sonographic findings in patients with respiratory symptoms. Thirty-six participants with different levels of competence in LUS, completed the test. The participants were divided into groups, i.e., novices, intermediates, and experienced, according to their experience with LUS, the number of examinations they had performed, and any research they had conducted. Their answers were used for item analyses. Results: The intraclass correlation coefficient, Cronbachs' α, was 0.69 summarized, and there was a statistically significant difference (<italic>p</italic> < 0.001) between the novices and the trained participants (intermediates and experienced). A pass/fail score of 16 points was calculated according to the contrasting-groups method. Conclusion: We developed a test for the assessment of clinical competencies in LUS. The test proved solid validity evidence, and a pass/fail standard without any false-negatives, and only 2 explained false-positives.