Objectives: Musculoskeletal ultrasound (MSUS) is increasingly used by rheumatologists in daily clinical practice. However, MSUS is only valuable in trained hands, and assessment of trainee competences is therefore essential before independent practice. Thus, this study aimed to establish validity evidence for the EULAR and the Objective Structured Assessment of Ultrasound Skills (OSAUS) tools used for assessing MSUS competences. Methods: Thirty physicians with different levels of MSUS experience (novices, intermediates, and experienced) performed four MSUS examinations of different joint areas on the same rheumatoid arthritis patient. All examinations were video recorded (n=120), anonymized, and subsequently assessed in random order by two blinded raters using first the OSAUS assessment tool followed by the EULAR tool 1 month after. Results: The inter-rater reliability between the two raters was high for both the OSAUS and EULAR tools, with a Pearson correlation coefficient (PCC) of 0.807 and 0.848, respectively. Both tools demonstrated excellent inter-case reliability, with a Cronbach's alpha of 0.970 for OSAUS and 0.964 for EULAR. Furthermore, there was a strong linear correlation between the OSAUS and the EULAR performance scores and the participants' experience levels (R2=0.897 and R2=0.868, respectively) and a significant discrimination between different MSUS experience levels (P<0.001 for both). Conclusions: MSUS operator competences can be assessed reliably and valid using either the OSAUS or the EULAR assessment tool, thereby allowing a uniform competency-based MSUS education in the future. Although both tools demonstrated high inter-rater reliability, the EULAR tool was superior to OSAUS.