TY - JOUR

T1 - Feasibility, quality and validity of narrative multisource feedback in postgraduate training

T2 - a mixed-method study

AU - Holm, Ellen Astrid

AU - Al-Bayati, Shaymaa Jaafar Lafta

AU - Barfod, Toke Seierøe

AU - Lembeck, Maurice A

AU - Pedersen, Hanne

AU - Ramberg, Emilie

AU - Klemmensen, Åse Kathrine

AU - Sorensen, Jette Led

N1 - © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

PY - 2021

Y1 - 2021

N2 - OBJECTIVES: To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.DESIGN: Qualitative text analysis supplemented by quantitative descriptive analysis.SETTING: Internal Medicine Departments in Zealand, Denmark.PARTICIPANTS: 48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).INTERVENTION: This study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either 'no reasons for concern' or 'possibly some concern', thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.RESULTS: Out of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7-1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.CONCLUSIONS: A purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees' performances based on the MSF reports.

AB - OBJECTIVES: To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.DESIGN: Qualitative text analysis supplemented by quantitative descriptive analysis.SETTING: Internal Medicine Departments in Zealand, Denmark.PARTICIPANTS: 48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).INTERVENTION: This study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either 'no reasons for concern' or 'possibly some concern', thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.RESULTS: Out of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7-1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.CONCLUSIONS: A purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees' performances based on the MSF reports.

KW - Clinical Competence

KW - Education, Medical, Graduate

KW - Educational Measurement

KW - Feasibility Studies

KW - Feedback

KW - Humans

U2 - 10.1136/bmjopen-2020-047019

DO - 10.1136/bmjopen-2020-047019

M3 - Journal article

C2 - 34321296

VL - 11

JO - BMJ Open

JF - BMJ Open

SN - 2044-6055

IS - 7

M1 - e047019

ER -