Rating the Rater: A Technique for Minimizing Leniency Bias in Residency Applications

Background:

Each program in the highly competitive match for a surgical residency needs a way to review applicants effectively. Often this task is undertaken by individual faculty members, reviewing an applicant’s file and assigning a score. Despite being asked to rate on a standardized scale, our program found that ratings of the same applicants varied dramatically, with certain faculty consistently scoring higher or lower than others. This is termed leniency bias, or the Hawk-Dove effect, and can affect who is invited to interview depending on which faculty are assigned to review an applicant’s file.

Methods:

A technique to minimize leniency bias was developed and applied to this year’s 222 applicants for our plastic surgery residency. The effect of the technique was evaluated by comparing variance between ratings of the same applicants by different faculty before and after our technique.

Results:

The median variance of ratings of the same applicants reduced from 0.68 before correction to 0.18 after correction, demonstrating better agreement between raters of the applicants’ scores after our technique had been applied. This year, applying our technique affected whether or not 16 applicants (36% of interviewees) were invited for interview, including one applicant who matched to our program but who otherwise would not have been offered an interview.

Conclusions:

We present a simple but effective technique to minimize the leniency bias between raters of residency applicants. Our experience with this technique is presented together with instructions and Excel formulae for other programs to use.