Effective assessors are the critical component in workplace-based assessment.
Selection of assessors
The question ‘who can assess?’ is addressed very specifically in the AMC workplace-based assessment accreditation guidelines for IMGs. Effective assessors are essential to any assessment program.
Workplace-based assessment is most appropriately conducted by clinicians who are:
- Experienced, and possess expertise in the clinical tasks assessed;
- Trained in workplace-based assessment for IMGs;
- Available, and have agreed to assess according to the principles of workplace-based assessment.
Typically, assessors are drawn from the clinician pool within the workplace where the assessment is conducted. Alternatively, the assessment may be conducted—or supplemented—by an appropriately experienced and trained team of externally-based clinicians.
Training and calibration of assessors
The expectation is that assessors will have adequate support in fulfilling their assessment responsibilities. Training is important to promote understanding of and confidence in assuming the assessor role.
The aim of a training and calibration program is to ensure that assessors are:
(a) oriented to the assessment methods to be used, to what each method assesses, and how it is applied
(b) made aware of their own performance as an assessor.
A full training program plan for a mini-CEX training session is included in Appendix 2. This includes nine taped patient encounters with different candidates and three feedback sessions that can be used as a basis of training for the mini-CEX.
The basic elements of any training program are shown below.
Part 1. Introduction
Explain the purpose and format of the workshop and present an overview of the assessment method being discussed (its history, and research evidence for its effectiveness).
Part 2. What is being assessed?
Before the participants view the details of the assessment methodology proposed, show examples of the types of performances you are assessing and ask the participants to identify key aspects of this performance – to identify what was good and what was not good, and what criteria could be used to judge this performance.
Part 3: The assessment method
Provide the participants with a sample of the assessment method used and any rating forms involved. Ensure the participants understand the dimensions being assessed and the scale used.
Part 4: Frame of reference training
This type of training can improve the accuracy of ratings. It works by providing the assessor with a context or ‘frame’ for their ratings. Participants view a sample of performances and are asked to rate these using the rating scale provided. After each ‘frame’, participants discuss their scores and the reasons for any variance.
Part 5: Giving effective feedback
Discuss the steps in giving effective feedback. Deal particularly with situations where the IMG has displayed poor performance, as many clinicians find this the most difficult feedback to give. Appendix 2 has three feedback video sessions recorded after mini-CEX patient encounters.
Dealing with conflict of interest as an assessor
The possibility of biased judgments arising through a conflict of interest may be present in the assessment process. Ideally, assessors should be ‘at arms length’ to the IMGs they are assessing, to ensure their assessments are not influenced by factors such as employment issues or collegial relationships. Assessors who are not at arms length from those whom they are assessing face extreme pressure in situations where they observe—and are obliged to document—unsatisfactory performance. It is extremely unlikely, however, that all assessors can be totally free of conflicts of interest.
Measures that may be taken to avoid or reduce the risk of bias in judgments include:
- The use of objective benchmarks in the assessment process;
- The use of multiple assessors, some of whom have no employment or supervisory relationship with the IMG;
- Training sessions in which assessors are advised on ways of distancing themselves from subjective considerations and dealing with conflict of interest.
It is possible that situations will arise in which a conflict of interest is declared and the assessor(s) absent themselves from the assessment of a particular IMG.