Overview
A key resource is the availability and time of clinician assessors.
Reliability and validity are two important concepts that characterise all measurements. To have confidence in our measurements of the IMG’s performance in the workplace, it is important the methods are deemed to be both reliable and valid. While workplace-based assessment has an important summative function in forming a judgment about an IMG’s capacity for safe and independent practice in Australian clinical settings, formative assessment is also a component of workplace-based assessment. These terms are explained below.
Assessment terms
Formative assessment
Assessment that is primarily aimed at helping IMGs to identify the effective and ineffective aspects of their clinical performance is called formative assessment. The likelihood that IMGs are able to improve performance as a result of formative assessment is increased markedly if the assessor:
- identifies, via dialogue with the IMG, precisely the aspects of the task that have been performed well
- identifies the areas that have been overlooked or need improvement
- develops a plan for improving in the specified area.
More details on feedback and how it can be effectively delivered are amplified in Lesson 7.
Summative assessment
An assessment that aims to pass judgment on an IMG’s clinical performance is called summative assessment. To orient IMGs to the system of workplace-based assessment that they will encounter, it is a good strategy to allow a couple of formative ‘practice’ assessments before the first summative assessment. Once the summative assessments are underway, a systematic recording of outcomes is essential, including the assessor’s global judgment of the IMG’s performance, and notes indicating the strengths and weaknesses observed and plans developed with the IMG for improvements. Finally, a well-constructed and managed spreadsheet or database is essential so that the assessing authority can keep careful track of the IMG’s progress.
Reliability
Reliability is a measure of the accuracy of the score derived from an assessment. First and foremost it is a function of the size of the sample of performance used as a basis for the assessment. It is known that performance in medicine differs widely from case to case, which means that performance must be sampled widely over cases to enable an accurate estimation of a clinician’s overall level performance. This issue is discussed further in this course in Lesson 4, in relation to specific assessment methods.
Secondly, score accuracy is a function of the consistency of measurement over different events and by different assessors. Consistency across assessors can be aided by reference to explicit, observable performance criteria against which a performance is judged. Consistency can also be aided by training assessors in the use of assessment methods.
Validity
In the context of assessing IMGs’ performance in the workplace, the question posed by validity is ‘Does the assessment measure what it was intended to measure and do the assessment data provide a basis for making a decision about an IMG’s overall level of performance?’ A precondition for validity is reliability. Reliability is largely a function of the adequacy of sampling of cases and different assessors, and is therefore influenced by the total assessment time. Validity, on the other hand, is largely a function of whether the sampling was representative across the types of patients, and types of skills for which the IMG is expected to exhibit competent performance. Assessment data can therefore be a reliable measure but may not be a valid measure if representative sampling is not achieved. Lesson 5 on blueprinting addresses the issue of content validity in greater detail.
Purposes of workplace-based assessment
Workplace-based assessment is used to:
- monitor IMGs’ progress through the required, supervised pathway, and signify satisfactory completion of the program
- provide guidance for specific learning needs of IMGs
- determine whether and when IMGs are ready to proceed to independent (or more independent) practice in Australia.
Making judgments about performance
Assessors and candidates need to be committed to the task.
In workplace-based assessment, judgments about an IMG’s level of performance are made by assessors. These judgments are recorded on rating scales, where each scale position is associated with a judgmental term such as unsatisfactory, or meets expectation, or exceeds expectations. It is essential that IMG assessors reference their judgments to the expected level of performance which is that expected at the end of the first postgraduate year (PGY1) for local graduates.
Workplace-based assessment involves the measurement of abilities and attitudes within highly complex areas of activity. To be effective, performance in a complex area of activity requires focused observation of IMGs in clinical practice. Assessors should be drawn from a team of clinicians who are experienced in relevant clinical practice and trained in the assessment of IMGs and other medical trainees and practitioners.
Principles underpinning workplace-based assessment
1. Use a variety of methods
Clinical performance is multifaceted and no single method can assess all of the elements that contribute to good medical practice. In addition, assessment methods vary in their effectiveness to assess across different elements of performance. It is essential therefore to use a set of assessment methods that complement each other in terms of what they assess well.
2. Use multiple observations
All clinicians perform in a content specific way; that is, they perform well in a clinical situation which they know well or have experience in, but may not perform well on other cases. To ensure that an assessment decision is reliable and justifiable, the assessment must be planned to sample broadly across clinical areas. It is thus recommended that multiple observations occur in various clinical settings over a period of time.
3. Use multiple assessors
Assessors vary in their tendency to mark harshly or leniently and each assessor brings different perspectives to the assessment task. All assessors should undertake appropriate training to try to reduce this variability. In addition, some assessors may not be ‘at arms length’ to the IMG they are assessing, and conflicts of interest may bias their assessments. The best way to moderate these influences is to ensure that the assessment of each candidate is based on the opinions of multiple assessors.
Summary
The selection of assessment methods is one of the most critical of all influences on workplace-based assessment.
The selection of assessment methods is one of the most critical of all influences on the success of workplace-based assessment. The medical education literature shows that the number and specific nature of tasks and the total time involved are key factors underpinning reliability and validity. Choice and design of assessment methods, clearly described criteria, and assessor training are essential elements of a successful workplace-based assessment program.
The following sections of this course outline the steps to consider in planning an assessment program:
Section 3: Decide on what is being tested
Section 4: Decide on assessment methods
Section 5: Blueprinting
Section 6: Standard setting – how does a candidate pass?
Section 7: Feedback to candidates
Section 8: Training and calibration of assessors
Section 9: Development of resources and systems
Section 10: Essentials for effective implementation.