Overview
This lesson summarises key messages from the preceding lesson. The factors cited in this lesson are essential to effective systems of workplace-based assessment and can serve as a checklist to guide the development and evaluation of your system of assessing IMGs.
- Visible commitment from leadership for workplace-based assessment
Developing and implementing a system of workplace-based assessment that will provide defensible decisions about the performance of IMGs is a significant undertaking. Creating such systems entails much more than simply ‘beefing up’ existing systems of in-training assessment every few weeks. Given the demands of effective systems of workplace-based assessment, they must exist in a context in which clinical, administrative, and other leaders in your setting convey public support for their importance, and provide staff involved with substantive support.
- An administrative infrastructure that supports assessors and IMGs in the workplace-based assessment process
A functional workplace-based assessment system must offer dedicated staff support time related to tasks such as the scheduling of assessments, the selection of patients, and the provision and collection of assessment forms and consent letters. Infrastructure support should also include a system and personnel for filing assessment forms, recording assessment data onto electronic spread sheets, and analysing assessment data.
- Effective assessors
The most important factor in workplace-based assessment is effective assessors. ‘Effectiveness’ entails a commitment to the task of assessing; clinical expertise related to the patients involved in the assessments; assessment expertise related to providing discriminating, consistent assessments that are indexed to the standards of performance expected at the end of PGY1 level of training; and an ability to provide effective feedback. Commitment should be premised on a system which provides for assessors time and support (for example, administrative, financial, educational), rather than adding workplace-based assessment to an already full schedule. Expertise in workplace-based assessment should be established through orientation and training programs for assessors.
- Orientation and commitment of IMGs to the assessment process and their role in it.
IMGs must understand the workplace-based assessment process, and that the process is used to judge and report their performance. With their understanding of the importance of these assessments to their future, IMGs are likely to be highly focused and ensure that they receive the prescribed number and types of assessments. IMGs must also understand their responsibilities in implementing the workplace-based assessment system that you use. You may, for example, assign IMGs the responsibility for scheduling a specific number of assessments in a term.
- A plan (blueprint) defining ‘what’ you wish to assess
An important challenge facing the workplace-based assessment system is to obtain a sample of an IMG’s performance which permits you to generalise about the overall level of performance. It is important that this sample be elicited over a wide range of patients (for example, age, complaints, gender, culture, acute and chronic illness) and clinical dimensions/areas and clinical settings (such as in-patient and ambulatory settings), and employ sufficient numbers of observations and assessors. As outlined in the AMC workplace-based assessment accreditation guidelines, representative sampling can be planned by developing a blueprint identifying the spread of observations across disciplines and across clinical areas.
- A plan which defines the selected assessment strategies
A workplace-based assessment strategy should include a plan for multiple snapshots over time, using multiple methods and multiple assessors. The assessment strategies should be selected on the basis of what you want to assess in your clinical context, and what each method assesses well. Using multiple assessors will take into account that some assessors are ‘hawks’ while others are ‘doves’, and that different assessors focus on different aspects of clinical performance. It is also important to ensure that assessors do not have a conflict of interest in their roles as assessors, through either their personal or work relationships with those whom they are assessing.
- A quality control system for monitoring the assessment process and the measurement qualities of the assessment data.
Steps should be taken to ensure that the workplace-based assessment system is providing reliable assessment data on the IMGs and serving as a basis for valid inferences about their performance. Evidence of validity and reliability can be obtained by eliciting data from assessors and IMGs on their perceptions of factors that may affect these attributes, and by calculating reliability estimates for your assessment data. Help in calculating reliability estimates can be obtained from the educational experts.
- A plan to identify and address practical considerations which may affect assessments
A key factor which may invalidate assessments of IMGs is the conflict of interest that assessors may have. Ideally, an assessor should be ‘at arms length’, not having a friendship, collegial or employment relationship with the individual being assessed. This is a difficult condition to meet in many settings. One of the principles underpinning workplace-based assessment is to use multiple assessors; and it is important to ensure that at least some of these should have no conflict of interest in their relationship with the IMG.
A second factor that may affect assessments is the restricted nature of the patient population in some settings. If, for example, the setting does not provide encounters with women’s health, paediatric or psychiatric patients, your assessment system will need to identify alternative clinical venues where such patients exist and where IMG assessment can occur.
- Mechanisms to evaluate and support feasibility
Effective systems of workplace-based assessment for IMGs require a significant commitment of resources. A key resource is the availability and time of clinician assessors. In planning systems of workplace-based assessment, projections should be made regarding the number of clinicians available and the time required of clinician assessors. These projections should be translated into costs, and strategies for addressing these costs should be defined.
Another key resource for workplace-based assessment is access to a suitable spectrum of patients. As discussed earlier, meeting the need for such patients may require elements of workplace-based assessment taking place outside the IMG’s normal work setting.