A blueprint depicts the relationship between what has to be assessed and how it is to be assessed.
Assessment involves a sampling of a candidate’s performance. It is impossible to test for all clinical dimensions for all medical conditions. The sample of performance that is tested has to be representative of the spectrum of that relevant to safe medical practice. A blueprint defines the aspects of an IMG’s performance that will be assessed, how each aspect will be assessed, and the number of performances that are to be assessed. It ensures that there is an appropriate balance between what is assessed and how it is assessed. Blueprints are constructed to ensure that assessments are balanced and valid, whether they be for summative or formative purposes.
In its simplest form, a blueprint is a matrix where the rows relate to what is being assessed and the columns relate to how to assess. The cells within the matrix are completed as decisions are made about which method best assesses which attribute.
Steps to develop a blueprint
- Levels of assessment: Identify the levels for which the blueprint is being constructed.
Blueprinting can be undertaken at different levels. For an undergraduate medical program, a blueprint might be constructed at the macro level, or whole program, showing where various methods are used to assess objectives or outcomes across all years of the program. There may be an additional blueprint to show how an individual year assessment is planned. There may then be a more detailed, micro-blueprint at the level of a semester-long course showing the relationship between learning objectives/semester content and written and clinical examination methods.
For workplace-based assessment, blueprints will also need to be constructed at different levels. At the macro level, a blueprint should be developed to show how the methods selected assess the defined clinical dimensions of performance; this will be the overall assessment plan. A micro level blueprint will also be needed to plan the implementation of each assessment method at the clinical level. For example, a blueprint for mini-CEX observations should be developed to define how to distribute observed encounters across clinical disciplines, clinical tasks and possibly clinical sites. Examples of blueprints are shown later in this lesson.
- What to assess: Decide on what is being assessed.
For workplace-based assessment in the IMG context, the dimensions of clinical performance that are to be assessed have been defined in the AMC workplace-based assessment accreditation guidelines for IMGs.
The clinical dimensions are:
- Clinical skills (history taking, physical examination, investigation and diagnosis, prescribing and management, counselling/patient education and clinical procedures).
- Clinical judgment.
- Communication skills.
- Ability to apply aspects of public health relevant to clinical settings.
- Cultural competence.
- Ability to work as an effective member of the health care team.
- Professionalism and attention to patient safety.
- Suitability: Decide on what assessment methods are suitable.
Lesson 4 includes a review of the methods available for workplace-based assessment. Consider which methods will appropriately assess the various dimensions of performance. Figure 2 shows one possible sample of observed encounters, using the mini-CEX method, across the clinical dimensions and the clinical areas. Figure 3 shows an example of a micro level blueprint showing the distribution of mini-CEXs across different aspects of the clinical dimensions and across clinical areas. Other plans are possible.
|Figure 2 – An example of a macro level blueprint|
|Clinical Dimensions||Direct observation of mini-CEX||ITA / Structured supervision||360° assessment|
|Communication skills||✓ (With patients)||✓ (With patients & colleagues)||✓ (With colleagues)|
|Ability to work as an
effective member of
the health care team
|Ability to apply
aspects of public health
relevant to clinical settings
|Figure 3 – An example of a micro level blueprint|
|This figure shows the distribution of mini-CEXs across different aspects of the clinical skills dimensions and across clinical areas|
|Predominant focus of
the encounter for
Direct Observation Assessment
|Adult Health – Medicine||Adult Health – Surgery||Women’s Health O&G||Child Health||Mental Health||Emergency Medicine|
|History Taking||Encounter G||Encounter I|
|Physical Examination||Encounter A||Encounter C|
|Encounter E||Encounter K|
|Encounter B||Encounter F||Encounter J|
|Clinical Procedures||Encounter D||Encounter L|
Decide on the balance between methods and the relative importance of content. This will enable the cells within the matrix to be completed. Decide if all of the dimensions of performance defined carry equal weight, and if there are mandatory components that must be completed by all candidates.
- Stakeholder input
Commitment to workplace-based assessment from senior leadership is essential to its success. The feasibility of the assessment is important. Thus, stakeholder input on the development of the blueprint is desirable to ensure that the plan devised is acceptable and feasible.
Ensure all assessors are familiar with the blueprint. Assessors need to understand how the assessment encounter that they are involved in fits into the whole assessment of a candidate and how the results of the assessment will be used. This can be incorporated into assessor training (see Lesson 8). Failure to do so runs the risk that individual assessors will fail to appreciate the importance of their individual task or may extend their assessment to include other attributes, and clinical cases which have or will be assessed by others.
- Access to blueprint
A macro level blueprint should be available to assessors and IMGs, to demonstrate how candidates will be assessed in the workplace.
Processes should be established to monitor adherence to the blueprint over the period of the assessment process. Ideally, assessments should be scheduled over a period of time.