RG-05: Blueprinting

A blueprint depicts the relationship between what has to be assessed and how it is to be assessed.

Assessment involves a sampling of a candidate’s performance. It is impossible to test for all clinical dimensions for all medical conditions. The sample of performance that is tested has to be representative of the spectrum of that relevant to safe medical practice. A blueprint defines the aspects of an IMG’s performance that will be assessed, how each aspect will be assessed, and the number of performances that are to be assessed. It ensures that there is an appropriate balance between what is assessed and how it is assessed. Blueprints are constructed to ensure that assessments are balanced and valid, whether they be for summative or formative purposes.

In its simplest form, a blueprint is a matrix where the rows relate to what is being assessed and the columns relate to how to assess. The cells within the matrix are completed as decisions are made about which method best assesses which attribute.

Steps to develop a blueprint

1. Levels of assessment: Identify the levels for which the blueprint is being constructed.

Blueprinting can be undertaken at different levels. For an undergraduate medical program, a blueprint might be constructed at the macro level, or whole program, showing where various methods are used to assess objectives or outcomes across all years of the program. There may be an additional blueprint to show how an individual year assessment is planned. There may then be a more detailed, micro-blueprint at the level of a semester-long course showing the relationship between learning objectives/semester content and written and clinical examination methods.

For workplace-based assessment, blueprints will also need to be constructed at different levels. At the macro level, a blueprint should be developed to show how the methods selected assess the defined clinical dimensions of performance; this will be the overall assessment plan. A micro level blueprint will also be needed to plan the implementation of each assessment method at the clinical level. For example, a blueprint for mini-CEX observations should be developed to define how to distribute observed encounters across clinical disciplines, clinical tasks and possibly clinical sites. Examples of blueprints are shown later in this lesson.

2. What to assess: Decide on what is being assessed.

For workplace-based assessment in the IMG context, the dimensions of clinical performance that are to be assessed have been defined in the AMC workplace-based assessment accreditation guidelines for IMGs.

The clinical dimensions are:

• Clinical skills (history taking, physical examination, investigation and diagnosis, prescribing and management, counselling/patient education and clinical procedures).
• Clinical judgment.
• Communication skills.
• Ability to apply aspects of public health relevant to clinical settings.
• Cultural competence.
• Ability to work as an effective member of the health care team.
• Professionalism and attention to patient safety.

3. Suitability: Decide on what assessment methods are suitable.

Lesson 4 includes a review of the methods available for workplace-based assessment. Consider which methods will appropriately assess the various dimensions of performance. Figure 2 shows one possible sample of observed encounters, using the mini-CEX method, across the clinical dimensions and the clinical areas. Figure 3 shows an example of a micro level blueprint showing the distribution of mini-CEXs across different aspects of the clinical dimensions and across clinical areas. Other plans are possible.

Figure 2 – An example of a macro level blueprint
Clinical Dimensions	Direct observation of mini-CEX	ITA / Structured supervision	360° assessment
Clinical Skills	&#x2713
Clinical Judgment	&#x2713
Communication skills	&#x2713 (With patients)	&#x2713 (With patients & colleagues)	&#x2713 (With colleagues)
Ability to work as an effective member of the health care team			&#x2713
Ability to apply aspects of public health relevant to clinical settings		&#x2713
Cultural competence		&#x2713
Professionalism and attention to patient safety	&#x2713	&#x2713	&#x2713

Figure 3 – An example of a micro level blueprint
This figure shows the distribution of mini-CEXs across different aspects of the clinical skills dimensions and across clinical areas
	Clinical Areas
Predominant focus of the encounter for Direct Observation Assessment
	Adult Health – Medicine	Adult Health – Surgery	Women’s Health O&G	Child Health	Mental Health	Emergency Medicine
History Taking				Encounter G	Encounter I
Physical Examination	Encounter A	Encounter C
Investigations and Diagnosis			Encounter E			Encounter K
Prescribing and Management	Encounter B		Encounter F		Encounter J
Counselling/Patient Education				Encounter H
Clinical Procedures		Encounter D				Encounter L

4. Balance

Decide on the balance between methods and the relative importance of content. This will enable the cells within the matrix to be completed. Decide if all of the dimensions of performance defined carry equal weight, and if there are mandatory components that must be completed by all candidates.

5. Stakeholder input

Commitment to workplace-based assessment from senior leadership is essential to its success. The feasibility of the assessment is important. Thus, stakeholder input on the development of the blueprint is desirable to ensure that the plan devised is acceptable and feasible.

6. Training

Ensure all assessors are familiar with the blueprint. Assessors need to understand how the assessment encounter that they are involved in fits into the whole assessment of a candidate and how the results of the assessment will be used. This can be incorporated into assessor training (see Lesson 8). Failure to do so runs the risk that individual assessors will fail to appreciate the importance of their individual task or may extend their assessment to include other attributes, and clinical cases which have or will be assessed by others.

7. Access to blueprint

A macro level blueprint should be available to assessors and IMGs, to demonstrate how candidates will be assessed in the workplace.

8. Monitoring

Processes should be established to monitor adherence to the blueprint over the period of the assessment process. Ideally, assessments should be scheduled over a period of time.