MSF 1: Multi-source feedback in workplace-based assessment (accessible)

Section 1: Introduction to workplace-based assessment

What is workplace-based assessment?

  • Workplace-based assessment (WBA) is assessment conducted in the context of a doctor’s everyday work.
  • Doctors are assessed and given immediate and constructive feedback in real clinical situations.
  • Feedback enables planning for personal and professional development based on actual performance.
  • Feedback may be formative or summative.

Common workplace-based assessment tools

  • Mini-CEX
  • Case-based discussion (CBD)
  • Direct observation of procedural skills (DOPS)
  • Multisource feedback (MSF)
  • In-training assessment (ITA)

Section 2: About multisource feedback

What is multisource feedback?

  • Multisource feedback (MSF), or 360° feedback, is questionnaire-based feedback provided to an individual regarding key performance behaviours.
  • MSF has been widely used in industrial settings.
  • Feedback is obtained from colleagues and/or co‑workers and/or patients who have had significant interaction with the individual.

Feedback from multiple sources

  • Feedback from supervisors is not anonymous, whereas feedback gathered from multiple sources is usually de-identified and aggregated.
  • Participants receive feedback results in the form of aggregated scores (e.g. mean scores) on each question and/or in a number of domains.

What does MSF provide?

  • MSF provides doctors with a range of views on their performance. It includes feedback on how others perceive their attributes and skills in areas that may be hard to assess using other conventional assessment methods.

The role of MSF in assessing clinicians

  • The major role of MSF is to provide feedback from a broad range of colleagues, team co-workers and those receiving treatment.
  • Different questionnaires are usually administered separately for colleagues, co-workers and patients as they can provide perspectives in different domains according to their roles and relationship with the doctor.
  • The feedback provides information on how well the doctor manages everyday clinical tasks and functions in a healthcare team.

Evidence base for MSF

  • Research shows that reliable data can be generated with a reasonable number of respondents, and that physicians will use the feedback to contemplate on their practices and initiate changes where needed.

Why use MSF?

  • MSF assessment recognises that doctors work with other doctors to deliver care, usually in inter‑professional teams, and that they interact with many different healthcare professionals in their everyday work.
  • MSF captures these essential perspectives for the benefit of the doctor’s personal and professional development.

Feedback and reflection

  • Perhaps the most important aspect of MSF is the opportunity for self-reflection followed by receiving collated results and feedback.
  • The value is enhanced by discussing the results and feedback with a trusted colleague or mentor and developing an action plan for improvement.

Section 3: What does MSF assess?

MSF provides a team view

  • A team view of the doctor is important because of the importance of team work in high-quality clinical practice.
  • MSF responses may elicit knowledge about potential areas for development that might not otherwise have been elicited.

The importance of level

  • Respondents should be aware of, and attuned to, the level of performance expected of the doctor.
  • Respondents should be experienced with clinicians at the expected level (e.g. involved in teaching and assessing junior doctors).
  • Those being assessed should be wary of recommending people at the same level as themselves as they may not have the perspective to assess their colleagues.

Areas for assessment with MSF

  • Some of the areas that may be assessed in MSF include interpersonal skills, communication, teamwork, patient advocacy, teaching, time management, professionalism plus self-evaluation.
  • MSF is not a replacement for clinical audit or other tools which assess clinical skills.

Key themes in domains

  • Behaviours examined must be appropriate and have face and content validity.

    Some examples include:
    1. Shows respect for co‑workers, colleagues and patients
    2. Respects the rights of patients
    3. Maintains patient confidentiality
    4. Communicates effectively with co‑workers, colleagues and patients
    5. Works well in a team

Respondents’ expectations

  • Respondents should have expectations in keeping with the doctor’s level of training when responding to questions such as the following:
    1. Does the doctor accept responsibility for their actions?
    2. Does the doctor refer appropriately?
    3. Does the doctor assist with care outside of regular office hours?

How does MSF differ from other WBA methods?

  • Other WBA methods (e.g. mini-CEX) rely on direct observation at a single point in time by a single assessor.
  • MSF relies on a number of observations over time from multiple respondents and from different perspectives.
  • MSF is a validated method for gaining feedback from colleagues co-workers and patients.

Section 4: Practicalities for MSF doctors and assessors

How does MSF work?

  • The doctor identifies a range of suitable colleagues and co-workers from whom responses could be sought.
  • The responsible party (e.g. the program director) ensures that the people put forward are suitable and there is no real or perceived conflict of interest.
  • Respondents must be given an explanation of the MSF process and the closing date.

What characteristics are necessary for respondents?

  • Respondents must have had a direct and recent experience of working with the doctor over a period of time.
  • Such experience enables them to make an informed judgement about the doctor’s capabilities.
  • A respondent who does not know the doctor well should not complete an assessment.
  • The respondent should not have a conflict of interest.

Who is a medical colleague?

  • Medical colleagues may include other doctors in the same practice or specialty, colleagues that a doctor refers their patients to, and colleagues that receive patients from the doctor on admission or that are consulted during an admission.
  • Colleagues can be trainees, advanced trainees, fellows and consultants.
  • Ideally, a mix of colleagues at different levels is best.
  • Medical colleague respondents should be limited to those with full Australian registration

Who is a co-worker?

  • Co-workers may include members of the nursing staff, allied health staff, administrative staff, ward clerks, and all other health professionals.
  • Co-workers can be drawn from those working in both hospital and out-of-hospital sites.
  • Co-workers are important respondents whose perspective may be different from that of colleagues.

How many responses are necessary?

  • The literature suggests that a reliable picture can be obtained with 10–30 responses from both colleagues and co-workers.
  • If patients are included, more than 30 responses are required.

Assessment scoring

  • A 5-point scale is usually used.
  • Scores must relate to how the candidate performs and not to factors beyond his/her control.

Collating results

  • If written responses are requested, respondents should ensure that they do not identify themselves directly. This may compromise their own and others’ confidentiality.
  • Electronic forms are more efficient.
  • De-identified results are collated on a summary form.
  • Illustrative comments, also de-identified, may be included and are valuable to the doctor in understanding the reasons for a score.

The feedback discussion

  • Feedback is part of a continuum of learning.
  • After an MSF assessments have been collated, an MSF review panel may be convened and the doctor, assisted by the administration team, should make a formal time of about 20–30 minutes to discuss their feedback with the WBA program director or nominee.
  • The literature suggests that it can be useful to discuss and reflect on the differences between a doctor’s self-assessment and the assessments provided by the doctor’s colleagues.

AMC Standard Pathway (workplace-based assessment)

  • The AMC accredits healthcare providers to conduct WBA programs for international medical graduates on the AMC Standard Pathway.
  • Eligible candidates can apply for a WBA program conducted by an AMC-accredited provider as an alternative to the AMC clinical examination.

MSF in AMC-accredited WBA programs

  • MSF is one of several assessment methods used by AMC-accredited providers of WBA programs.
  • MSF is not used in isolation; it is part of a program of assessment, feedback and professional development.
  • Results of MSF must be sufficient to obtain a pass in the MSF component.

Formative and summative assessments

  • A formative MSF assessment should be given early in the WBA program so that the candidate may receive feedback on their identified strengths and areas for improvement. An action plan should result.
  • The summative MSF assessment is given later in the program and counts towards successful completion of the WBA program.

Level of assessment for AMC candidates

  • AMC candidates are assessed at an intern level. However, the level at which they are employed may be higher than this (e.g. they may be in a PGY2/3 or registrar position in a single discipline).
  • Respondents must be mindful of the level of expectations for fairness, and assess at the intern level for all disciplines being assessed.

Feedback techniques

  • The program director should be trained and experienced in giving feedback on a wide range of professional attributes.
  • They should:
    1. minimise any threat to the candidate
    2. highlight the candidate’s strengths first
    3. engage the candidate in discussing the reason for any less positive results
    4. discuss the candidate’s approach to potential areas for development in their everyday work.

Feedback discussions

  • The candidate should reflect on their self‑assessment, articulate how they could improve on their results (if necessary) and consider the possibilities for further professional development.
  • For a summative assessment, progress since a previous formative MSF should be highlighted.
  • The program director and the candidate then develop an action plan.

Tips for candidates

  • Do not nominate a respondent with a conflict of interest (e.g. a personal relationship).
  • Nominate a range of people who know you well and have worked with you recently.
  • Use a wide range of respondents to get the broadest picture possible of your workplace activity.
  • Allocate enough time to advise the administrative unit of the respondents, contact them and receive their responses.
  • Develop an action plan after a formative assessment.

Tips for respondents

  • Make sure you know the candidate.
  • Allocate uninterrupted time to complete the task (usually 10–15 minutes).
  • Keep in mind the level at which you are assessing; for AMC standard pathway (WBA) you are assessing at the intern level.
  • Be fair and honest.
  • Use the whole scale.
  • Consider using illustrative comments, if requested, which can be very helpful.
  • Do not identify yourself in comments as this may compromise others’ confidentiality.
  • Give a balanced report – strengths are as important as areas for development.
  • If you cannot complete on time, do not agree to assess.

“It seems, therefore, that multisource feedback can lead to improved performance, but individual factors, the context of the feedback, and the presence (or absence) of facilitation can have a profound effect on the magnitude of the response.”

Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ 2010;341:c5064