What is MSF

MSF, or multi-source feedback is a form of collated feedback from supervisors, peers, and subordinates.  During Foundation Training, there are no true 'subordinates', but it should still be possible to obtain feedback from a representative sample of colleagues within the health care professions.  A self MSF is also completed to allow comparison between the perceptions of self and others.

There are two tools currently in use - miniPAT and TAB (see below).  This Foundation School is currently conducting an evaluation to compare the utility of each tool.

TAB

TAB stands for Team Assessment of Behaviour and is the required MSF tool approved for all foundation trainees after August 2010.

miniPAT

miniPAT stands for mini Peer Assessment Tool.  It has various other names including 360 appraisal.

Purpose

It's purpose to help the trainee gain insight into their progress and performance against the GMC domains of "Good Medical Practice". To be maximally useful it depends on:

• Assessments from a minimum of ten (15 should be nominated) colleagues who represent seniors and peers, including medical, nursing and Allied Health Professionals.  In NYECFS the process requires the educational supervisor to be one of the raters
• A self assessment of the same domains by the trainee.
• A meeting (appraisal) between the trainee and the educational supervisor to discuss the results of Peer and self assessment and the differences between the two.

information for trainees

When is MSF done

In F1 it is done during month three of the FIRST post (October)
  such a round appears as yyyy11
In F2 it is done during month three of the SECOND post (February)
  such a round appears as yyyy22

IF there is a requirement for it to be repeated it will be done during the third month (February) of the SECOND post for F1 and/or the third month (May/June) of the THIRD post for F1 and F2.
  additional rounds for F1:  yyyy12 or yyyy13
  additional round for F2:  yyyy23

using the NYEC e-MSF

1. Log on using your usual login and password. Select 'training' and then 'multi-source feedback' (which will bring you back to this page).
2. Select 'assign raters' from the submenu at the top of this page (only visible when you are logged in).
3. The choice of raters should be agreed with and include your educational supervisor and should follow the recommendations below.  Please note we do not use non-clinical raters.
4. Enter the details including a valid email address for each rater in an online form.
5. An automatic email is sent to each rater asking them to log on and complete the MSF for the trainee.  (If the email address is not on the NYEC system an MSF rater account will also be created.)
6. Once the form has been completed, it will be indicated on the trainees rater assignment page and an automatic email will be sent to the trainee confirming the form has been completed.  At no stage are individual raters scores revealed to trainees
   
   If a rater does not make a return, the trainee can deselect that rater and assign another.  Once the time window for completion is over, the submitted scores  and summary report will be returned to the trainee by the Educational Supervisor for discussion.  The summary will be added to the trainee assessment records automatically.
    
7. All changes logged within the e-MSF system will be confirmed by email to all relevant parties in line with the school data security policy.

Selecting assessors

The Foundation Curriculum (2010) sets out the requirements for assessors. The recommended mix of raters/assessors is as follows:

1. 2–8 doctors more senior than F2, including at least one consultant or GP principal
2. 2–6 senior nurses (band 5 or above)
3. 2–4 allied health professionals
4. 2–4 other team members including ward clerks, secretaries and auxiliary staff. These members of staff are NOT used in NYECFS MSF. In NYECFS we use F1s, F2s, and auxiliary nursing staff in this category (because TAB is combined with miniPAT which requires clinical staff to complete it).

Note: There must be a minimum of two in each group providing reports.

Completing the self assessment MSF

1. Log on using your usual user login
2. Select the 'self MSF' option.
3. Complete the form.
4. Once submitted it will be locked AND CANNOT BE ALTERED.
5. A 'tick' will appear at the bottom of the table of assessors to indicate it has been completed.

Viewing the summary

1. Once the MSF round is complete, a summary page for MSF will be available which will indicate the raters scores, the trainees self score, and the cohort score for the school.
2. These results should form part of the final meeting with the educational supervisor.

information for raters

Using the NYEC e-MSF

1. All raters including trainees and educational supervisors who are already registered as users, will need to log in - the link for this can be found in the tabs at the top of this page.
2. The first time you log in, or at any time later if you forget your password, leave the password field blank, click submit and follow the instructions.  More detailed help is available by clicking the 'password help' link from the log in page
3. Once logged in, you will be asked to verify your personal details.  If you have been added as a new assessor, a trainee may have some of your details incorrect.  We also ask you to confirm your current grade, as the grade of many staff changes from year to year and we log your current grade against the assessment provided.
4. Once your details are confirmed you can proceed to submit your scores. A drop down box listing all trainees who have requested an MSF report will be shown.
   Please note:  If you have given different email addresses to different trainees you will need to log in with each different email address to submit all your reports.
   TIP:  stick to one email address for all MSF reports.
5. Once the MSF is completed, that record will be locked AND CANNOT BE ALTERED.
   The trainees name will be highlighted in the drop down box and the trainee will see a 'tick' by that rater to indicate the same.
6. An automated email will be sent to you and the trainee confirming that the MSF report has been submitted.
7. A trainee cannot view an individual report, and the results are only made available once the round is completed.

The results

The results are first made available to the Educational Supervisor and the Foundation Training Programme Directors (FTPDs).  Once an Educational Supervisor has reviewed the results, they will be 'unlocked' for the trainee to view.  Once the results are unlocked they will be accessible to the trainee. 

TAB report

Trainees should then add the TAB report to their e-portfolio in the personal library in the format F{yr}TAB{post}.  For example F1TAB1 for the MSF done during the first post of F1.

miniPAT report

The miniPAT report is not REQUIRED for the e-portfolio, but trainees are encouraged to add it to their personal library as evidence of professional development. It should be added in a similar format to the TAB report (see above) using the format F{yr}PAT{post}.

What is shown?

Five pieces of information are shown on the miniPAT report:

1. Raters scores
   The number of raters, average scores, and the grouped averages are shown.  Meaningful results can only be gained where there are more than eight raters AND . . .
2. Distribution of raters by seniority/profession
   A good distribution makes interpretation more meaningful - there should be the Educational Supervisor, senior medical colleagues, peer colleagues, and other clinical professionals.
3. Self Score
   The trainees' self assessment
4. Cohort score
   The average for each domain and each group of domains is shown from all the assessments submitted by colleagues (this does NOT include self assessment scores).
5. Comments
   Trainee comments are shown first, followed by raters comments in RANDOM order!

How to interpret the results

The results should be interpreted summatively (how well am I doing), formatively (where are they areas I need to improve), AND comparatively (how well do I perceive I am doing when compared to my assessors view of me, and how am I doing compared to the school as a whole)

Whilst the standard for completion of each year is a score of 4, it is well recognised that even at this stage a proportion of trainees will be exceeding this, and the national mean is nearer to 5 than 4.

The raters scores

1. Individual domains
   Score of 4 or more are fine.  Occasional scores below 4 should be used to determine areas for development, rather than as an intrinsic sign of failure.
2. Domain groups
   A more rounded way of considering questions from similar domains.  They can be interpreted as in 1 above.

Summative - scores alone
Formative - scores in context of comments

My scores v raters scores

1. Self scores less than raters scores (domain or domain groups)
   Most trainees will score below their raters.  This is usual, and is often simply a reflection of lack of confidence on the part of the trainee
2. Self scores above raters scores
   Your educational supervisor will explain this to you, but it is often a sign of confidence being ahead of competence.

Summative - compare scores
Formative - scores in context of comments.  Compare your own comments with those of your raters

Raters scores v school cohort

1. These are the most reflective of overall capability.  It provides some idea of how an individual trainee can be compared to the school as a whole.  However, it does not necessarily mean that those below the average for the school are not meeting the expected standard.
2. miniPAT is a well recognised and reliable measure of overal performance.  We live in a competitive world, and whilst the scores are not currently used in the application process, those who are much below the school average may face significant challenges to reach the standards for recruitment to training posts beyond the Foundation Programme.