When evaluating SPs, one of the devices for this is for an Observer to fill out the same questionnaire that the SP filled out, then compare the two.
Observers can review the video recording of encounters, this way making sure of what and how the SP and the learner did during the encounter.
Difference Value
To evaluate the Inter-Rater Agreement, the system calculates the difference value. For an Item, this value is the absolute value of the difference between the value of the answer selected by the SP and the one selected by the Observer. The difference value of the section is the sum of the difference value of the non-open-ended items.
The Difference value is in points, not percentages.
Oi = value of the answer selected by the observer for an item
SPi = value of the answer selected by the SP for an item
n = number of the items in the SP part of the case
NOTES:
- If both the SP and the Observer select an answer with the same value for an item, the difference value for this item is zero.
- The difference value for an item is zero even if they selected different answers if those answers have the same point value.
The reason for this is that the difference does not affect the score of the learner: it is the same if the SP does exactly as the Observer.
IMPORTANT: If the selection of the answer for an item is important and the SP's choice affects the student's scores, the answers should have different point values.
- The difference value calculation only uses responses to items with pre-defined answers. Responses to open-ended items like text boxes are NOT used, even if they are scored.
Section Agreement Value
The Difference Value is used for calculating the Section Agreement Value. This percentage value refers to one section of the SP part, and is the difference of the total achievable non-open-ended point value in the section and the difference value, divided by the total achievable point value.
The Section Agreement Value is a percentage value that refers to one section of the SP part: it is the difference of the total achievable non-open-ended point value in the section and the difference value, divided by the total achievable point value.
t = total achievable non-open-ended point value in the section
d = difference value for the section
From the section agreement value of each section of the SP part, the system calculates the Inter-Rater Agreement Value: their weighted average using the section weights as set in the Case Manager for learner scoring. This ensures that disagreement about points in more important (hence higher weighted) sections will affect the Total Agreement Value more.
Inter-Rater Agreement Value
Inter-Rater Agreement Value: sum of the products of each Section Agreement Value within one particular SP part, and the corresponding section weight, divided by the sum of the section weights.
ss = section agreement value for each section of the SP part of a certain Case
ws = section weight belonging to each section within the SP part, defined in the Case Manager
n = number of sections in the SP part
Inter-Rater Agreement Score
The Inter-Rater Agreement Value refers to one observation the SP has about their performance for a certain learner encounter. However, SPs may be evaluated according to several observations in a Case.
The total Inter-Rater Agreement Score is an average of the Inter-rater Agreement Values from all observations of the SP for a certain Case.
io = inter-rater agreement value of an SP, for one particular encounter (i.e. observation)
n = number of observations the SP has for a certain Case
Related pages
For further information, see the Reports - How to pages and the Reports Module Training Videos.
Overview
Content Tools