Enterprise Help Site

Page tree

Versions Compared

Old Version 6

changes.mady.by.user CAE author

Saved on

compared with

New Version Current

changes.mady.by.user CAE author

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

When evaluating SPs, one of the devices for this is for an Observer to fill out the same questionnaire that the SP filled out, then compare the two.

Note
Observers can review the video recording of encounters, this way making sure of what and how the SP and the learner did during the encounter.

Image Modified
Table of Contents

Difference Value

To evaluate the Inter-Rater Agreement, the system calculates the difference value. For an Item, this value is the absolute value of the difference between the value of the answer selected by the SP and the one selected by the Observer. The difference value of the section is the sum of the difference value of the non-open-ended items.

Note

The Difference value is in points, not percentages.

Image Modified

Oi = value of the answer selected by the observer for an item
SPi = value of the answer selected by the SP for an item
n = number of the items in the SP part of the case


Info

NOTES:

  • If both the SP and the Observer select an answer with the same value for an item, the difference value for this item is zero.
  • The difference value for an item is zero even if they selected different answers if those answers have the same point value.
    The reason for this is that the difference does not affect the score of the learner: it is the same if the SP does exactly as the Observer.
Note
IMPORTANT: If the selection of the answer for an item is important and the SP's choice affects the student's scores, the answers should have different point values.
  • The difference value calculation only uses responses to items with pre-defined answers. Responses to open-ended items like text boxes are NOT used, even if they are scored.

Image Modified


Section Agreement Value

The Difference Value is used for calculating the Section Agreement Value. This percentage value refers to one section of the SP part, and is the difference of the total achievable non-open-ended point value in the section and the difference value, divided by the total achievable point value.

Info

The Section Agreement Value is a percentage value that refers to one section of the SP part: it is the difference of the total achievable non-open-ended point value in the section and the difference value, divided by the total achievable point value.


Image Modified

t = total achievable non-open-ended point value in the section
d = difference value for the section

From the section agreement value of each section of the SP part, the system calculates the Inter-Rater Agreement Value: their weighted average using the section weights as set in the Case Manager for learner scoring. This ensures that disagreement about points in more important (hence higher weighted) sections will affect the Total Agreement Value more.



Inter-Rater Agreement Value

Inter-Rater Agreement Value: sum of the products of each Section Agreement Value within one particular SP part, and the corresponding section weight, divided by the sum of the section weights.

Image Modified

ss = section agreement value for each section of the SP part of a certain Case
ws = section weight belonging to each section within the SP part, defined in the Case Manager
n = number of sections in the SP part



Inter-Rater Agreement Score

The Inter-Rater Agreement Value refers to one observation the SP has about their performance for a certain learner encounter. However, SPs may be evaluated according to several observations in a Case.
The total Inter-Rater Agreement Score is an average of the Inter-rater Agreement Values from all observations of the SP for a certain Case.

Image Modified

io = inter-rater agreement value of an SP, for one particular encounter (i.e. observation)
n = number of observations the SP has for a certain Case

Info
titleRelated pages

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
excludeCurrenttrue
cqllabel = "sppa"

Tip

For further information, see the Reports - How to pages and the Reports Module Training Videos.