Evaluating Item Reliability

In LearningSpace, there is a statistical number that can help identify whether an item can reflect the necessary knowledge and can show the correlation between certain answers and Case total scores.

Point Biserial Value (PBS)

The Point Biserial (PBS) value shows how likely it is that students who selected a certain answer for an item would get a high overall score in the Case. The value of the point biserial ranges form -1 to +1.
The higher this value is, the higher the correlation between the answer and the Case total score.

In CAE LearningSpace, this value can be applied either only to the correct answer or to each answer, depending on the Activity Reporting Setup.

If applied to the correct answer, the more the point biserial value tends to +1, the "better" the item. That is, items that are reliable have a high PBS value, meaning that they are selected by learners having high overall results in the Case. By contrast, items with low PBS value (below zero) tend to get selected by learners having weaker final results in the Case.

M_c = Case total average for learners answering item correctly
M_i = Case total average for learners answering item incorrectly
S_t = standard deviation for the whole Case
c = proportion of the learners in the learner group of the activity answering correctly
i = proportion of the learners in the learner group of the activity answering incorrectly

Example:

1. A checkbox item has three possible answers, with 1, 2 and 3-point answers respectively.
2. There are 10 learners in the learner group.
3. Seven of them got the maximal 6 points, but they have achieved only 60 percent as their Case total scores.
4. The other three members of the group achieved 3 points for the item, but they have 90 percent Case totals.
5. In this Case, if the point biserial (PBS) value is calculated for the "correct" answer of the item, and the answer is regarded correct if the achieved point value for it is the 100% of the max (set on the Activity Reporting Setup page), then the PBS is the following:

This shows that all the learners who had the "correct" answers for the item belong to the "bad" group considering the Case Total Performance. In a real situation, it is obviously more complicated; however, the methodology is the same.

On the Activity Reporting Setup page, it can be determined if the "correct" answer(s) should be those that result in the maximum achievable point value for the item, or a certain portion of the maximum points should already be regarded as a correct answer. The maximum achievable score is obvious for questions with one possible answer choice, while for items that have multiple possible answer choices (i.e. checkbox or scrollbox questions), the maximum score is the sum of the achievable points for all the answers.

The point biserial value is displayed in the Case Item Analysis.