An investigation of Mathematical Literacy assessment supported by an application of Rasch measurement

Authors: Caroline Long, Sarah Bansilal and Rajan Debba

Mathematical Literacy (ML) is a fairly new subject in secondary schools and work needs to be done on ways of assessing students’ competence. Tests need to be very carefully designed and marked if they are to be a reliable instrument and provide useful information to teachers and to the Department of Education.

This article reports on a Grade 12 ML preparatory examination, which is intended to prepare students for the final matric examination. The preparatory exam was remarked to ensure that the final marked version was entirely consistent with the marking memorandum supplied by the Department of Education. Using the Rasch measurement theory, the authors reviewed and revised the scoring procedures.

In their study, the authors investigated each item and the logic behind the scoring, as well as learner responses. Using the output of the Rasch analysis, together with expert teacher input, they explored the cases where rescoring was necessary. This article reports on the analysis of the test prior to rescoring, the analysis and rescoring of individual items, and the post-rescore analysis.

Scoring problems identified included an item showing haphazard misfit, items where two marks were allocated rather than just one mark, and an item which showed disordered thresholds. The fact that the Rasch measurement theory allows for both dichotomous responses (choice between two) and polytomous responses (choice between three or more) was an advantage; also that it aligns item difficulty and person proficiency.

The study revealed the important role of scoring rubrics in contributing to the validity of assessments. Examiners need to ensure that each mark that is allocated can be justified educationally. Marks should not be allocated on the basis of time that is required to be spent during the examination. Neither should marks be allocated for guessing. If these checks are not taken into account, then the total score has diminished meaning. The authors found that their new scoring procedures allowed greater precision than the original one.

Full Text