Automated marking system for essay questions.

Obot, Okure U.; Obike, Peter; James, Imaobong

Abstract

The stress of marking assessment scripts of many candidates often results in fatigue that could lead to low productivity and reduced consistency. In most cases, candidates use words, phrases and sentences that are synonyms or related in meaning to those stated in the marking scheme, however, examiners rely solely on the exact words specified in the marking scheme. This often leads to inconsistent grading and in most cases, candidates are is advantaged. This study seeks to address these inconsistencies during assessment by evaluating the marked answer scripts and the marking scheme of Introduction to File Processing (CSC 221) from the Department of Computer Science, University of Uyo, Nigeria. These were collected and used with the Microsoft Research Paraphrase (MSRP) corpus. After preprocessing the datasets, they were subjected to Logistic Regression (LR), a machine learning technique where the semantic similarity of the answers of the candidates was measured in relation to the marking scheme of the examiner using the MSRP corpus model earlier trained on the Term Frequency-Inverse Document Frequency (TF-IDF) vectorization. Results of the experiment show a strong correlation coefficient of 0.89 and a Mean Relative Error (MRE) of 0.59 compared with the scores awarded by the human marker (examiner). Analysis of the error indicates that block marks were assigned to answers in the marking scheme while the automated marking system breaks the block marks into chunks based on phrases both in the marking scheme and the candidates' answers. It also shows that some semantically related words were ignored by the examiner.

Citation

OBOT, O.U., OBIKE, P. and JAMES, I. 2024. Automated marking system for essay questions. Journal of engineering research and reports [online], 26(5), pages 107-126. Available from: https://doi.org/10.9734/jerr/2024/v26i51139

• Journal Article Type: Article
• Acceptance Date: Apr 2, 2024
• Online Publication Date: Apr 8, 2024
• Publication Date: May 31, 2024
• Deposit Date: Dec 4, 2024
• Publicly Available Date: Jan 10, 2025
• Journal: Journal of engineering research and reports
• Electronic ISSN: 2582-2926
• Publisher: Journal of Engineering Research and Reports
• Peer Reviewed: Peer Reviewed
• Volume: 26
• Issue: 5
• Pages: 107-126
• DOI: https://doi.org/10.9734/jerr/2024/v26i51139
• Keywords: Microsoft research paraphrase (MSRP) corpus; Semantic similarity; Machine learning; Logistic regression; Marking scheme; TF-IDF; Natural language processing
• Public URL: https://rgu-repository.worktribe.com/output/2613442
• Publisher URL: https://doi.org/10.9734/jerr/2024/v26i51139