Evaluating Sentence-BERT-powered learning analytics for automated assessment of students' causal diagrams

Abstract

Background: When learning causal relations, completing causal diagrams enhances students' comprehension judgements to some extent. To potentially boost this effect, advances in natural language processing (NLP) enable real-time formative feedback based on the automated assessment of students' diagrams, which can involve the correctness of both the responses and their position in the causal chain. However, the responsible adoption and effectiveness of automated diagram assessment depend on its reliability. Objectives: In this study, we compare two Dutch pre-trained models (i.e., based on RobBERT and BERTje) in combination with two machine-learning classifiers—Support Vector Machine (SVM) and Neural Networks (NN), in terms of different indicators of automated diagram assessment reliability. We also contrast two techniques (i.e., semantic similarity and machine learning) for estimating the correct position of a student diagram response in the causal chain. Methods: For training and evaluation of the models, we capitalize on a human-labelled dataset containing 2900+ causal diagrams completed by 700+ secondary school students, accumulated from previous diagramming experiments. Results and Conclusions: In predicting correct responses, 86% accuracy and Cohen's κ of 0.69 were reached, with combinations using SVM being roughly three-times faster (important for real-time applications) than their NN counterparts. In terms of predicting the response position in the causal diagrams, 92% accuracy and 0.89 Cohen's κ were reached. Implications: Taken together, these evaluation figures equip educational designers for decision-making on when these NLP-powered learning analytics are warranted for automated formative feedback in causal relation learning; thereby potentially enabling real-time feedback for learners and reducing teachers' workload.