Computational Models for Evaluating Digital Education Effectiveness

Abstract

Evaluating the effectiveness of digital education interventions -- determining whether a new adaptive algorithm, instructional design change, or platform feature actually improves learning outcomes -- is complicated by the high-dimensional, longitudinal, and observational nature of educational data. Randomised controlled trials (RCTs) are the gold standard for causal inference but are often infeasible (ethical concerns about withholding interventions, logistical complexity across institutions) or underpowered (insufficient sample sizes for subgroup analyses). Computational evaluation models -- Bayesian hierarchical models, structural equation models, interrupted time-series analysis, propensity score methods, and meta-analytic synthesis models -- provide rigorous alternatives and complements to RCTs that can exploit the rich longitudinal data generated by digital learning platforms. This paper proposes the Computational Education Effectiveness Evaluation Framework (CEEEF), a systematic comparison of six computational evaluation models applied to eight digital education effectiveness questions across 18 datasets from prior studies in this journal. CEEEF establishes the conditions under which each model provides valid causal inference and introduces the Evaluation Model Quality Score (EMQS) integrating causal validity, statistical power, and practical applicability. Key results: Bayesian hierarchical models achieve the highest EMQS (0.912) for multi-institutional evaluation; interrupted time-series achieves highest sensitivity to intervention effects in longitudinal LMS data; propensity score matching achieves 96.4% accuracy in replicating RCT estimates from observational data. The framework provides computational evaluation model selection guidance for educational technology researchers.