Global and Individual Treatment Effects Using Machine Learning Methods

Publication Information

Authors:

  • Bevan I. Smith, University of the Witwatersrand
  • Charles Chimedza, University of the Witwatersrand
  • Jacoba H. Bührmann, University of the Witwatersrand

Pages:

  • 431-458

Keywords:

  • Machine learning, Treatment effects, Counterfactuals, Propensity score matching, Personalized learning, SHAP

Abstract:

  • Identifying students at risk of failing a course has potential benefits, such as recommending the At-Risk students to various interventions that could improve pass rates. The challenges however, are firstly in measuring how effective these interventions are, i.e. measuring treatment effects, and secondly, to not only predict overall (average) treatment effects, but also individual treatment effects. Previous work has used machine learning (ML) to estimate treatment effects by training models on treatment and control groups and using these models to predict counterfactuals. This is called the machine learning for predicting treatment effects, or MLTE method, in this study. This paper extends this previous work by (1) being the first to compare, or validate, the MLTE method, with the traditional propensity score matching (PSM) method; (2) using the MLTE method to compute not only global, but also subgroup and individual (heterogeneous) treatment effects and (3) by developing a method that combines the MLTE method with methods such as SHAP (SH apley A dditive exP lanations), to predict final grades, predict individual treatment effects and characterize individual students. This is a comparative study between control and experimental groups, using observational data from a first year engineering mechanics course. Students could self-select to attend extra weekly tutorials (called the intervention), resulting in treatment and control groups. The aim was to measure the treatment effects of the intervention by comparing the difference in outcomes between treated and control groups. One of the main results was to develop a method for not only predicting final grades for subgroups and individuals, but also predicting treatment effects. This is valuable for predicting how a student/subgroup would perform and whether the intervention would benefit them. It allows for determining which types of students (or subgroups) would benefit from the intervention. This method shows the most promise for catering to individual students by not only predicting individual students’ final grades (mean absolute errors of between 6% and 8% were computed), but also predicting by how much the intervention would benefit the student (individual treatment effect, ITE) and by characterizing these students using SHAP. In terms of the ML models used, tree-based models performed the best in predictions. The method developed in this study therefore promises much for using ML to help individual students pass a course by identifying individuals that will receive the most benefit from the intervention.