SSC Campus - Student Profile: Risk, Skills Analysis, 2.5 Risk Model
The Student Success Collaborative data platform predicts individual students’ likelihood to graduate in a wide range of majors on each campus. Data on historical student cohorts is collected from your information systems and used to generate an initial series of candidate academic metrics. Advanced multivariate statistical and machine learning techniques perform variable selection and conduct hypothesis testing to highlight the most impactful trends in student data. Results are cross-validated for accuracy and compared with known research on student success outcomes. This process may be referred to as “model training” by your consultant. More information is provided in additional sections in regards to our most up to date predictive model.
The Risk Score Analysis shows the risk areas and how the student compares to students who successfully completed the major. Looking at risk area performance will also indicate why the student is or is not at high risk of not completing the major. The first skill area usually rises to the top because the student’s performance has a high importance to major completion and the student’s comparative performance is below average compared to the historical norm. In addition, the risk score analysis will make suggestions about what the student can do to improve chances of succeeding in each risk area.
2.5 Risk Model: Purpose
The risk model is used to help prioritize advisor and staff interaction with students based on their likelihood to graduate and support advisors in quickly identify at risk students The model estimates the likelihood that current students will graduate by comparing them to historical students at your institution with similar academic and demographic characteristics.
Creation of the Predictive Model Using Historical Data
Our data scientists use the academic and demographic data of your historical students to create a predictive model specific to your campus. Then current students are ‘scored’ or applied against your campus’s predictive model on a scale from 0 - unlikely to graduate - to 1 - most likely to graduate. We work with your institutional leadership team to translate ranges of those scores to red, yellow, and green colors. As our Data Science team creates a model, they test many different academic and demographic data factors for their predictive quality. Some of these data points are available directly in your SIS, others are derived by our Data Science team using proprietary methods. The data points used in your model include courses student’s took, grades earned, majors declared, the way they earned credit over time, in addition to more static facts, like whether they were a transfer student or their high school GPA. These data points have different weight in the model, and the weight vary over the progress of the student through their degree.
Our analytic research and work with member information has shown that students tend to receive similar grades in particular groups of courses, and we call the reasons behind these grade patterns "skills". Individual courses can be associated with multiple skills, and courses most associated with a particular skill won't necessarily belong to the same or similar academic discipline. Skills provide the predictive model information about students' aggregate coursework not available through individual course grades or GPAs. The risk model analysis displays an individual current student's estimated skills as a percentile compared to all students who graduated in that student’s current major(s). We have created the following lists of courses most associated with each skill to help us identify names for the skills, as well as optional descriptions, for display in the platform.
The data science team looks at both the accuracy and calibration of your institutional model by comparing your customized model to a baseline model. The baseline model is based purely on accumulated credits and cumulative GPA. Your institutional model adds more data points, sophistication and nuance of how it evaluates those data points, and therefore performs better than the baseline in terms of two metrics that we use to evaluate model quality: Kappa and AUC. We also look at both accuracy and calibration. Your institutional model is more predictive because there is a bigger spread between the predicted grad rates between the bottom and top quartile (looking from top to bottom), and it is better calibrated because there is less difference between the predicted and actual grad rate.
Our data science team is confident that this model will help your advisors accurately prioritize interaction with students based on their likelihood to graduate and identify students who are at risk.