Predictive Learner Analytics

Implementing predictive learner analytics in your product with Predicted Score

knAtom Looking for the technical specs on the Predicted Score API Endpoint?


In this section:


Predicted Score

The Predicted Score analytic answers the question “what score would the student achieve on an exam given today, covering specific sets of learning objectives.”

It can be used to:

  • To help students and instructors focus their review activities to best prepare themselves or their classes for an upcoming exam
  • As an approximation of students’ current level of proficiency with the content covering a learning objective.

Predicted Score leverages

  • All prior work performed by the student against the specified learning objectives.
  • A principled and data driven treatment of the gradual decay of knowledge when not practiced (the forgetting curve).
  • Our latest learning models tuned to achieve optimal predictive performance.

How Does Predicted Score Behave?

The partner application can specify multiple confidence levels C. For each such C, we’ll return a predicted score value S, which is the score at which the model thinks the student has probability C scoring below (and thus probability 100 - C scoring above). This supports a variety of use cases including displaying probability bands for the score, as well as more simple representations such as point estimates (see examples at the end of the document).

Since Predicted Score represents the likelihood of a student achieving a certain grade, by definition it never reaches 100%. However, the partner application can choose to bin or round the resulting value.

Having a forgetting curve means that the student performance will degrade over time to a certain lower-bound if the student does not work on a particular LO. The limit is data-driven and varies for different LOs.

Considerations for Implementing Predicted Score

  • When handling low data/low confidence situations: The partner application can decide in which conditions to show or not show Predicted Score.
    • E.g. don’t show the metric until the student has done x amount of work per learning objective. The partner application can set a learning_objective_coverage_requirement in predicted score requests and receive learning_objective_coverage metrics to gate their frontend behavior.
    • E.g. when showing PS as a range, only show the range if it’s up to 20% wide (i.e. when the band is too wide to be meaningful, don’t show).
  • Assessments with a very small number of questions are not well suited to the “predicted score” interpretation of the metric. Consider a quiz of 5 items, where a student can only ever score 0, 20, 40, 60, 80, or 100. The PS model returns predictions on a hypothetical comprehensive assessment of the LOs, assuming a large number of questions. Thus, it might return a predicted score range of 83-93 (with 80% confidence). Note the student cannot possibly score in this range! Thus, for small quizzes/tests, products should use different messaging. Instead of saying “Knewton thinks your score on the quiz will be between 83 and 93”, instead message “Knewton thinks your overall mastery on the LOs assessed by this quiz is between 83 and 93”.

Predicted Score Performance Considerations

Depending on the following request characteristics, Predicted Score’s response times may range from a few milliseconds to over a second.

Request Characteristic Impact on Response Time
Number of accounts requested Scales linearly with number of accounts
Number of learning objectives requested per group Scales quadratically with number of LOs in each group
Overall score requested (get_overall_score = True) Scales quadratically with number of LOs across all groups
Individual scores requested (get_learning_objective_scores = True) Scales linearly with number of LOs
Number of confidence levels requested Very low impact
Different values for learning objective weights No impact

Note that requesting the same number of LOs broken down into multiple groups will generally be faster than requesting all LOs in a single group. As a best practice, if you only care about individual LO scores, form the request so that each LO is in its own group and set get_learning_objective_scores = false and get_overall_score = false.

Due to the wide variety of use cases that Predicted Score supports, our default rate/entity limits may not work for your application. If your requests will be well below our established entity limits, we may be able to raise your rate limit. Conversely, if you require significantly higher entity limits, we may be able to raise your entity limits and lower your rate limit. Reach out to your Knewton representative to discuss alternatives if your use case requires it.

What is Predicted Score NOT For?

While UX implementations of the analytics can vary, predicted score should not:

  • Be used to illustrate the progress of a student towards completion of an adaptive assignment.
  • Be used as a grade.

For these use-cases, use the Status & Progress analytic instead.

Example Implementation and Use Cases of Predicted Score

Here is an example where the partner requests the predicted score on a group of LOs at the 10th, 50th, and 90th “confidence levels”. The API might return the following predicted scores: {10: 0.71, 50: 0.82, 90: 0.91}. These scores can be used to create the following in-product displays:

Predicted Score Basic Example Example UI: Predicted Score on an Upcoming Exam

As an instructor, I want to identify students who are at risk of failing a test so that I can intervene and help them succeed

Display Predicted Score as a visual range for single student on exam. In this example, the call could have the confidence levels 10, 50, and 90 resulting in a band that represents the student’s Predicted Score with a confidence of 80%.

Predicted Score - Visual Range

Display Predicted Score as a numeric range.

Predicted Score Numeric Range

Display class-wide predicted scores for students on exams, binned into groups. Each student could be binned into a group in several different ways – most simply, using the point value of PS at some confidence (50 for median, 30 for more conservative). Or perhaps you might want to also include a “not enough info” bin if the difference between the score at confidence 80 and score at confidence 20 is too large.

Predicted Score - Class Wide

As an instructor, I want to understand the knowledge state of my class on the contents of an upcoming test so I know what to review in my next session

Display per-student or class-wide “overall course mastery” on LOs (or aggregations of LOs into topics or chapters). One way to achieve this is by using just a single confidence level (50 for median value, 30 to be more conservative) with get_learning_objective_scores set to true, and displaying the scores for each LO averaged across all students. The display below might have clickable “Review Now” buttons.

Predicted Score - Per Student

As a student I want to identify which topics to focus on as I prepare for the test

Request Predicted Score for different groups of learning objectives representing different topics/chapters. Bin the Predicted Score for each group of learning objectives into different buckets (e.g. “Needs Work”, “In Progress”, “Looking Good”).

Predicted Score - By LO

As a student, I want to understand how I might do on this test so I can judge whether I should keep studying

Display the student’s Predicted Score as a range. The distribution curve is created by the app, based on historical data.

Predicted Score - Range Analysis

Imposing Predicted Score Coverage Requirements

The Predicted Score API will return scores regardless of whether any given student has done work on any given LO included in the request. The partner may decide not to show predictions to users unless the student has done some minimum amount of work on a sufficient number of learning objectives across their entire usage of the app. This can be accomplished by specifying a learning objective coverage requirement and making use of learning objective coverage metrics returned with the predicted scores.

Knewton’s Predicted Score API will return a number between 0.0 and 1.0 to describe the proportion of the partner’s specified learning objective coverage requirement a given student has completed on each learning objective. Partners can use this coverage metric to gate learning-objective-level score prediction displays in their app, choosing instead to display an “insufficient data” label or icon, where appropriate.

The Predicted Score API also handles aggregation of the coverage metric across learning objectives in a group or across an exam as a whole: if, for example, a student has covered only 50% of the learning objectives on an exam to an extent sufficient to satisfy the partner’s coverage requirement (either by covering half the relevant learning objectives completely, or by covering all of the relevant learning objectives incompletely), the partner can use the returned overall learning objective coverage of 0.5 to decide whether or not to display the overall predicted score.

Comparison to Other Current Analytics

How is Predicted Score different from Progress?

Predicted Score and Status & Progress work very well together! Predicted Score is meant to give a high-level view of how we think a student would perform if assessed on some learning objectives, right now, given their whole history in a course. Progress is meant to provide a self-contained view of the progress that a student is making towards demonstrating mastery on a particular assignment.

Key differences are:

  • Predicted Score uses the full user history instead of the assignment history.
  • Predicted Score leverages a forgetting curve - if the student does not work on the learning objectives for some time, Predicted Score can fall but Progress will stay the same.
  • Predicted Score does not incorporate the concept of requiring minimum work on the learning objectives in question. Learning objective coverage requirements are for partner use in deciding whether or not to display the resulting score prediction; they do not directly impact the returned score values.
  • Predicted Score provides the ability to request a probabilistic range of values. Progress is reported as a single value.
  • Progress is designed to allow the student to reach 100%. Predicted Score is not designed for students to be expected to reach 100%.
  • Progress is designed to be called repeatedly in an item-by-item flow to provide students with a real-time view of their progress towards completion of an assignment.

Comparison to Deprecated Analytics

How is Predicted Score different from Expected Score?

  • Leverages new proficiency model with item difficulties, graph-structured propagation, and new features capturing the correlation between disparate content. In contrast, the Expected Score endpoint uses a simpler model that does not account for item difficulties and or for the relationship between concepts.
  • Based on learning objectives, not a goal or set of questions. The LOs can be weighed differently and can be grouped into “groups” on the fly.
  • Leverages a forgetting curve - if the student is not doing any work, Predicted Score can fall but Expected Score will stay the same.
  • Has no “estimate status” indicating whether a partner should/shouldn’t expose the estimate to their users.
  • Expected Score has a default value of 0 for unseen concepts/LOs. In contrast, Predicted Score will have non-zero default values for any confidence level you may request, and these defaults are subject to change whenever we tune the model or when the student works on nearby concepts, even if they’ve never worked directly on the LO(s) requested.

How is Predicted Score different from Taxon Proficiency?

  • Leverages new proficiency model. While the proficiency model powering Taxon Proficiency does incorporate item difficulties and graph-structured propagation, the newer model provides better, more-consistent behavior.
  • Based on learning objectives in arbitrary groupings, not a pre-defined taxon.
  • Leverages a forgetting curve - if the student is not doing any work, Predicted Score can fall but Taxon Proficiency will stay the same.
  • Has no “estimate status” indicating whether a partner should/shouldn’t expose the estimate to their users.