Sending Events

Record learner interactions with content items

knAtom Start sending learner events with the Events API reference.


In this section:

What are Events?

Events refer to those learner interactions within an application which Knewton’s models use to calculate recommendations and analytics. The application broadcasts this information to Knewton via the Knewton API. Each learner event references content items described in the content inventory.

There is not a one-to-one mapping of each learner event to a new recommendation. Rather, each event contributes to an overall understanding of the learner’s level of focus, performance, and understanding of instructional and assessment content — all of which informs every Knewton recommendation. For detailed information about how Knewton makes a recommendation, please see the documentation on getting recommendations.

Types of Events

There are two types of events that a partner should send to Knewton:

Graded Events

Applications should send a graded event to Knewton when a learner completes a module correctly or incorrectly (for example, the learner answers a single question on an assignment.)

Applications may send a graded event not just for a module in a summative assessment, but also for any interaction with a module where the application can evaluate a right or wrong answer.

Ungraded Events

Applications should send an ungraded event to Knewton when a learner completes an instructional module that has no assessable outcome (for example, the learner watched a video).

How Knewton Processes Events

When Knewton receives an event from an application, it stores the event in the learner’s history. Knewton then returns an HTTP 204 No Content response. Once the learning application receives the 204 response, the Knewton platform will account for the event when responding to subsequent requests for recommendations and goal-level analytic values.

Sending Events

Each event URI path includes both the registration of the user performing the action and the name of the event. For example, to notify Knewton that a user just finished watching an instructional video, the application might send the following event:

POST /v0/registrations/83ce46e1-ae1e-493a-95ab-826ee02a4118/ungraded-events
{
  "module_id": "mref-30c1337b-4193-4964-82b3-df52dbc8b4d0",
  "interaction_end_time": "2002-10-02T15:00:00Z"
}

Knewton responds to valid events with:

HTTP/1.1 204 No Content

As noted above, while the successful 204 response to a request to submit events does not return any information, an application should wait for the response before making a request for a recommendation or analytic that must reflect the latest events.

In rare cases, an error may occur. Refer to Error Handling to view suggested approaches to handling these errors.

When an application does not require up-to-date analytics or recommendations, it may opt to handle the event response asynchronously. One approach is to hand events to an event dispatcher in a separate thread, which may send events without interrupting the main application flow.

How to Send a Batch of Student Events

The Knewton API allows events occurring within a learning application to be sent individually as they occur (as discussed above), or grouped and sent in a batch. Knewton prefers to receive events as soon as they occur. However, this is not possible for applications that allow students to work offline or where events are generated very rapidly. It also isn’t possible in cases where students can’t rely on having a constant Internet connection. All types of student events can be sent using the batch endpoint. In order to create batches that comply with the endpoint’s requirements, the application needs to have specific logic added for batch formation. A batch must:

  • order events chronologically (oldest event first);
  • not exceed 500 events in the batch; and
  • only include events for a single registration.

The queue of batched events should check that each batch submitted has been successfully accepted before sending the next batch in the queue. Success and error responses for batch events will follow the status codes.

Upon first integrating with the Knewton API, an application might also employ batched events in order to backfill a learner’s history, so that, on first use, the learner will receive appropriate recommendations. An application might schedule such a backfill as an offline job, or in a just-in-time manner, depending on the number of events that it needs to send.

knAtom Check out the batch events API reference here.

When to Send Events

Generally, an application should send events when it knows the complete outcome of a student’s interaction with a module; all information should be gathered and then sent as a single event. An Application should not send events that contain only partial information.

Essay questions, for example, often need to be graded by an instructor rather than the application. Part of the information (the module_id and interaction_end_time) is known immediately, while the result (is_correct) is not known until the essay is graded. Applications that support such questions should wait until the essay is graded, and then send a single event representing the essay information and its grade.

Sometimes it might not be clear when an event should be sent. For example, if a learner starts to view an instructional video but stops midway, should an ungraded event be sent? Or if a student does part of an assessment module correctly, but doesn’t complete the last step, should a graded event be sent? This is a decision for each partner. Instructional videos and assessment modules are examples of atoms — the most granular piece of content which Knewton knows about. For Knewton, an atom is either viewed or not; an assessment item is either completed or not. There is no notion of “partially instructed” or “partially assessed.” The partner must decide whether partial completion is synonymous with completion; if so, an event should be sent. If partial completion is not good enough, then an event should not be sent unless 100% of the atom has been completed.

There is flexibility within the Knewton API for each partner to express its own product vision. For example, one application might require a student to have watched 100% of an instructional video to send an ungraded event. Another application might require that the student watched only 90%. Another application that presented content repeated in various ways, in the same instructional module, might only require 30%. Often, an application is not instrumented to know how much of a video a student has seen, or whether that student has scrolled through all the available text content. For example, if the application has separate “continue” and “skip” buttons, the application might send an ungraded event after the learner clicked “continue,” but not send an event if the learner specifically clicked the “skip” button.

Send An Event When A Question Is Answered

Graded events are the primary way that Knewton gains information about what a learner knows. When a learner correctly answers a question and an event is sent, proficiency models are adjusted and the question becomes less likely to be recommended again. Incorrectly answered questions have the opposite effect: either similar content or prerequisite content is more likely to be recommended again. Every question answered, and so every event sent, provides a better insight into what a learner knows at a specific point of time.

After the learner answers a question and a grade is calculated, the application should send a graded event.

The basic flow when using graded events is:

  1. The application displays a question.
  2. The learner answers it.
  3. The application grades the answer.
  4. The application sends a graded event to Knewton.

The fields in the graded event are:

  • module_id: the ID of the question
  • interaction_end_time: the time step #2 occurred
  • duration: the time between step #1 and step #2
  • is_correct: the result of #3
  • instance_hash: the instance hash should be specified for algorithmic questions, and is a string that is unique among all variants, or “instances”, of the algorithmic question. For example, if the question is adding two numbers, and those two numbers are 2 and 25, then the instance hash could be “2,25” or “2+25”. The partner can choose the best way to generate these unique identifiers. Some common approaches might be:
    • As the name of the field suggests, this identifier could be generated by hashing the dynamically generated portions of the question.
    • If the length of the dynamically generated parts of the question is bounded and relatively small, the parts could simply be concatenated without the need for any hashing.
    • If the variants of the question are actually stored in some system of record, and are given unique IDs in that system, those IDs may be used.

An event should only be sent after the answer can no longer be changed.

Send An Event Only When An Answer Is Final

Some applications allow learners to change their answers. Knewton supports such applications; however, it is important that the model’s interpretation of the event correctly captures the learner’s state of knowledge.

Sending an is_correct=false graded event means the learner thinks the answer is A, but the correct answer is B. Sending such an event when the learner believes she can still change her answer would cause the models to believe the learner has lower proficiency than she actually does. This in turn would negatively affect recommendations and analytics. As in Who Wants to Be a Millionaire, contestants are not punished for answering incorrectly until they confirm “Is that your final answer?”

Tip: The application should not send a graded event until it knows the learner’s “final answer.” At that point, the event should be sent as soon as possible.

In a traditional homework assignment, learners can write their answers in pencil, and frequently go back and revise their answers before handing their work in. While writing their initial answers, they are not overly concerned with correctness: there is a chance to review answers before submitting them. Students are not punished for erasing an answer and replacing it with another. In the same way, graded events should not be sent until the question can no longer be changed.

There is a difference between changing an answer and retrying a question answered incorrectly. Learners who change their answer are simply changing their mind. However, those who retry a question answered it wrong first, and then attempted it again.

Tip: The application should send a single event no matter how many times an answer is changed. It should send a separate event every time an answer is attempted.

The interaction_end_time should be set to the latest time that the question was answered. The application might first save the current time when it is answered; if the learner changes the answer, the value can be replaced with the new time. When all answers can no longer be changed, the most recent saved time can be sent.

How To Calculate An Accurate Event Duration

Tip: Although the field is optional, applications should compute and send duration data in order to ensure the best recommendation quality and accurate values for any metrics that Knewton might offer now or in the future based on these durations.

To calculate the duration, a timer should be started when a learner is first shown the item, and stopped when the item is no longer visible. Actions not directly related to the question, like displaying a help screen or a leaderboard, should pause the timer. However, displaying a calculator or a scratch pad should not pause the timer, since these actions are directly related to the question being answered. The application might decide to pause the timer when a user logs out of the system and resume it when the question is displayed again, especially if work that the user performed is persisted with the question. Duration should include all time spent answering the question.

If some content is viewed multiple times before sending an event, the total of all time spent viewing that item should be sent. However, if an item is viewed on two separate occasions with an event sent for each, the timer should be reset before it is viewed a second time. Finally, note that the duration field is in milliseconds.