The Bridge for Adaptivity: Enabling the Use of Adaptive Assessments in edX

Andrew Ang, Harvard University

[You’re reading part 3 of an 4-part series on adaptive learning, a deep dive into the technology and approach deployed in HarvardX’s Super-Earths and Life in 2016-17. You can also read Part 1, Part 2, Part 4.]

It was late August 2016 - about 2 months before the planned launch date of Super-Earths course: We had an edX course with additional course content being created for the adaptive pilot study. We had an adaptive engine that would determine what content students should see next based on their prior activity. But we also had a problem: how exactly do we use them together? How would we display edX content in a dynamically determined sequence?

To enable the use of an adaptive engine in an edX course, we developed the Bridge for Adaptivity (BFA). BFA is a web application that acts as the interface between the edX course platform and the Tutorgen SCALE (Student Centered Adaptive Learning Engine) system, and handles the display of problems recommended by the adaptive engine. For more on SCALE see Part 1.

Functional Requirements

After a bit of brainstorming, we decided on some functional requirements for what such a web application would need to do:

  1. The tool would have to be able to be embedded in the course content and would need to have some idea of who the edX user was, in order to differentiate between students and pass back grades.
  2. Take in students’ problem responses from edX in real-time, then relay that data to the adaptive engine via API.
  3. Display edX content based on a real-time recommendation from the adaptive engine.
  4. Basic UI functionality for moving on the the next problem, and navigation between previously seen problems.
  5. Collect and store student and problem data that can be used for research and analytics.
  6. Deployed in a way that handles traffic from thousands of users, with minimal downtime.

Integrating with edX via LTI

Luckily, the first set of requirements - user integration with edX - outlined a use case we were already experienced with, using LTI provider functionality.

The LTI (Learning Tool Interoperability) specification allows external web applications to integrate with a learning management system (LMS) such as edX or Canvas. Typically LTI tools are launched as iframes embedded inline with other course content. The LTI specification outlines the type and format of data (called “LTI parameters”) that can be sent with the launch - these include information for authenticating the request, and an anonymized user id associated with the LMS user account, that can be used to identify the user across multiple distinct launches.

Probably the most important problem LTI solves in this context is, “how do you tell who the learner is, when they access the LTI tool?”  By parsing the LTI parameters the tool provider can identify the user and their attributes, and thus customize the experience to the specific user.

For more information on the LTI standard and LTI apps, Edu App Center has an excellent overview of LTI and a repository of LTI apps.

LTIOften, LTI tools are used in courses so that advanced content - that might not be implementable in the course platform (e.g. custom problem types, interactive activities) - can be seamlessly included alongside the the rest of the course material. We had, however, a slightly unusual use case where even though we were launching out of the edX course, we actually still wanted to serve edX content - just that we wanted to serve the content in an adaptive sequence. This was accomplished with an interesting nested setup where edX iframes in the LTI tool, which in turn iframes in a edX XBlock (a link directly to standalone problem or piece of content). See Part 2 for more on why it was important to serve content directly from edX, and how the adaptive content was set up in the course.

xblock

Implementing the LTI portion of the app was done with the help of a couple open-source libraries:

Executing adaptivity: Bridging edX and the adaptive engine

A key challenge was to come up with a way to get student problem submission data from edX in real-time.

Why was this needed? To realize adaptive assessment, we want the next question a student sees to depend on the answer they might have submitted just a second ago.

Why was this a challenge? While the edX XBlock may look like it is inside the LTI window, it is still a separate system and the problem data is going to the edX backend, not our LTI tool. While we also collect data from the edX backend that we use for other research projects, we only are able to get data updates weekly, which is not sufficient to inform real-time adaptivity.

Our solution was to embed a piece of Javascript inside every problem in the course, that sends data about the student’s response (student, problem, and score) to an API endpoint implemented on the Bridge LTI tool whenever a problem submission is made. See Part 2 for more details about how the Javascript was implemented in the course.

On the other end of the bridge was the interface with SCALE. Data had to be passed to the adaptive engine, so that the engine has the appropriate information to make its recommendations when queried. To make this possible, Tutorgen implemented two important API endpoints to SCALE.

  1. Transaction: submit student problem attempts by making a web request with student, activity, and grade data.
  2. Activity: get an activity recommendation for a student, by sending a web request with a student ID. The returned response contains an ID representing the activity recommended by the engine for the student.

With these interfaces to both edX and Tutorgen in place, the application executes the following sequence of actions, in order to act as the adaptivity bridge between these two platforms:

integrationWhen the student submits an answer in the edX activity window:

  1. Embedded Javascript in the edX content sends data about the learner and their response to BFA.
  2. This data is then processed and sent to SCALE.

When a student clicks the “next” button on the navigation bar:

  1. BFA makes a query to SCALE for the next recommended activity for that learner.
  2. SCALE sends a response back with an activity id representing the next activity the learner should do.
  3. BFA translates the activity id into a XBlock URL.
  4. BFA loads that activity in the activity window by iframing in the XBlock.

Toolbar and navigation features

The user interface seen by a student when they encounter an adaptive LTI assignment in the course is shown below:

user viewProblems from the edX course are displayed one at a time in a center activity window, with a surrounding toolbar that provides the following features:

  1. Activity sequence navigation (top left). This is a dynamically generated sequence of buttons that grows as the student progresses through their individually determined activity sequence, and allows the student to go back to previous problems for reference or to make additional attempts.
  2. Score display (top right). The score that appears here is the total number of points the student has earned in the activity sequence.
  3. Shareable link (bottom left). The XBlock URL of the current activity was provided so that if students posted to the forum with a question, they could include the link to the specific activity for reference. We actually saw an increase in forum activity in the experimental group that had access to the adaptive assignments, which might be related to students being more encouraged to use the link to share on the forums for help. See more on Part 4.
  4. Next question button (bottom right). If the student is on the last problem in their current sequence, this triggers the adaptive serving of a new problem. (otherwise this brings the student to the next problem in their current sequence)

The toolbar was designed to be unobtrusive and to blend in with the rest of the content (helped by using Twitter Bootstrap for the visual/CSS styles), yet still providing the essential set of tools in an intuitive way.

Data schema

Because this tool was designed with research in mind, special attention was paid to collecting the right kinds of data and ensuring that we would be able to connect student data collected by the application back to student data that we collect independently from edX, while also designing a data schema that made sense for the application, and supported the needs of the UI.

Diagrammed below is a Entity-Relationship Diagram for the underlying data schema of the application.  Each object corresponds to a table in the PostgreSQL database used by the web application:

Entity-Relationship DiagramEach table corresponds to an object type in the web application:

  • Module: Place in the course where an LTI assignment is embedded. In our pilot study, there were 4 in total.
  • Activity: A piece of edX content - this could be a problem or an HTML page. Each activity has a URL attribute, which stores the (XBlock) URL of the content.
  • UserModule: A particular student’s LTI window instance. This has a one-to-one relationship with LtiParameters, which stores the LTI parameters that were passed to the tool when the tool was launched from the LMS.
  • SequenceItems: Each UserModule contains a sequence of ordered SequenceItems. Each has an “order” attribute corresponding to the index of the item’s position in the sequence, and an “activity” attribute containing an identifier for the activity that was served to the student.
  • Attempt: A student’s attempt at a problem. SequenceItems can have multiple Attempts associated with them, and new Attempt database entries are created whenever a Javascript web request is sent by an edX problem to the Bridge for Adaptivity.

Web application deployment on AWS

The Bridge for Adaptivity web application was developed using Django, a popular web framework for Python. The Django web application code is available on Github, at https://github.com/harvard-vpal/bridge-adaptivity. We used Elastic Beanstalk, an orchestration service offered by Amazon Web Services, for application deployment and monitoring. Specific AWS resources used in production include:

  • Database: 1x RDS, running PostgreSQL, with restricted network access.
  • Web servers: 2 - 4x EC2 instances. These are set up in an Autoscaling Group that scales from 2 to 4 instances depending on network traffic. Web server instances have network access to the database.
  • Load Balancer: 1x Elastic Load Balancer. The load balancer routes web requests to one of the web server instances.

Our infrastructure architecture is shown below:

infrastructure architecture Because the number of web servers auto-scales according to web traffic, the production instance is well-equipped to handle spikes in usage - such as the rush of activity right after course launch. A minimum of 2 web servers are in operation at all times, making the application tolerant to individual web server failures.

Data analysis and results

As the course is wrapping up, we are already analyzing the data collected, and seeing some very interesting results from our pilot study. Check out Part 4.

 

Disclaimer: Responsibility for the information and views expressed in this blog lies entirely with the author and are NOT endorsed in any way by organizations mentioned in the blog.