The Adaptive Experiment: Implementation

by Colin Fredericks, HarvardX

[You’re reading part 2 of an 4-part series on adaptive learning, a deep dive into the technology and approach deployed in HarvardX’s Super-Earths and Life in 2016-17. You can also read Part 1, Part 3, Part 4.]

My name is Colin Fredericks. I’m a Senior Project Lead at HarvardX, which means I help Harvard professors create online courses like Super-Earths. If you’d like to see some other courses I’ve worked on, check out Energy or John Snow and Cholera.

I’m going to get into the technical details of how we implemented an adaptive learning system within edX. If you’re not intimately familiar with the edX platform, that’s going to get kind of boring, so here’s the takeaway for non-experts:

  1. EdX is not currently designed to support adaptivity. There was a lot of programming work that needed to be done, not just on my end, but by the VPAL technical team and TutorGen, in order to make this solution work.
  2. Because of that, our solution will only work for technically-oriented teams. This is not a plug-and-play approach.
  3. Nevertheless, we successfully provided adaptive content, reliably and flexibly delivering material based on students’ performance.

Let’s dive into the details.

Overview

In general, to create an adaptive learning system, you need four things:

  1. More Materials. We roughly tripled the amount of material in the subsections where adaptive learning was deployed. It would be nice to have even more.
  2. Knowledge of the Student State. You can’t provide the “right” materials to the right students without knowing something about the students.
  3. An Adaptive Strategy. So many options. Do you ramp up slowly, or start hard and pull back? Are you using a truly adaptive approach, or is this a choose-your-own-adventure? In Super-Earths, TutorGen gave us our strategy in the form of an algorithm based on Bayesian Knowledge Tracing. They’ll give more details on that in their blog post.
  4. Materials Delivery. Once you’ve decided what materials to show, you need a way to get them to the students. This was accomplished via an LTI tool created by VPAL. Andrew Ang from VPAL will provide more details on this in his post, but I’ll touch on this a little bit here because the way we provided the materials to the LTI tool was unique to edX.

Here’s a simplified view of what happens in our system in particular:

basic adaptive diagram

The adaptive tool pulls homework problems (and sometimes other materials) from a list of activities and sends them to the edX system. Learners work on these activities and receive grades through edX. When they submit a problem, data from that submission is sent to the adaptive learning engine, where the algorithm adjusts the activity list based on how well the student is doing.

If you’re an edX expert, you’re probably seeing a couple of roadblocks here. First, there’s no way to do any of this purely within edX. You need an external tool to keep the list, collect and analyze the data, and alter what problems students see. Second, the edX system has no way to pass information about student performance to outside systems. Here’s how we solved those problems.

The LTI Tool

The VPAL LTI tool handles several tasks (see Andrew Ang’s blog for more details). It receives learner activity data from edX, passes a sanitized version to TutorGen’s SCALE® (Student Centered Adaptive Learning Engine), receives updates from SCALE, and provides the appropriate next activity to our learners.

We could have built this as an XBlock instead of an LTI tool, but LTI is the clear winner for Harvard. We use Canvas for on-campus courses, and it’s important that we end up with a tool that we can use in both platforms.

Assessments in edX have a multitude of useful features. Trying to replicate that in an external tool would require substantial coding. Therefore, we instead decided to use edX’s XBlock URL functionality. Every assessment and web page that the students see within the LTI tool is a component served from within the edX platform. In fact, they are served from within the exact same course. The LTI tool doesn’t serve the assessments, just a frame with some UI niceties, like keeping a list of previous activities for students to go back and review. Within that frame is another frame that’s essentially a pass-through from edX back to edX. This allows us to reset and regrade faulty problems, view individual student assessments, and more, because all the assessments are kept within the platform.

This pass-through approach was essential for retaining the quality of the assessments, but it did (and still does) present a challenge when it comes to experimental design. We considered it important that the students in our experimental group would not see the assessments in multiple locations - that they would only able to access them via the adaptive tool. Therefore, we had to hide the assessments from them in the course and in their Progress page, but in a way that still allowed us to deliver them.

Hiding the Assessments

Several methods were considered and rejected:

  • Cohorts. We tried creating a zero-student cohort to hold the material. At the time we were implementing this, XBlock URLs didn’t deliver cohorted content regardless of which cohort the viewer was in. Now they deliver it to learners in the appropriate cohort, which is still no good for this particular purpose.
  • Staff-only. Creating a staff-only section to hold the material will hide it from the learner’s view in edX, but won’t prevent it from appearing on their Progress page. Ditto for simply hiding the problems with javascript.
  • Release date. Creating a section with a release date far in the future hides the assessments from the Progress page. Sadly, it also prevents the assessments from being viewed via XBlock URL, because they’re not yet released.
  • Open edX: Serving the assessments from an Open edX instance would require some way to synchronize login information across the two instances, not to mention the general headache of running the platform in the first place.
  • Conditionals. Conditional blocks work for this purpose, but there is no support for them in Studio, making course development substantially more difficult and error-prone. We might have still tried this approach had we not found a different solution first.

A/B Testing was the answer. It turned out that the best place to hide the assessments from the experimental group was within the control group’s assignment.

The diagram below shows the approach in more detail:

ab diagramAll of the problems for the adaptive subsections are within an A/B test component (in edX parlance, a Content Experiment). In fact, all of the problems are kept only in the control group’s side. Learners in the experimental group see just the LTI tool. XBlock URLs don’t care what A/B group the learner is in, so we can still provide the assessments in that manner. However, grading in edX does care about A/B testing status, so students in the experimental group will only see the grade that comes from the LTI tool.

If that sounds like a fairly narrow path to tread, it is. This is a substantial constraint on the research design space. Our approach doesn’t allow for the control group not to see any of the assessments. Every last problem that we want to serve to the experimental group must be available to the control group somewhere.  (In our case, they’re all visible in the subsections marked “Extra”.) Though there may be benefits to this approach, we currently lack the ability to create alternative approaches and compare them. It would be great to have other options, and I’m looking forward to the day that Studio supports Conditional blocks.

Passing Data

As previously mentioned, edX has no way to pass data to external systems. There’s no API to call to get student performance data, either in aggregate or individually. What we ended up creating was, to be generous, a bit of a hack.

We added a bit of javascript to every problem in the course, called from Files & Uploads. First, this script gets the problem’s unique ID, both for tracking which problems the learner does, and to make sure that it reports only its own data, and not the data for every problem on the page. Then it scrapes the page for the learner’s username, the grade from the last submission, and the problem’s maximum grade. All of this is posted securely to the LTI tool.

This javascript is called when a problem reloads after the “Check” button is clicked. Since there is no specific event triggered on reload, we used a sophisticated technique we call “waiting 2000 miliseconds after the click”. (I told you that “a bit of a hack” was being generous.) There’s clearly room for some improvement in this approach, but without an API or some additional Javascript support, this is the best we can do.

Final Notes: The Instructor’s View

Enabling adaptivity in your course increases your workload. It’s a substantial amount more. We didn’t even try to do anything with video - there wasn’t enough time. Don’t fool yourself by saying “Oh, we already did the hard stuff, this is just adding more content” because content creation is the hard part. Course design is fun. Implementation is boring. Boring is hard.

My colleagues will discuss some of the initial findings from this experiment in other posts. However, I’ll point out one great thing here before I finish: our students didn’t notice. No one came to the discussions and said “I think this homework is weird,” or, “This doesn’t look like edX,” or, “Where are these problems coming from?” As far as the students were concerned, this new material fit right into the course. That’s great because the last thing we wanted was for learners to be influenced by the details of our implementation. Invisible implementation is a definite win.

Our next goal is to expand this implementation to the rest of Super-Earths. I’m also very interested to see what we could do with a course that has adaptivity planned from the beginning.

I would like to thank my colleagues at VPAL and TutorGen for their hard work on all this, as well as Dr. Alessandro Massarotti, who wrote much of our adaptive content, and Dr. Dimitar Sasselov, whose enthusiasm for this project never wavered.

 

Disclaimer: Responsibility for the information and views expressed in this blog lies entirely with the author and are NOT endorsed in any way by organizations mentioned in the blog.