Post by Daniel Seaton, Selen Turkay - Harvard University
Since 2012, HarvardX has developed over 38,000 learning assets for its Massive Open Online Courses (MOOCs) on the edX platform, yet, with the exception of a small number of specialized experiments, the use of these resources in residential settings has been significantly limited. Allen & Seaman (2016) assert that two of the most cited obstacles to adoption of open educational resources by university faculty in the U.S. are the difficulty of finding high-quality resources and the lack of a comprehensive catalog of these open educational resources. MOOCs suffer further in that educational assets -- e.g., through edx.org -- are only accessible behind a registration firewall. MOOC learners, course staff, and faculty leads must all register for a course and then discover resources from a student perspective.
To address such challenges, VPAL-Research in collaboration with HUIT and HarvardX have created an application intended to help the Harvard community discover and reuse HarvardX content on campus. Coined Harvard DART (Digital Assets for Reuse in Teaching) -- the project seeks to create a modern search interface for discovering and previewing educational assets from across Harvard and making those assets easily integratable into classroom learning management systems.
DART is built upon a few foundational goals:
Promoting discovery of existing educational assets on campus through search and recommendation.
Providing tools and interfaces for integrating content across platforms.
Surfacing views of course structure and pedagogy that go beyond single-item search.
Although there are numerous use cases for this application, our team’s initial focus has been to make HarvardX resources discoverable through search and deployable through LTI (Learning Technology Interoperability) in Canvas, the on-campus Canvas LMS. Our desired users in this effort are Harvard faculty and course staff assembling Canvas courses; we hope to pilot the application in the Fall 2017.
HarvardX - the organization responsible for Harvard University’s MOOCs - has launched over 100 courses made up of approximately 7,600 problems, 6,500 videos, 10,100 html pages, and 1,000 custom activities employing advanced educational technology. With a catalog that covers essentially all disciplines, and a conscious decision to explore a myriad of course design patterns, HarvardX resources have the potential to be integrated in diverse setting across campus. The amount of content being created also signals a need for tools intended to catalog metadata, provide search, and allow resources to be reused easily in multiple learning management systems.
Using the average number of resources in a HarvardX course (353.9), we estimate the total amount of content generated by edX partners since 2012 to be over 560,000 individual resources. Relying on the total number of edX courses (~ 1600) extracted from the edX Course API, the growth in the number of courses and resources is staggering. For perspective, one can compare the number of resources developed by around 160 partners in the LON-CAPA network: from 1999 to 2013, approximately 446,000 educational assets were part of a shared library of resources.
It is worth noting that LON-CAPA was an early example of the edX platform and consortium model, that is, numerous partner universities sharing a platform and resources for constructing on-campus blended course content. However, sharing of resources between institutions has not yet been addressed by edX from either the platform or resource-sharing perspectives.
DART: Digital Assets for Reuse in Teaching
Toward our goal of making HarvardX resources discoverable on campus at Harvard, we have developed DART: Digital Assets for Reuse in Teaching. The core functionality of this application includes: 1) text search of all HarvardX content (including video transcripts), 2) preview of videos, problems, and HTML pages within the application, and 3) Canvas integration that allows HarvardX content to embedded directly into an existing Canvas course.
The DART application is built around a microservice architecture that gives us the flexibility to experiment with different services (e.g., search and recommendation algorithms) while maintaining a highly-available production environment. By limiting communication between components to a well-defined set of REST APIs, we can evolve and improve the implementation of each service independently, without affecting the other parts of the system. A high-level diagram of the DART architecture can be found on the site’s “About” page (https://dart.harvard.edu/about) along with other insights into the system.
User experience research
User experience research is a priority for the DART team. We conducted repeated in-depth interviews with seven people over the course of the development process: five from HarvardX course development teams and two instructors from the Harvard Graduate School of Education. Each brought different perspectives and viewed the tool through a different lens. HarvardX course developers are excited that their content is cataloged and will be more accessible for future development. One course developer exclaimed that this tool can allow her to find relevant materials from other courses to recommend to learners in her course. Instructors thought that being able to search for high-quality materials is useful but also highlighted the importance of contextualization, i.e. knowing the overall location of content within an entire course. They further emphasized a need to search for materials by broader classification (e.g. videos that include animations) in a specific field (e.g., restrict search to only economics).
In addition, HarvardX course developers emphasized that even though HarvardX has thousands of individual resources and is diverse in subject matter, users of DART need to be aware of the limited amount of content compared to searching the Internet at large. In DART, “search” will not always return a result, an experience people are unfamiliar with due to the extensive use of Google, which always provides some result. To this end, the DART team is exploring methods (e.g., query rewriting, recommendations for low number of results) to deal with this when faced with search terms that result in zero results. This includes potential cross-institutional collaboration opportunities to grow the content for DART.
User experience research is paving the way for future features in DART through targeted A/B tests and solicited feedback from potential users. For instance, at the annual conference of the Harvard Initiative for Learning and Teaching (HILT), we collected feedback via an interactive demo session. We set up five computer stations that allowed conference attendees to interact with the app and provide us feedback verbally and/or through a short survey. Overall, participants found the search interface very clear and the app very promising. Demo users suggested fruitful directions for the project. For example, the potential to include different content sources at Harvard, intellectual property, and Canvas LMS integration all came up during the user experience interviews that we have started to address. We hope the DART application will help accelerate important conversations, particularly since the reuse of digital content is not only a Harvard issue, but an issue facing all of higher education.
We are building the DART application with a flexibility to explore multiple-use cases. Our current focus is on the integration of HarvardX content on campus, but below we postulate potential use cases for both MOOCs and residential courses.
Residentially, the functionality of DART will allow faculty and instructional staff to efficiently add content to their residential courses. This is immediately apparent for HarvardX faculty who have designed courses solely for the edX platform. DART affords those faculty an opportunity to revisit and embed their content for blended and flipped instruction residentially. In addition, faculty are now able to more readily discuss and share their teaching materials with colleagues across campus using DART.
From a MOOC perspective, how would a browsable HarvardX catalog transform our perceptions of learner behavior? The majority of MOOC-related research and media conversations have centered around the dreaded certification rate. The Year-1 report from Harvard and MIT established a category of MOOC learner called “explorers,” individuals who browsed less than half the content before stopping a course. Such users likely exist because people want to browse content before they commit to completion. Providing a venue for browsing outside the course instance would allow learners to make more informed decisions about enrolling. Enrollments may go down, but certification rates would likely increase.
We are excited to explore such opportunities in the near future. Users on the Harvard network can explore search and preview here:
Please tell us about your experiences with the DART. You can do that using the “feedback” button available on the DART interface; we also invite you to contact us for longer conversations. To read more about DART, see the recent feature in the Harvard Gazette.
Andrew Ang, Vittorio Bucchieri, Phil McGachey, Dustin Tingley, and Elliott Yates.
Disclaimer: Responsibility for the information and views expressed in this blog lies entirely with the author and are NOT endorsed in any way by organizations mentioned in the blog.