A good way to assess learners in online courses is to evaluate essay-type text responses through a set of well-defined quantized criteria (called rubrics). 

The Open Response Assessment (ORA) is a component available in the Open edX platform that allows creating free-text input and offers a variety of evaluation methods based on rubrics: peer-graded, staff-graded, or self-assessment. 

In early 2020, edX decided to improve the usability and user experience of the ORA block, as well as implement new features through a blended development project with OpenCraft.

Blended development projects are funded by edX to implement roadmap features and promote collaboration between edX teams and partners in the community while increasing edX’s development throughout. You can find more information about these projects in the Blended Development Runbook.

Here is a summary of the improvements that were made as part of the development project:

Authoring Ease & Usability Improvements

Many changes on this project were aimed at making the ORA authoring experience easier and simpler for instructors. The ORA block is now displayed as a top-level block in Studio, and offers some pre-configured templates.

The ORA configuration menu was improved. The schedule and assessment steps settings were moved to separate tabs to simplify the user experience. The assessment steps tab now allows easy configuration of the evaluation steps and hides unnecessary settings. The schedule tab handles all dates and deadlines settings separately.

Reporting & Moderation

When peer assessment steps are enabled, it’s common for some students to get stuck waiting indefinitely, especially at the start or end of a course, when there’s not much activity. Students in this state are either waiting for peers to grade their assessment or waiting for assessments to be graded.

One of the new changes is a screen in the instructor dashboard that displays a list of students stuck in the Waiting step and allows instructors to override grades and apply staff grading directly from that screen. 

You can read more about this feature on Accessing Metrics for ORA Assignments.

Previously, ORA data reports were hard to use since no human-readable user information or block was exported. This left instructors to map students using the exported anonymous user ID manual outside of the platform. On this project, they were improved and now include the real username (instead of only the anonymous ID username), block id, and the readable problem name. This makes it easy to interpret the files, parse submission information, and map this data to actual students.

Additionally, there’s a new report in the Data Downloads tab that allows downloading of all responses submitted by students for offline grading. See the detailed documentation for this report on Accessing Metrics for ORA Assignments. 

Learner-facing improvements 

Learners will now be able to submit nicely formatted answers through a WYSIWYG editor instead of plaintext submissions. 

See Create an Open Response Assessment Assignment for information on how to set this up on your instance.

Additionally, peer assessment steps (assess peers and waiting for peer assessments) are now non-blocking, allowing students to assess their own response and get a staff grade at any time in the process. 

Final notes

These improvements were developed and merged to the master branch of Open edX: they are partially available in the Koa and fully available in the Lilac release. Check out the Open Response Assessments documentation for details and detailed usage instructions for the new ORA features.