This article was written by OpenCraft team member Giovanni Cimolin da Silva.

A good way to assess learners in online courses is to evaluate essay-type text responses through a set of well-defined quantized criteria (called rubrics). 

The Open Response Assessment (ORA) is a component available in the Open edX platform that allows creating free-text input and offers a variety of evaluation methods based on rubrics: peer-graded, staff-graded, or self-assessment. 

In early 2020, edX decided to improve the usability and user experience of the ORA block, as well as implement new features through a blended development project with OpenCraft.

Blended development projects are funded by edX to implement roadmap features and promote collaboration between edX teams and partners in the community while increasing edX’s development throughout. You can find more information about these projects in the Blended Development Runbook.

Here is a summary of the improvements that were made as part of the development project:

Authoring Ease & Usability Improvements

Many changes on this project were aimed at making the ORA authoring experience easier and simpler for instructors. The ORA block is now displayed as a top-level block in Studio, and offers some pre-configured templates.

Animated image showcasing the ORA templates feature.

The ORA configuration menu was improved. The schedule and assessment steps settings were moved to separate tabs to simplify the user experience. The assessment steps tab now allows easy configuration of the evaluation steps and hides unnecessary settings. The schedule tab handles all dates and deadlines settings separately.

New view of ORA settings, showing separate tabs for schedules and assesment steps configuration.

Reporting & Moderation

When peer assessment steps are enabled, it’s common for some students to get stuck waiting indefinitely, especially at the start or end of a course, when there’s not much activity. Students in this state are either waiting for peers to grade their assessment or waiting for assessments to be graded.

One of the new changes is a screen in the instructor dashboard that displays a list of students stuck in the Waiting step and allows instructors to override grades and apply staff grading directly from that screen. 

Animated image showcasing the new Waiting step dashboard functionality, where an instructor can identify stuck learners and apply grade overrides.

You can read more about this feature on Accessing Metrics for ORA Assignments.

Previously, ORA data reports were hard to use since no human-readable user information or block was exported. This left instructors to map students using the exported anonymous user ID manual outside of the platform. On this project, they were improved and now include the real username (instead of only the anonymous ID username), block id, and the readable problem name. This makes it easy to interpret the files, parse submission information, and map this data to actual students.

Additionally, there’s a new report in the Data Downloads tab that allows downloading of all responses submitted by students for offline grading. See the detailed documentation for this report on Accessing Metrics for ORA Assignments

Learner-facing improvements 

Learners will now be able to submit nicely formatted answers through a WYSIWYG editor instead of plaintext submissions. 

See Create an Open Response Assessment Assignment for information on how to set this up on your instance.

Additionally, peer assessment steps (assess peers and waiting for peer assessments) are now non-blocking, allowing students to assess their own response and get a staff grade at any time in the process. 

Final notes

These improvements were developed and merged to the master branch of Open edX: they are partially available in the Koa and fully available in the Lilac release. Check out the Open Response Assessments documentation for details and detailed usage instructions for the new ORA features.

If you want to start using the latest ORA improvements, you can try OpenCraft’s Pro&Teacher offering free for 30 days. You’ll get a fully dedicated Open edX deployment running on the latest release of the platform.