Every industry has its best practices, and we follow them for a simple reason: to walk the path already forged and successfully tested by others. In this blog post, I will describe some of the practices we employ to reduce maintenance and ensure maximum reliability in our hosting of Open edX instances.
Closely following Open edX named releases, which have release cycles of about six months, is an excellent goal. These named releases also receive upgrades in the form of point releases, which have a simpler upgrade path due to fewer radical changes.
The reason these releases are a good target is that they are fully supported by Open edX. The features and security of the latest named release is always improving, while older versions are not supported, don’t receive bug fixes, and develop vulnerability patches. This is why operators that don’t want to upgrade their instances always have to backport code from newer releases.
One difficulty that might arise when upgrading an Open edX instance is merging conflicts. Any developer trying to merge a large number of changes knows how unwieldy this can be, as the number of changes increases. This, therefore, is a good reason to get rid of code drift.
Closely following the upstream code and minimizing code drift makes the upgrade path much easier when a new named release or point release lands. However, if introducing code drift is unavoidable, it’s better to contribute to it further upstream and get it as part of a new release. We also think one of the best reasons is to give back to the community built around the platform.
At OpenCraft, we closely monitor Open edX releases and upgrade our clients’ instances as soon as possible, with plans starting as soon as the new version is announced This is not possible without getting rid of code drift. This is why we aim to upstream all our code changes to Open edX.
Not only does this procedure benefit the community, but it also directly benefits our company, by reducing the upgrade burden, and our clients, by offering them faster and more affordable upgrades.
Nothing is as frustrating as going through your online course just to stumble into an issue in your LMS.
Yet, maybe more frustrating is being an Open edX provider and getting a message from one of your learners trying to describe an issue. The easiest way to prevent this situation is finding issues before your learners. Here are three strategies for that.
We strongly recommend deploying a staging environment, which is a safe place to test changes and let them break. A staging instance representative of production is a good place to catch issues that might have been missed during development.
One gotcha here is having a “reduced” representation of the Production environment. For example, you might not set up SSO, or MongoDB failover on the staging instance. This can be dangerous, as you might be deploying a breaking change into production fully believing everything will work as intended. We therefore recommend having a staging environment that is as close to Production as possible.
After deploying changes to the platform, ensure that you test the whole instance. While we might think we can test only parts of the platform touched by a code change, correctly finding out what is touched by the change is actually very hard in a platform with the size of the Open edX LMS. To help with testing, OpenCraft developed a Manual Instance Checklist.
Monitoring is an automated way to identify issues. Strategies for this abound, and many services can be used: hardware monitoring, VM monitoring, process monitoring, performance monitoring, endpoint monitoring, etc. This topic will be more deeply investigated in a future blog post; stay tuned!
One gotcha is having “too much noise” in your monitoring. A monitoring plan that’s too strict and with frequent alerts can lead to desensitization to alerts. This can be dangerous, and can, in turn, create issues that are not checked, and if an alert doesn’t get checked, it’s a waste of time and effort.
Anything can fail without warning on the Internet, including servers managed by others. One important way of making your Open edX instance more resilient is increasing its redundancy by ensuring that critical system components have another identical component with the same data that can take over in case of failure.
I won’t get into the details of MySQL and MongoDB replicas and failover as you’ll find some great writing on the subject. For example, check this article.
Here, I specifically want to bring attention to the importance of getting rid of single points of failure: single database server, single load balancer, single application server, etc.
This is also interesting from a performance perspective, being used for horizontal scalability.
You can have all the patches in the world, but without encryption, an attack on your infrastructure can reveal everything anyway. This is why encryption is necessary if you are hosting anything, especially the Open edX platform, as it stores Personally Identifiable Information.
You can find more information on encrypting your Amazon S3 Object Storage, for example, in our previous blog post.
Besides encryption, following Web applications security practices is very important to prevent headaches in the future. This is another topic that is vast and has been already explored in articles around the internet, and each environment will have its own requirements.
What you should look for here is reducing your data and application servers’ exposure to a minimum, while hardening the servers that are required to be exposed, such as bastion hosts and load balancers.
At OpenCraft we host and maintain OpenEdX instances for many organizations including Harvard University, Arizona State University (ASU), Cloudera, and many more. If you’d like to avoid the headache and have your instance professionally managed, reach out to us.
Cover Photo by Markus Spiske on Unsplash
We’ve consciously built OpenCraft on four values: Openness, quality, commitment and empathy. When it comes to quality, especially when it comes to ensuring the best privacy, security and flexibility for our users, we don’t do half-measures.
We work as hard as we can to ensure that our clients get the best possible Open edX development we can deliver. And that commitment doesn’t stop with custom code development. We also host instances of Open edX for our clients, and each of them is logically separate from all the others.
We only host "dedicated" instances of Open edX, which means that each instance that we deploy belongs to a single client. At the entry-level price range, other vendors offer hosting on Open edX sites, which are hosted on a shared instance along with other clients.
With a shared instance, multiple clients share the same resources, from the database to APIs. When you have a dedicated instance, the server belongs to you and you alone, and ownership confers many advantages. Let’s go through the most important of them: security, scalability and performance, data portability and control, and flexibility and customization.
Our instances are hosted on highly automated and robust cloud infrastructure, with a design and deployment strategy that has been optimized specifically for hosting Open edX, following industry-leading practices for performance and security.
With a dedicated instance, our clients never have to worry about performance lagging because another client is hosting a class with hundreds of simultaneous users and it’s hogging more than its fair share of resources.
If you need to scale up on a shared instance, it can be done, but it's impossible to predict when multiple users of the same site might have a lot of traffic at the same time, which means shared instances aren't scaled to handle the full load for each individual site they contain. In a nutshell, you end up with less available resources than you’ll probably need to handle traffic spikes. With a dedicated instance, you’ll always be able to scale to your own needs, so you can have as many courses and users as you like.
Security is also stronger with a dedicated instance. Security flaws or admin errors on one site on a shared instance can affect everyone else. For example, admins on a shared site could elevate their privileges on other sites from the same instance, which opens up all kinds of unpleasant possibilities. And if one XBlock on a shared instance has a security hole, everyone whose site is on that instance is vulnerable. By controlling your own instance, you don’t run the risk of suffering the consequences of your “roommates’” mistakes.
If you’re on a dedicated instance, the data belongs to you, and it’s easy to extract or migrate it to another host. But that’s a very tricky task if you’re on a shared instance, because you share a database with every other site on that server. Your data is all mixed up with everyone else, and extricating yours from the shared database will require a ton of expensive, time-consuming work. It’s the ultimate form of vendor lock-in. Once your site is on a shared instance, it’s very unlikely you’ll be able to leave, and if you try, you’ll be in for a terrible case of sticker shock when the bill arrives.
On a dedicated instance, you have complete administrative control, which can save you a lot of headaches. You have direct access to the Django admin console and to the database and APIs, a level of access that’s very restricted on shared instances. If you want to make even minor changes to your site on a shared instance, often the only way to get it done is to make a support request. A task that could take five minutes for you to complete on a dedicated instance takes a day or two because your ticket has to wait in the queue for an available admin. More than once we’ve had clients come to us because they were tired of having to ask their vendor to handle issues they could have easily handled themselves.
At OpenCraft, customization is our bread and butter. We eat custom code for breakfast, so this advantage is close to our heart. When you’ve got a dedicated instance, you're in full control of your customizations. If you’re sharing an instance, however, the host has to be very careful about how customizations are done and the dependencies of components, because changes to one site can have repercussions for everyone else. So if you want further customization of optional functionality like XBlocks, ecommerce or single sign-on, you’re going to face some challenges in a shared instance.
This inflexibility carries over even into areas like localization. While it’s possible to install additional languages on a shared instance, it could have technical implications for other sites, especially if you’re dealing with languages such as Arabic or Hebrew that are read from right to left. For us, installing a new language on a dedicated instance is about one hour of work.
Finally, by providing dedicated instances, we’re able to ensure reliability. Maintenance and upgrades are included with all our hosting plans, and we have a track record of exceptional, 99% availability. We also create daily backups for every instance and can restore backups quickly and easily.
So, while it’s a bit more costly for us to provide our clients with dedicated instances, we think it’s worth it in the long run. Stronger security, fast performance, full control, scalability and as much customization as you want —these are important to anyone who’s running an Open edX instance. Besides, if you want to provide a high quality experience, you can’t take half measures.
Photo by Paul Gilmore on Unsplash
