A risky business

When releasing a change or new feature, there is always a risk of something going wrong. People make mistakes, and we at Easy LMS are no exception. How we handle those risks makes all the difference.

Koen
Back-End Engineer
Posted on
Updated on
Reading time 8 minutes

“Are you ready?”

Ready as I can be, I think as I go through my mental checklist one final time.

Pilot: Shoelaces, leg straps, chest strap, gloves, radio, chin strap, check.

Lines: All free and unobstructed, left over right so I can turn to my preferred side, check.

Wing: The leading edge is open and spread out nicely in a horse-shoe shape, check.

Wind: A light breeze is coming up, a bit too weak for a reverse launch. I decide to wait it out for a few minutes.

“Pick your moment.”

I nod to the instructor as I watch the wind vane. After another minute, I feel the wind picking up a bit. The wind vane is looking good. I wait a few seconds to see if it lasts. It does.

Wind: check.

Airspace: No other gliders flying near launch and no other gliders ready for launch, check.

I take a deep breath... “START!”, I pull the A-risers towards me. The wing inflates as it rises. I let go of the risers and pull the brakes to bring it to a stop above me. The shape of the wing looks familiar, good. There are no knots visible, excellent. A gust of wind turns the wing slightly. It is no longer centered above me. Should I abort? I apply some brake to point the wing towards the front and take two steps to bring myself under the wing again. The wing settles. I keep the wing in this position for a second or two to verify I am in control. With the wing settled, all signs are now green. While keeping constant pressure on the brakes, I turn myself 180 degrees. After a short sprint, I feel the leg straps tighten as the wing pulls me off the ground.

I'm flying.

You have guessed it; I have a risky hobby. That means I am a risk-taker, right? Well, yes, sort of. Also, no! As a paraglider, I spend a significant amount of energy in mitigating risks. With solid preparation and keeping healthy margins, I bring the risk down to an acceptable level. Therefore, a better term is risk manager.

Check out our vacancies

No risk, no business

This brings me to the subject at hand, risk management in software development. More specifically, how do we manage our risks so that we can confidently deploy improvements without stress? After all, deploying software is kind of similar to a paragliding launch. Both are likely moments for risks to materialize into actual problems.

Unfortunately, the only way to eliminate all risks is by avoiding them entirely. Stop paragliding or, equivalently, stop the deployment of software improvements. The first is actually a valid choice. The second, well, some risks we will just have to live with. But we can still reduce risk, so how do we do that at Easy LMS?

Unfortunately, the only way to eliminate all risks is by avoiding them entirely

What even is risk?

A risk is an event with a probability of occurring and a particular (negative) impact. The combination of probability and impact determines how high a risk is. So an event that is very likely to happen and has a great (negative) impact is a high risk. An event that is unlikely to happen and has a very limited impact is a low risk. If the probability and/or impact gets higher, the risk increases, and vice versa.

Managing risk

Since risk is a combination of probability and impact, it follows that to reduce a risk, you can:

  • Lower its probability of occurring (make sure it happens less often).
  • Lower the impact (make sure it is not a big deal if it happens).
  • Do both.

For example, as a paraglider, let's consider the risk I crash because I launch with a nasty knot in my lines. Among other measures, I reduce this risk by:

  • Always following the same launch procedure (reducing the probability).
  • Wearing a helmet (reducing the impact).

Combining all measures provides me with enough confidence to run happily down towards the mountain's edge. I trust my preparation, skills, equipment, and decision-making. The risk is still there, but I have reduced it to a level I'm willing to accept. When paragliding, the launch will always be a tense moment for me. It requires me to be alert. But I'm not stressed.

Similarly, we have measures in place at Easy LMS to manage risks and keep us calm. Let us consider the risk of introducing bugs into the system when releasing a new feature. Two ways in which we minimize this risk are:

  • Practicing Test Driven Development (TDD)
  • Work in small iterations
We have measures in place at Easy LMS to manage risks and keep us calm

So how do these measures help us at Easy LMS?

Test Driven Development (TDD)

Test Driven Development, or TDD, is the practice of writing an automated test for the desired behavior of a new functionality before writing the code for the functionality itself. TDD helps in reducing the risk we introduce bugs in the following ways:

  • By testing the behavior in automatic tests mistakes are more likely to be caught during development.
  • Mistakes are caught earlier in the development process by writing tests before writing code.
  • Writing tests before writing the code forces the developer to think about how to test the desired behavior independent of the final code.
  • Writing testable code is a bit like writing a readable book. It forces the developer to apply structure to the code. As a result, testable code tends to be maintainable code.
  • By religiously following TDD, we build up a test suite over time, comprising all tests written in the past. Before deployment, we run all these tests. If our new functionality causes a problem with existing functionality, we will know.

In these ways, TDD helps to greatly reduce the probability of introducing bugs in the system.

Small iterations

At Easy LMS, we work in small iterations, releasing small changes often. We often perform one or two deployments per day. We do this in small steps, even when introducing a big piece of functionality. This helps us in the following way:

  • We receive feedback from clients much earlier than if we were to wait with deployment until a feature is completely 'finished'. Based on the early feedback, we can easily change direction if required.
  • It is much easier to recover from a mistake if the difference with the last stable version is small.
  • Short iterations are easier to plan and manage. A team simply cannot fall behind schedule for a significant amount of time. This leads to less stress and prevents teams from feeling the need to cut corners.
  • Deploying daily creates a strong drive toward a more streamlined and reliable deployment process. If not, we would feel the pain daily.

In these ways, short iterations help reduce the probability and impact of bugs being introduced in the system.

TDD and working in small iterations give us confidence. We are strengthened in our belief that what we are about to deploy works. If there are any issues, these will be small.

Short iterations are easier to plan and manage

Unit tests are green, check.

I move the ticket to 'In QA Review, ' which automatically triggers a deployment to our staging environment.
A few days earlier, we deployed a small change in the statuses shown in the participant overview in several places in the app. The change was supposed to make the statuses more accurately reflect the status of a participant.

It might have been more accurate, but it was also confusing. Clients who looked at the new situation through old lenses were under the impression that invitations were not sent out.

Now here I was, ready to deploy the remedy. Based on the feedback we received in the days after the release, we now improved upon our original change. I look at the progress of the automated deployment to our staging environment, where we perform our final tests before deployment.

The acceptance tests turn green, check.

I open the browser and navigate to the staging environment to test the change for a final time. The staging environment is set up in the same way as our live application.

It works as I expect, check.

The change is small, which makes testing easy. The unit tests and acceptance tests further bolster my confidence existing functionality is working as it did before. All lights are green. I trigger the deployment.
I take a sip of my coffee and check my messages while the deployment runs. I think about how the change should help our clients. We will check in with support in a day’s time. We expect that clients will no longer contact support concerning this status. If we get more feedback, I am confident we will have another deploy ready this week to further improve our application and build toward our clients needs.

After a few minutes, the final step in the deploy turns green.

We're flying.

Check out our vacancies