Rollback Strategies and Deployment Safety

Why Rollback Matters

Every deployment carries risk. A rollback is the controlled act of returning a production system to a previously known-good state after a bad release is detected. The goal is to restore service quickly and safely, before users or business operations suffer lasting harm. Experienced developers treat rollback not as a sign of failure but as a planned, rehearsed procedure — just like a fire drill.

In a professional environment, the time between detecting a bad release and completing a rollback is measured and tracked. A common target is to restore service within ten minutes of detection. Meeting that target requires preparation: you must know your commands, your trigger criteria, and your steps before an incident happens.

Detecting a Bad Release

Before you can roll back, you must recognize something is wrong. The primary tools are your application logs and a smoke test.

Reading logs on a Linux VPS — If your application runs under systemd, check its status and recent log lines:

```

sudo systemctl status your-app.service

sudo journalctl -u your-app.service -n 50

```

For web servers such as Nginx or Apache, check the error log in `/var/log/nginx/error.log` or `/var/log/apache2/error.log`. Look for stack traces, HTTP 500 responses, import errors, or crash loops. These are common signals of a broken release.

Smoke test — A smoke test is a minimal check that the application is responding correctly after a deploy. At the simplest level this is a `curl` command that checks for a 200 HTTP status:

```

curl -I http://your-server-ip

```

A smoke test should run automatically immediately after every deployment.

Rollback with Git

If your application is deployed directly from a Git repository on the VPS, you have two main options.

Option 1 — `git checkout` to a previous commit (detached HEAD)

Use this when you need to quickly return the working files to a specific earlier state without adding a new commit to history:

```

git log --oneline -5 # identify the last good commit hash

git checkout <good-commit-hash>

sudo systemctl restart your-app.service

```

Be aware that detached HEAD is a temporary state. After stabilizing the system, you should create a branch or revert properly.

Option 2 — `git revert` (preferred for shared repositories)

`git revert` creates a new commit that undoes the changes introduced by the bad commit. This preserves full history and is safer when other team members are working in the same repository:

```

git revert <bad-commit-hash> # opens an editor for the commit message

git push origin main

sudo systemctl restart your-app.service

```

After either approach, run your smoke test again to confirm the application is serving correct content.

Rollback Trigger Criteria

A runbook must state clearly when a rollback should be initiated — not left to guesswork under pressure. Common trigger criteria include:

• HTTP 500 error rate exceeds a defined threshold within the first two minutes after deployment
• Application process crashes or enters a restart loop
• Smoke test returns a non-200 status or unexpected content
• A critical user-facing function (login, checkout, data save) is broken
• Deployment takes longer than the expected window and the cause is unknown

Define your triggers before you deploy, not after.

Writing a Deployment Runbook

A runbook is a written, step-by-step procedure document for a specific operational task. It serves as the single source of truth during an incident, when stress makes improvisation dangerous. A professional deployment runbook contains these sections:

1. Pre-deployment checklist — backups verified, tests passing, team notified, maintenance window opened.

2. Deploy steps — exact commands in order, with expected output noted.

3. Smoke test steps — commands and expected results that confirm success.

4. Rollback trigger criteria — specific, measurable conditions that require a rollback.

5. Rollback procedure — exact commands to return to the previous release and restart services.

Write commands that are specific to your stack. A runbook with vague steps like "restart the server" is far less useful than one that says `sudo systemctl restart myapp.service`.

Safety Considerations

Deployment safety on a shared VPS overlaps with general system administration safety:

• Never perform a production deployment without a backup. Verify backups complete before you begin.
• Work in a non-root account with `sudo` only when required. Avoid running your entire workflow as root; a typo with root privileges can destroy a system.
• Use version control for all configuration files — not just application code — so infrastructure changes are reversible.
• Communicate. In a team setting, announce deployments and rollbacks in a shared channel so no one is working against you.
• Document every rollback. After the incident, record what failed, what triggered the rollback, how long it took, and how to prevent recurrence. This is called a post-mortem or incident report and is standard professional practice.

Connecting to Your Credentials

The CompTIA Linux+ exam tests your ability to manage services with systemd, read log files, and perform administrative tasks on Linux servers — all skills exercised in this lesson. Git version control and deployment procedures also appear in professional development workflows assessed in broader IT certification contexts. Building runbook habits now prepares you for the operational expectations of any development or DevOps role.