Service Management, Logging, and System Monitoring

Service Management, Logging, and System Monitoring

In a data center, Linux servers run critical services around the clock. Knowing how to start, stop, and diagnose those services — and how to read the logs that explain what went wrong — is a core daily skill for every data center technician.

systemd and systemctl

Most modern Linux distributions (RHEL, CentOS, Debian, Ubuntu) use systemd as their init system. systemd is the first process started by the kernel (PID 1) and is responsible for bringing up all other services in a defined order.

You interact with systemd through the systemctl command. The table below covers the essential subcommands:

| Command | What it does |

|---|---|

| `systemctl status sshd` | Shows current state of the SSH daemon |

| `systemctl start sshd` | Starts the service immediately |

| `systemctl stop sshd` | Stops the service immediately |

| `systemctl restart sshd` | Stops then starts the service |

| `systemctl enable sshd` | Marks the service to start at boot |

| `systemctl disable sshd` | Prevents the service from starting at boot |

When you run `systemctl status sshd`, the output contains three fields you must be able to read:

• Loaded — whether the unit file was found and parsed, and whether it is enabled or disabled for boot.
• Active — the live runtime state: `active (running)`, `inactive (dead)`, `failed`, or `activating`.
• Main PID — the process ID of the service's primary process. If the service is stopped, no PID is shown.

A service can be enabled (starts at boot) but currently inactive, or active (running right now) but disabled (will not survive a reboot). These are independent states — keep that distinction clear.

The System Journal and journalctl

systemd records every service's standard output, standard error, and kernel messages in a structured binary database called the journal. You query it with journalctl.

Useful journalctl options:

• `journalctl -u sshd` — show all journal entries for the sshd unit.
• `journalctl -u sshd -n 50` — show the last 50 lines for sshd.
• `journalctl -u sshd --since "10 minutes ago"` — time-bounded query.
• `journalctl -xe` — show the most recent entries with context; especially useful right after a failure.

The journal is your first stop when a service shows `failed` status. Look for lines containing `error`, `failed`, `permission denied`, or specific port/address conflicts.

Traditional Log Files in /var/log

Before systemd, services wrote logs directly to text files under /var/log. Many services still do both. Key files for a data center technician:

• /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS) — authentication events, SSH login attempts, sudo usage.
• /var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL/CentOS) — general system messages.
• /var/log/kern.log — kernel messages.

Use `grep`, `tail -f`, or `less` to search and follow these files. For example: `grep sshd /var/log/auth.log | tail -20` narrows results quickly during an incident.

Diagnosing a Simulated Service Failure

A common real-world scenario: someone edits `/etc/ssh/sshd_config` and enters an invalid port number or a syntax error. sshd then refuses to start. Your diagnostic workflow:

1. Run `systemctl status sshd` — note `failed` or `inactive (dead)`.

2. Run `journalctl -u sshd -n 30` — look for the specific error message (e.g., `Invalid port` or `error: Bind to port`).

3. Check `/var/log/auth.log` or `/var/log/secure` for corroborating entries.

4. Open the configuration file with a text editor (as root or with sudo), correct the error, and save.

5. Run `systemctl restart sshd` and verify with `systemctl status sshd` that it returns `active (running)`.

6. Document every step in your incident note: time, symptom, root cause, corrective action, verification.

Basic System Monitoring Tools

Before escalating a ticket or opening a change request, a technician takes a snapshot of server health using standard command-line tools:

• top — real-time view of CPU usage, memory, load average, and the top processes consuming resources. Press `q` to quit.
• htop — an enhanced, color-coded version of top; easier to read at a glance (may need to be installed).
• df -h — disk filesystem usage in human-readable form. Flag any filesystem above 80% used as a warning threshold in most data center operations policies.
• free -m — RAM and swap usage in megabytes. High swap usage alongside low free RAM indicates memory pressure.
• uptime — shows how long the server has been running and the load averages for the past 1, 5, and 15 minutes. A load average consistently above the number of CPU cores is a performance concern.

Safety Notes

While this lesson is software-focused, physical safety still applies in the data center:

• Never open a server chassis to reseat hardware without following your site's lockout/tagout (LOTO) procedure and wearing appropriate PPE — even a software service call can turn into a hardware swap.
• Maintain proper cable management when working at a rack; a trip hazard from a loose cable is an OSHA recordable incident.
• Always confirm with your supervisor before restarting a production service — a restart that takes a service offline for even seconds can affect business operations.