Error Handling, Exit Codes, and Logging

Why Error Handling Matters in Database Scripts

A Bash script that runs silently and returns a zero exit code tells the operating system and any monitoring tool that everything succeeded. Without explicit error handling, a failed `pg_dump` command may produce an empty or corrupt file while the script continues to completion and reports success. An on-call DBA who trusts that green status at 2 a.m. may not discover the missing backup until a restore is needed — exactly the wrong moment. Robust error handling closes that gap by making failures loud, traceable, and actionable.

The set -euo pipefail Directive

Place `set -euo pipefail` near the top of every database administration script, immediately after the shebang line.

• `-e` (errexit): causes the shell to exit immediately when any command returns a non-zero status, unless that command is part of an `if`, `while`, or `||` / `&&` construct.
• `-u` (nounset): treats any reference to an unset variable as an error and exits. This prevents subtle bugs such as `rm -rf "$BACKUP_DIR/"` expanding to `rm -rf "/"` when `BACKUP_DIR` is accidentally empty.
• `-o pipefail`: makes a pipeline fail if any command in it fails, not just the last one. Without this, `pg_dump mydb | gzip > backup.gz` could silently succeed even if `pg_dump` failed, because `gzip` exits zero.

These three options together form the standard safety net recommended in the PostgreSQL documentation's server-administration section and in every major Bash style guide.

trap and the ERR Signal

The `trap` built-in registers a command or function to run automatically when a specified signal or pseudo-signal occurs.

```bash

trap 'log_msg "ERROR: script failed at line $LINENO"' ERR

```

`ERR` fires whenever a command exits with a non-zero status (subject to the same conditions as `-e`). The `$LINENO` variable expands to the line number where the failure occurred, which is invaluable when reading a log hours later. Combine `trap ERR` with `set -e` so that every unexpected failure is both logged and terminates the script.

A `trap ... EXIT` handler can also perform cleanup — removing temporary files or sending an alert — regardless of whether the script succeeded or failed.

Meaningful Exit Codes

By convention, an exit code of `0` means success; any non-zero value signals failure. Use distinct codes for distinct failure types so that calling scripts, cron monitors, and tools such as Nagios or systemd can branch on the result.

```bash

EXIT_BAD_ARGS=1

EXIT_DB_CONNECT=2

EXIT_DUMP_FAILED=3

```

Document these codes in a comment block at the top of the script. Oracle DBA CA operational-monitoring objectives and Microsoft DP-300 automation topics both emphasize that automated jobs must surface actionable status to monitoring layers — exit codes are the primary mechanism.

The log_msg() Function

A reusable logging function keeps output consistent and searchable:

```bash

log_msg() {

local message="$1"

local ts

ts=$(date -u '+%Y-%m-%dT%H:%M:%SZ') # ISO-8601 UTC

echo "${ts} [$$] ${message}" | tee -a "${LOG_FILE}" \

| logger -t "db_backup" # also sends to syslog

}

```

• ISO-8601 timestamps (e.g., `2025-06-10T02:17:44Z`) sort correctly, are unambiguous across time zones, and are required by many compliance frameworks.
• `tee -a` appends to a flat log file while also passing the line downstream.
• `logger` routes the message into the system journal (viewable with `journalctl -t db_backup`), centralizing it with other system events.

Applying These Techniques to pg_dump

A production backup function might look like this:

```bash

run_backup() {

local db="$1"

log_msg "INFO: starting backup of database '${db}'"

pg_dump -Fc "${db}" > "${BACKUP_DIR}/${db}.dump" \

|| { log_msg "ERROR: pg_dump failed for '${db}'"; exit $EXIT_DUMP_FAILED; }

log_msg "INFO: backup of '${db}' completed successfully"

}

```

When `pg_dump` fails — for example, because the database name is misspelled — the `||` branch fires immediately, logs a timestamped error message, and exits with the designated non-zero code. No corrupt file is silently left behind.

Reading the Log and Responding to Failures

When paged at 2 a.m., an on-call DBA should:

1. Run `journalctl -t db_backup --since "1 hour ago"` to see the syslog entries.

2. Open the flat log file to read the full timestamped sequence of events.

3. Note the line number from `$LINENO` and the exit code to identify exactly which command failed.

4. Check PostgreSQL server logs (`pg_log` or `journalctl -u postgresql`) for accompanying database-level errors.

This workflow directly supports Oracle DBA CA exam objectives around operational monitoring, incident response, and maintaining backup integrity.

Safety Note

Never test destructive failure scenarios — such as intentionally dropping a database — on a production server. Use a dedicated Linux VM or container. Confirm that `BACKUP_DIR` is set before any script run to avoid the nounset trap expanding to an unintended path. Keep log files on a separate volume or rotation schedule so that a verbose script cannot fill the disk and cause its own outage.