Operations: monitoring & backups

How to run a production Diffbridge: what to watch, what's backed up, how to restore, and a go-live checklist. It assumes the single-VM setup from Self-hosting on Oracle Cloud (Docker Compose in /opt/rapidforge); the same steps apply to any Docker host.

#Monitoring

#Health endpoints

Endpoint	Checks	Use it for
GET /healthz	The API answers and reaches the database	Load balancers, the container's Docker HEALTHCHECK
GET /healthz?deep=1	Database, file storage (writes and reads a probe file), and a worker that reported in the last 2 minutes	Uptime monitoring

The deep check answers 200 with {"ok": true, "checks": {…}}, or 503 with the failing check and its error, for example "worker": {"ok": false, "error": "no worker seen for 340 s"}. Results are cached for 30 seconds. It contains no customer data, so it's safe to leave public.

A stopped worker is the failure that's easiest to miss: the app still loads, but deployments, CI runs, backups and scheduled jobs wait forever. The deep check catches it.

#Uptime alerts

• GitHub Actions (built in): .github/workflows/uptime.yml checks the deep endpoint every 10 minutes. Set the repository variable RAPIDFORGE_URL (Settings → Secrets and variables → Actions → Variables) to turn it on. When two checks 30 s apart fail, it opens an issue labelled downtime, which GitHub emails to watchers. It comments while the outage lasts and closes the issue when the service recovers.
• An external monitor as well: GitHub can delay scheduled workflows by several minutes, and can't page anyone. Point a monitor such as Better Stack, UptimeRobot, Pingdom or OCI Health Checks at https://<your domain>/healthz?deep=1, expecting status 200, every 1–5 minutes, from two or more regions, with a phone or Slack alert.

#Capacity alerts

Oracle sends no warning as an Always Free VM fills up; a full box just starts killing containers. The Terraform in deploy/terraform/oci can create OCI Monitoring alarms (free) that email you: set alert_email in terraform.tfvars and apply, then click the confirmation link OCI emails you. It alerts on

• CPU above alert_cpu_percent (default 80%) for 10 minutes;
• memory above alert_memory_percent (default 85%) for 5 minutes (uses the Compute Instance Monitoring agent plugin, which the same change enables);
• the VM going silent for 10 minutes.

OCI doesn't publish disk usage or egress as alarm-ready metrics. Check disk with df -h and docker system df, or run a cron check that pings a healthchecks.io URL when usage is above 80%. Always Free includes 10 TB of outbound transfer a month; see the VCN's metrics in the OCI console.

#Backup alerts

Set BACKUP_ALERT_EMAIL to get an email (through the app's SMTP settings) when a backup or restore test fails. To also notice a backup that silently stops running, set BACKUP_PING_URL and BACKUP_VERIFY_PING_URL in .env to dead man's switch URLs (for example from healthchecks.io). The backup and verify scripts ping them on success and <url>/fail on failure, so you hear about a backup that fails or silently stops running.

#Errors in Diffbridge itself

Unexpected API errors, failed background jobs and browser crashes are grouped by cause in Admin panel → Errors, with how often they happened, where, and the stack. Platform admins (PLATFORM_ADMIN_EMAILS) get an email the first time an error appears, again at most hourly while it continues, and again if it comes back after being marked resolved.

#Metrics

Set METRICS_TOKEN and point Prometheus (or Grafana Agent, Datadog, OCI Monitoring) at GET /metrics with Authorization: Bearer <token>. It reports:

• request counts and latency by route (rapidforge_http_requests_total, rapidforge_http_request_duration_seconds);
• deployments and background jobs waiting or running, and jobs failed in the last hour;
• workers alive and seconds since one last reported;
• open error groups, and workspaces.

Good first alerts: rapidforge_workers_alive == 0 for 5 minutes, the p95 latency over 2 s, and rapidforge_background_tasks{status="queued"} growing for 15 minutes.

#Logs and metrics

• docker compose logs -f api worker: structured JSON, one line per request or job.
• Failed background jobs log task failed or worker tick failed with the error.
• docker stats for CPU and memory; the VM's disk with df -h. Postgres and the blob volume are the parts that grow.
• In the app: Admin panel → audit log for who did what. Each workspace's deployments and CI runs show their own errors.

#Backups

What	How	Where	Kept
Database	backup-to-object-storage.sh, nightly at 03:15 (cron, installed by cloud-init)	s3://<backup bucket>/backups/<UTC time>/db.dump	35 days (Terraform backup_retention_days), or BACKUP_KEEP_DAYS for a bucket without a lifecycle rule
Database, on the VM	pgbackup container, one dump a day	Docker volume rapidforge_pgbackups (db-1.dump … db-7.dump)	7 days
Blobs on local disk (STORAGE_DRIVER=fs)	Archived with the nightly backup	…/blobs.tar.gz	35 days
Blobs in Object Storage / S3	Bucket versioning: overwritten and deleted blobs keep their old versions	The app's bucket	30 days (Terraform blob_version_retention_days)
Restore test	verify-backup.sh, Sundays at 04:45	Restores the newest backup into a throwaway Postgres and checks it	—

Every backup folder has a SHA256SUMS file, checked on restore. Terraform puts backups in their own bucket (rapidforge-backups), so the app's credentials and its bucket rules can't delete them. For stronger protection, copy them to another region or account: set BACKUP_S3_ENDPOINT, BACKUP_REGION, BACKUP_ACCESS_KEY_ID and BACKUP_SECRET_ACCESS_KEY.

What backups don't contain: the .env file, and with it ENCRYPTION_KEY. The database holds each workspace's encryption key wrapped with ENCRYPTION_KEY, so a backup can't be read without it. Keep a copy of .env in a password manager or secrets vault, separately from the backups. With Terraform: terraform output -raw encryption_key.

Stored backup files (the blob storage) are encrypted with a per-workspace key kept in the database, so a copied storage disk or bucket is unreadable without the database and ENCRYPTION_KEY. Back up all three together. After the first deploy with this feature, press Encrypt them now in Admin → Storage to convert older files.

Run a backup by hand any time:

sudo /opt/rapidforge/backup-to-object-storage.sh

#Restoring

#Database

cd /opt/rapidforge
sudo ./restore-db.sh --list                 # what's there
sudo ./restore-db.sh latest --yes           # newest backup
sudo ./restore-db.sh 2026-11-05T0315Z --yes # a specific one

The script downloads the backup, checks its checksums, stops the API and worker, replaces the database, and starts them again. Anything written after the backup is lost. Deployments that were running at backup time show as running; the worker re-attaches to them and reports their final state from Salesforce. Add --blobs to also restore the on-disk blob volume from that backup.

#Blobs in Object Storage

Each overwritten or deleted object keeps its previous versions for 30 days. In the OCI console: open the bucket, show deleted objects and object versions, and restore the version you need. With the OCI CLI, find it with oci os object list-object-versions and copy it over the current object with oci os object copy --source-version-id <id>. On AWS S3, use ListObjectVersions and CopyObject the same way.

#Losing the whole VM

1. Create a new VM: terraform apply again, or any Docker host with deploy/ copied over.
2. Put back the old .env. You need the same ENCRYPTION_KEY and SESSION_SECRET.
3. docker compose up -d postgres, then ./restore-db.sh latest --yes (add --blobs if blobs were on local disk), then docker compose up -d.
4. Point DNS at the new IP. Caddy gets a new certificate on its own.
5. Check https://<domain>/healthz?deep=1, sign in, and open a recent comparison and backup.

Targets: losing at most a day of data (the nightly backup), back in service within about 2 hours. For tighter targets, move Postgres to a managed service with point-in-time recovery (OCI Database with PostgreSQL, or Amazon RDS; see Migrating to AWS).

#Restore drills

verify-backup.sh proves weekly that the newest backup restores. Once a quarter, also do a full recovery drill on a scratch VM using the steps above, time it, and record the result. Auditors ask for this record.

#Load testing

See Performance and load testing: a script that measures throughput and latency of a running install, results from a laptop, and what to scale when.

#Security settings

Setting	Default	What it does
TRUST_PROXY	none (1 in docker-compose.yml)	Which reverse proxy's X-Forwarded-For to believe. Wrong values let clients fake their IP past the IP allowlist.
ALLOW_PRIVATE_EGRESS	false	Webhooks, Jira/Linear/Asana/Azure, UI-test hooks and git remotes must be public https addresses. Turn on only for self-hosted installs that integrate with tools on their own network.
METRICS_TOKEN	unset	Enables /metrics.
SECURITY_CONTACT	unset	Publishes /.well-known/security.txt.
PUBLIC_SITE	true	Public home page at / (for everyone, signed in or not), legal pages and indexing. The app is at /dashboard. false for internal or invite-only installs: then there is no home page, and / is the sign-in page (signed in goes to /dashboard).

Every response carries X-Content-Type-Options, X-Frame-Options: DENY and Referrer-Policy; pages carry a Content-Security-Policy that only allows the app's own scripts; production adds HSTS.

#Upgrades and rollback

cd /opt/rapidforge
sudo ./backup-to-object-storage.sh          # a fresh backup first
docker compose pull && docker compose up -d

Migrations run on start and only move forward. To roll back a release, pin the previous image tag (RAPIDFORGE_IMAGE=ghcr.io/…:sha-<commit> in .env). If the new release ran a migration, also restore the backup you took before upgrading.

#Go-live checklist

• Domain points at the VM; https://<domain> has a valid certificate.
• .env reviewed: NODE_ENV=production, DEMO_MODE=false, real PUBLIC_URL, email (SMTP) settings, PLATFORM_ADMIN_EMAILS, SUPPORT_EMAIL.
• A copy of .env (with ENCRYPTION_KEY) is in a password manager or vault.
• Salesforce Connected App created for the production URL (Salesforce Connected App).
• SSH restricted to your IP range (ssh_allowed_cidr in Terraform); no other ports open besides 80 and 443.
• sudo /opt/rapidforge/backup-to-object-storage.sh succeeds, and verify-backup.sh passes.
• BACKUP_PING_URL and BACKUP_VERIFY_PING_URL set and receiving pings.
• RAPIDFORGE_URL repository variable set, and the uptime workflow passes when run by hand.
• An external uptime monitor alerts someone's phone.
• Ubuntu unattended security upgrades on (sudo dpkg-reconfigure -plow unattended-upgrades).
• A full recovery drill done once, with its time noted.
• SECURITY_CONTACT set, so /.well-known/security.txt tells researchers where to report.
• LEGAL_EFFECTIVE_DATE and LEGAL_HOSTING_REGION set, after the terms and privacy policy have been checked against how you run the service; /terms and /privacy no longer say "Draft".
• METRICS_TOKEN set and scraped, with alerts on workers and latency.
• PLATFORM_ADMIN_EMAILS reaches someone who reads it (error emails go there).
• Single sign-on domains verified with their DNS TXT records (Settings → Single sign-on).
• Billing (BILLING_ENABLED) and sign-in providers configured as intended (Billing, sign-in & email setup).