---
title: "Backups and restore"
description: "What a database backup holds, and how to bring wervt back from one."
canonical_url: "https://docs.lightdrop.cloud/self-hosting/backups"
---
# Backups and restore

> What a database backup holds, and how to bring wervt back from one.

All app data lives in the `wervt` database, one schema per app (`app_<name>`). A backup is a
dump of that database (`pg_dump --format=custom --no-acl --no-owner`): on Coolify its scheduled
backups, on a [bare server](https://docs.lightdrop.cloud/self-hosting/bare-server) the `postgres-backup` service.

## Postgres dumps on a bare server

`postgres-backup` dumps the database when it starts (so after every deploy) and then once a day,
into the `pg-backups` volume as `wervt-<time>.dump`. It keeps `WERVT_BACKUP_KEEP_DAYS` days of
dumps there (default 14). With `WERVT_BACKUP_S3_URL` and `WERVT_BACKUP_S3_BUCKET` set (plus
`_REGION`, `_ACCESS_KEY_ID`, `_SECRET_ACCESS_KEY` and optionally `_PREFIX`, default
`wervt/postgres`), it also copies each dump to that bucket. Any S3 API works (AWS, Hetzner
Object Storage, Cloudflare R2, …); old dumps there expire through the bucket's lifecycle rules.
Each run logs `postgres backup: done` or `postgres backup: FAILED`.

A volume on the same disk isn't a backup on its own: set up S3, or copy the volume off the server.
To restore, copy a dump into the Postgres container and follow the steps below:

```bash
docker compose -f compose.yaml -f compose.caddy.yaml cp wervt-<time>.dump postgres:/tmp/wervt.dump
```

The dump doesn't hold:

- **App roles.** Roles are cluster-wide, not part of a database dump. Their passwords are
derived from `WERVT_DB_SECRET`, so wervt can recreate them.
- **Ownership and grants**, with `--no-owner --no-acl`. After a restore, everything belongs to
the restoring superuser and the app roles can't connect.

`wervt reprovision` puts both back. It needs the same `WERVT_DB_SECRET` as before.

## Restore

1. Stop the runtime, so apps don't write during the restore.
2. Restore the dump into an empty `wervt` database:```bash
pg_restore --username postgres --dbname wervt --no-owner --no-acl wervt.dump
```
3. Recreate the roles, CONNECT grants and ownership, and run each app's migrations:```bash
wervt reprovision --all
```

<br />

Migrations only run when the backup is older than the active release.
4. Clear Electric's storage (the `electric` volume) and restart Electric. Its shape logs belong
to the old database.
5. Start the runtime.

`wervt reprovision <app>` does the same for one app. It is safe to run any time.

## Files (Garage)

App files live in Garage, one bucket per app (`app-<name>`). The `storage-backup` service copies
Garage to a Hetzner Storage Box when it starts (so after every stack deploy) and then once a day:

- `data/`: Garage's data blocks. They never change once written, so each run only sends new
blocks.
- `meta/db.lmdb`: a metadata snapshot (buckets, keys, object lists) taken right before the copy.

It needs `STORAGE_BOX_HOST` (e.g. `u123456.your-storagebox.de`), `STORAGE_BOX_USER` and
`STORAGE_BOX_PASSWORD`, plus optionally `STORAGE_BOX_DIR` (default `wervt/garage`). It connects over
SFTP on port 23. For a sub-account, turn on **SSH support** and **External reachability** in the
Hetzner Console, or the login is refused. Until the variables are set it doesn't back anything up,
and says so in its log. Each run logs `storage backup: done` or `storage backup: FAILED`.

### Test a restore

`deploy/restore-drill.sh` downloads the backup, restores it into a throwaway Garage next to the
running one (without network), and compares it with the live files:

```bash
ssh root@<server> 'APP=<coolify app uuid> bash -s' < deploy/restore-drill.sh
```

It stays read-only towards the running stack. For live reads it uses each app's own S3 key, which
it takes from the restored metadata, so it creates no key and writes nothing. Per bucket it:

- reads every restored file back and hashes it (SHA-256), so a missing or damaged data block
shows up;
- compares the restored and the live object list: key, size and last-modified time of every
object;
- downloads a sample from the live bucket and compares its SHA-256 with the restored copy: the 5
largest files, random ones up to 50, and every ablage metadata object (`<group>/meta/<id>`) up to
64 KiB. ETags can't replace this, they are only MD5 sums for single-part uploads.

Expected differences are listed but don't fail the drill:

- **changed**: the live object is newer than the backup (overwritten since); not compared.
- **new**: only live, created after the backup.
- **deleted**: in the backup, gone live. Deletes from the dashboard are in the audit log.
- **swept**: bytes of a direct upload that was never completed (marked
`x-amz-meta-ablage-direct-upload`, no `<group>/meta/<id>`), which the abandoned-upload sweep
removed after a day.

The drill fails (exit code 1) on a **mismatch** (same object, different content or size, or the
live copy can't be read), an **unreadable** restored file, or a **missing** one: a live object
older than the backup that isn't in it. It ends with a table per bucket and `restore drill: ok`
or `restore drill: FAILED`; failed objects are listed by key above it.

Environment variables change the sample: `SAMPLE` (default 50), `LARGEST` (5), `META_MAX_BYTES`
(65536) and `SEED`. The random part depends only on `SEED`, which the drill prints, so
`SEED=<printed seed>` repeats a run with the same sample. The backup time comes from the
snapshot's modification time; `BACKUP_TIME=<unix seconds>` overrides it.

### Restore

1. Stop Garage.
2. Copy the backup into the `garage` volume: `data/` to `/var/lib/garage/data`, and the snapshot
to `/var/lib/garage/meta/db.lmdb/data.mdb`:```bash
rclone copy box:wervt/garage/data /var/lib/garage/data
rclone copyto box:wervt/garage/meta/db.lmdb /var/lib/garage/meta/db.lmdb/data.mdb
```
3. Start Garage. Buckets and keys come back with the metadata. The keys are derived from
`WERVT_DB_SECRET`, so with the same secret apps reach their files as before.
4. On a new server (an empty `garage` volume), Garage starts as a new node without a role. Give it
one:```bash
docker exec <garage container> /garage layout assign -z dc1 -c 1TB <node id from garage status>
docker exec <garage container> /garage layout apply --version 1
```


## Sitemap

See the full [sitemap](https://docs.lightdrop.cloud/sitemap.md) for all pages.
