Take and restore Prometheus TSDB snapshots via the admin API for backup and disaster recovery

Steps

1. Start Prometheus with the --web.enable-admin-api flag set, since the admin API (including snapshotting) is disabled by default and returns an error/forbidden response otherwise.
2. Trigger a snapshot with a POST (or PUT) request to /api/v1/admin/tsdb/snapshot; optionally pass ?skip_head=true to skip snapshotting in-memory head-block data not yet compacted to disk.
3. The response returns a JSON object naming the created snapshot directory, which is written under <data-dir>/snapshots/<datetime>-<rand>; copy this directory off-host (e.g. to object storage) as your backup artifact.
4. To restore, stop the target Prometheus instance, copy the snapshot's block directories into a (typically empty) TSDB data directory, then start Prometheus pointed at that data directory — Prometheus will pick up the restored blocks on startup.
5. For related maintenance, POST /api/v1/admin/tsdb/delete_series?match[]=<selector> marks matching series as deleted, and POST /api/v1/admin/tsdb/clean_tombstones physically removes deleted data and reclaims space; both also require --web.enable-admin-api.
6. Automate periodic snapshotting and off-host copying (e.g. via a sidecar/cron job) rather than relying on ad hoc manual snapshots for real disaster-recovery coverage.