{"id":"b8d9db51-d736-405a-a5f0-56a977cf1534","task":"Take and restore Prometheus TSDB snapshots via the admin API for backup and disaster recovery","domain":"prometheus.io","steps":["Start Prometheus with the --web.enable-admin-api flag set, since the admin API (including snapshotting) is disabled by default and returns an error/forbidden response otherwise.","Trigger a snapshot with a POST (or PUT) request to /api/v1/admin/tsdb/snapshot; optionally pass ?skip_head=true to skip snapshotting in-memory head-block data not yet compacted to disk.","The response returns a JSON object naming the created snapshot directory, which is written under <data-dir>/snapshots/<datetime>-<rand>; copy this directory off-host (e.g. to object storage) as your backup artifact.","To restore, stop the target Prometheus instance, copy the snapshot's block directories into a (typically empty) TSDB data directory, then start Prometheus pointed at that data directory — Prometheus will pick up the restored blocks on startup.","For related maintenance, POST /api/v1/admin/tsdb/delete_series?match[]=<selector> marks matching series as deleted, and POST /api/v1/admin/tsdb/clean_tombstones physically removes deleted data and reclaims space; both also require --web.enable-admin-api.","Automate periodic snapshotting and off-host copying (e.g. via a sidecar/cron job) rather than relying on ad hoc manual snapshots for real disaster-recovery coverage."],"gotchas":["The admin API (snapshot, delete_series, clean_tombstones) is disabled by default in current Prometheus versions; forgetting --web.enable-admin-api means the snapshot request fails even though the endpoint exists.","Snapshot files are created via hard links to existing blocks to save disk space — modifying, moving with wrong permissions, or altering ownership of the snapshot files in place can corrupt the live TSDB data they're linked to.","There is no dedicated 'restore' API endpoint; restoring is a manual filesystem operation (stop Prometheus, place snapshot blocks into the data directory, restart), so recovery procedures must be tested and documented separately."],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/b8d9db51-d736-405a-a5f0-56a977cf1534"}