Implement Prefect 3 result persistence and caching to avoid redundant task re-execution across flow runs
domain: docs.prefect.io · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Configure a result storage backend on a task using result_storage=S3ResultStorage(bucket='...') or via the PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK setting for a server-wide default
Enable caching on a task with cache_key_fn=task_input_hash; Prefect computes a hash of the task inputs and returns the cached result if the hash matches a previously completed run
Define a custom cache_key_fn to control cache invalidation logic, for example including a date or upstream version in the key to expire results daily
Set cache_expiration=timedelta(hours=24) to automatically invalidate cached results after a duration, ensuring stale results are not served indefinitely
Use persist_result=True on tasks whose outputs must survive flow run retries; without it, results are stored only in memory and lost on worker restart
Inspect cached results in the Prefect UI under the task run details or query the results storage backend directly
Known gotchas
task_input_hash hashes all task arguments including unhashable types like DataFrames; always use a custom cache_key_fn that converts inputs to a stable string representation for complex argument types
Result persistence writes to the configured storage backend synchronously during task execution; slow storage (e.g., high-latency S3 bucket) adds wall-clock time to every task run
Cache hits bypass task execution entirely, including side-effect code — if a task writes to a database in addition to returning a value, the write is also skipped on a cache hit
Give your agent this knowledge — and 6,400+ more routes
One MCP install gives any agent live access to the full route map across 2,100+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp