Detect soft 404s across a large site by combining URL Inspection API coverage states with content-based heuristics, then fix root causes

Steps

1. Pull each URL's indexStatusResult from the URL Inspection API and flag entries whose indexingState or coverage reason indicates a soft-404 determination
2. Cross-reference flagged URLs against your actual HTTP status codes — a soft 404 is a page returning 200 OK whose content Google judges to be an error/empty page
3. Build a heuristic pass over server responses for pages with near-empty bodies, generic 'not found' phrasing, or content nearly identical to a known error template, even if status is 200
4. For legitimately empty results (e.g. an out-of-stock or zero-results search page), decide per template whether to return a real 404/410, add substantive fallback content, or explicitly noindex the page
5. Fix the root template so it returns the correct HTTP status for genuinely missing resources, rather than relying on Google's soft-404 heuristic to sort it out
6. Re-inspect fixed URLs after redeploy and monitor the Page Indexing report's 'Soft 404' bucket for it to shrink over subsequent recrawls