Tune Pinecone serverless metadata filtering for high-cardinality fields using disk-based filtering

Steps

1. Design metadata schema to avoid unbounded string fields as filter targets; prefer low-to-medium cardinality fields (e.g. category, region) for frequent filters
2. Upsert vectors with structured metadata: {'category': 'electronics', 'price': 49.99, 'active': true}
3. Issue a query with a metadata filter object: client.query(vector=[...], filter={'category': {'$eq': 'electronics'}, 'price': {'$lte': 100}}, top_k=20)
4. For high-cardinality string fields (e.g. user_id), prefer namespace isolation over metadata filtering to avoid full metadata scans
5. Benchmark recall vs latency trade-off: metadata filtering performs a pre-filter pass before ANN search, so overly selective filters on large indexes reduce recall
6. Use $in operator for set membership filters instead of multiple $eq OR conditions to reduce query complexity