Bulk index documents into OpenSearch or Elasticsearch efficiently while handling backpressure

Steps

1. Use the Bulk API endpoint POST /_bulk which accepts newline-delimited JSON (NDJSON): alternate action lines ({"index":{"_index":"<name>","_id":"<id>"}}) with source document lines; each pair is one operation; end the body with a trailing newline
2. Target bulk request sizes of 5–15 MiB per request and 1,000–5,000 documents per batch as a starting point; tune based on document size and cluster capacity — too-large batches cause GC pressure and timeouts, too-small batches waste HTTP overhead
3. Check the bulk response body for per-item errors even on HTTP 200 responses — the bulk API returns 200 even when individual items fail; iterate over the items array and check errors: true on each entry to identify and retry failed documents
4. Handle backpressure by watching for HTTP 429 (Too Many Requests) with an es_rejected_execution_exception; implement exponential backoff with jitter and retry the entire batch; do not drop documents on 429
5. Tune indexing performance: set refresh_interval to 30s or -1 during bulk loads (disable auto-refresh) and increase number_of_replicas to 0 during initial load, then restore both after loading; this significantly improves ingest throughput
6. Use the _bulk API with routing specified in the action metadata to target specific shards and reduce coordination overhead for high-volume writes into time-series indexes