configure torchserve dynamic batch inference with batch_size, max_batch_delay, and a custom handler

Steps

1. Register or update the model with the batch_size and max_batch_delay parameters, either via the POST /models management API query params or the batchSize/maxBatchDelay keys in config.properties
2. Implement the handler so its preprocess/inference/postprocess methods accept and return lists (batches) of requests, since a handler expecting single inputs won't gain batching benefits
3. Understand that TorchServe forms a batch using whichever condition is met first: reaching batch_size requests, or waiting max_batch_delay milliseconds
4. Load-test with concurrent requests below and above the configured batch_size to confirm the delay-vs-size tradeoff behaves as expected for your traffic pattern
5. Monitor per-request latency percentiles, since max_batch_delay directly adds to tail latency for early-arriving requests in a batch window