Amazon SageMaker: run an A/B test between two models using weighted production variants on a single real-time endpoint

Steps

1. Create two SageMaker Model objects (one per model version/algorithm/instance combo you want to compare), each pointing at its own S3 model artifact and container image.
2. Build two production variants with sagemaker.session.production_variant(model_name=..., instance_type=..., initial_instance_count=..., variant_name="Variant1"/"Variant2", initial_weight=1) — equal initial_weight values (e.g. 1 and 1) split traffic 50/50 since SageMaker distributes traffic proportionally to weight.
3. Deploy both variants onto one endpoint at once with sm_session.endpoint_from_production_variants(name=endpoint_name, production_variants=[variant1, variant2]) (or the equivalent CreateEndpointConfig/CreateEndpoint API calls).
4. Send test traffic with the standard InvokeEndpoint call to exercise the weighted split, or force a specific variant per-request by passing TargetVariant="Variant1" to invoke_endpoint — a set TargetVariant overrides the weighted routing for that call.
5. Compare per-variant performance using the SageMaker/CloudWatch metrics (e.g. Invocations, Latency) that SageMaker AI emits per production variant, alongside your own offline accuracy/precision/recall evaluation of each variant's predictions.
6. Once a winner is clear, shift traffic gradually (or all at once) by calling UpdateEndpointWeightsAndCapacities with new DesiredWeight values per VariantName (e.g. 25/75, then 0/1) — this changes routing without redeploying the endpoint.