{"id":"a64207c4-0375-4d6e-ba55-1a47e734f6cf","task":"Amazon SageMaker: run an A/B test between two models using weighted production variants on a single real-time endpoint","domain":"ml-ops","steps":["Create two SageMaker Model objects (one per model version/algorithm/instance combo you want to compare), each pointing at its own S3 model artifact and container image.","Build two production variants with sagemaker.session.production_variant(model_name=..., instance_type=..., initial_instance_count=..., variant_name=\"Variant1\"/\"Variant2\", initial_weight=1) — equal initial_weight values (e.g. 1 and 1) split traffic 50/50 since SageMaker distributes traffic proportionally to weight.","Deploy both variants onto one endpoint at once with sm_session.endpoint_from_production_variants(name=endpoint_name, production_variants=[variant1, variant2]) (or the equivalent CreateEndpointConfig/CreateEndpoint API calls).","Send test traffic with the standard InvokeEndpoint call to exercise the weighted split, or force a specific variant per-request by passing TargetVariant=\"Variant1\" to invoke_endpoint — a set TargetVariant overrides the weighted routing for that call.","Compare per-variant performance using the SageMaker/CloudWatch metrics (e.g. Invocations, Latency) that SageMaker AI emits per production variant, alongside your own offline accuracy/precision/recall evaluation of each variant's predictions.","Once a winner is clear, shift traffic gradually (or all at once) by calling UpdateEndpointWeightsAndCapacities with new DesiredWeight values per VariantName (e.g. 25/75, then 0/1) — this changes routing without redeploying the endpoint."],"gotchas":["Weights are relative, not percentages: if Variant1 has weight 1 and Variant2 has weight 3, Variant2 gets 75% of traffic (3 of the total 4 weight units) — set both variants' weights deliberately.","TargetVariant on invoke_endpoint always overrides the configured traffic-distribution weights for that specific call, so mixing targeted-invocation testing with live weighted A/B traffic on the same endpoint can distort your read of the natural split.","Each variant can use a different instance type and count, so an A/B test can simultaneously compare model quality and infrastructure/cost tradeoffs, not just accuracy."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/a64207c4-0375-4d6e-ba55-1a47e734f6cf"}