ml-ops

Vertex AI Experiments: track and compare training runs and metrics

MLflow Tracing: instrument an LLM/GenAI application with autolog and view traces in the MLflow UI

Vertex AI Feature Store: set up online feature serving (current Feature Store, BigQuery-based) with Bigtable online serving

Kubeflow Trainer v2: run a distributed PyTorch training job with TrainJob and the torch-distributed runtime

Kubeflow Katib: run a hyperparameter tuning Experiment with the Katib Python SDK (or Experiment YAML)

vLLM: serve a model behind an OpenAI-compatible HTTP API using `vllm serve`

MLflow Deployments Server (AI Gateway): stand up a gateway server to proxy a third-party LLM provider endpoint

Run standard benchmark evaluations on a Hugging Face model using EleutherAI's lm-evaluation-harness (lm-eval CLI)

Amazon SageMaker: use deployment guardrails (canary or linear traffic shifting) to safely update a real-time endpoint

BentoML: build a Bento and deploy it to BentoCloud

NVIDIA Triton Inference Server: monitor server and per-model metrics via the built-in Prometheus metrics endpoint

Quantize an LLM to 4-bit for inference using bitsandbytes with Hugging Face Transformers (BitsAndBytesConfig)

KServe: deploy a model as an InferenceService with autoscaling on Kubernetes

TorchServe: check model and server status/health using the Management API and inference health endpoint

KServe: perform a canary rollout by splitting traffic between two InferenceService revisions

Ray Serve: configure autoscaling for a deployment (min_replicas, max_replicas, target_ongoing_requests)

Detect and report data/feature drift between a reference and current dataset using Evidently (Evidently AI)

Amazon SageMaker: run an A/B test between two models using weighted production variants on a single real-time endpoint

vLLM: serve multiple LoRA adapters from a single base model deployment (multi-LoRA)

NVIDIA Triton Inference Server: configure a model repository backed by Amazon S3 instead of local disk

Feast: use a Push Source to stream real-time feature updates into the online (and optionally offline) store

ONNX Runtime: deploy a converted ONNX model behind a REST API (e.g. FastAPI) using an ONNX Runtime inference session

Amazon SageMaker Pipelines: build a pipeline with Processing, Training, and RegisterModel steps using the SageMaker Python SDK

W&B Launch: enqueue and run a training job on a remote target using a launch queue and launch agent