Define ServiceMonitor and PodMonitor custom resources for the Prometheus Operator to auto-discover scrape targets

Steps

1. Create a ServiceMonitor resource with spec.selector (a label selector matching one or more Kubernetes Services) and spec.endpoints, where each endpoint specifies the port (by name, matching a named port on the target Service), path (defaults to /metrics), and optionally interval/scrapeTimeout/honorLabels/relabelings/metricRelabelings.
2. For pod-level scraping without a Service, create a PodMonitor instead, using spec.selector to match Pods directly and spec.podMetricsEndpoints (instead of endpoints) to define the port (by container port name or number) and path/interval settings.
3. Set spec.namespaceSelector on both CRDs to control which namespaces are searched for matching Services/Pods; if omitted, only the ServiceMonitor's/PodMonitor's own namespace is considered.
4. On the Prometheus (or PrometheusAgent) custom resource itself, ensure spec.serviceMonitorSelector / spec.serviceMonitorNamespaceSelector (and the podMonitor equivalents) are set to match your ServiceMonitors/PodMonitors — an empty {} selector matches all, while a nil/unset selector may mean none are selected.
5. Apply the CRDs with kubectl and verify pickup via the Prometheus Operator's generated configuration or the Prometheus UI's Targets page/Service Discovery page.
6. Use relabelings (pre-scrape, e.g. to rewrite target labels) vs metricRelabelings (post-scrape, applied to scraped samples) appropriately within the endpoint/podMetricsEndpoint definitions.