Create a ClickPipe via the ClickHouse Cloud API from Kafka, Kinesis, object storage, Postgres, MySQL, MongoDB, BigQuery, or Pub/Sub, then poll until it is running

Documented steps

1. Authenticate with HTTP Basic auth (key ID as username, key secret as password). Auth and rate limits: https://clickhouse.com/docs/cloud/manage/openapi ; source and destination field reference: https://clickhouse.com/docs/integrations/clickpipes
2. Resolve IDs: GET /v1/organizations for organizationId, then GET /v1/organizations/{organizationId}/services for the serviceId of a running target service.
3. Recommended for Kafka and Kinesis: discover the source schema first with POST /v1/organizations/{organizationId}/services/{serviceId}/clickpipes/schemaDiscovery and a body of {"source":{"kafka":{...}}} or {"source":{"kinesis":{...}}}. The response's inferred {name, type} field list is what you build destination.columns and fieldMappings from.
4. Build the create body: {"name":..., "source":{<exactly one of> kafka|objectStorage|kinesis|pubsub|postgres|mysql|bigquery|mongodb}, "destination":{...}, "fieldMappings":[...], "scaling":{...}, "settings":{...}}. Only one source key may be present.
5. Set source.type where the family supports it: kafka type is one of kafka|redpanda|msk|gcmk|confluent|warpstream|azureeventhub|dokafka; objectStorage is s3|gcs|dospaces|azureblobstorage|cloudflarer2|ovhobjectstorage; postgres is postgres|supabase|neon|alloydb|planetscale|rdspostgres|aurorapostgres|cloudsqlpostgres|azurepostgres|crunchybridge|tigerdata; mysql is mysql|rdsmysql|auroramysql|mariadb|rdsmariadb. Kinesis, Pub/Sub, MongoDB, and BigQuery have no type selector.
6. Match the credentials block to the declared authentication method, sourcing every value from a secret manager: username and password for PLAIN auth; accessKeyId and secretKey for Kafka IAM_USER and Kinesis IAM auth; certificate and privateKey for MUTUAL_TLS; connectionString for Azure Event Hubs; a base64-encoded service-account JSON for BigQuery and Pub/Sub.
7. For Kafka, Kinesis, object storage, and Pub/Sub set destination as {"database":..., "table":..., "managedTable":..., "columns":[{"name":...,"type":...}]}. Database CDC sources (Postgres, MySQL, BigQuery, MongoDB) instead use their own tableMappings array and omit destination.table and destination.columns.
8. Set fieldMappings as [{"sourceField":...,"destinationField":...}] covering every destination column - an uncovered column is rejected.
9. POST it: curl --user "$KEY_ID:$KEY_SECRET" -X POST -H 'Content-Type: application/json' https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes -d @create_pipe.json
10. Read the returned id (clickPipeId) and initial state, then poll GET /v1/organizations/{organizationId}/services/{serviceId}/clickpipes/{clickPipeId} every 5-10 seconds. Streaming and object-storage pipes go Provisioning to Running (or Completed for a one-shot load); database CDC pipes go Provisioning to Setup to Snapshot to Running.
11. Stop polling at Running (or Completed). Treat Failed or InternalError as terminal and read the response detail before retrying; back off on throttling since the limit is 10 requests per 10 seconds per key.