Drains
Forward logs, run events, environment health, traces, and audit entries to Datadog, an OpenTelemetry backend, or a webhook.
On this page
Overview
A drain watches one kind of project data and forwards new entries to a channel, so monitors, dashboards, and paging run in the tools your team already uses. Drains and channels require an Enterprise plan and the channels.manage permission.
| Drain | Configure In | Forwards | Filters |
|---|---|---|---|
| Run drain | Agent Runs → More actions → Run Drains | Run events with inputs, outputs, and context, traces, and environment health every minute. | Environment, run status, trigger source, entry expression. |
| Log drain | Logs → Log Drains | Lines captured from custom code and the runtime, up to 500 per request. | Environment, level, source, trigger source, entry expression. |
| Audit drain | Project Settings → Audit Log | Audit-log entries, about every 60 seconds, up to 200 per request. | Action category, user, entry expression. |
Channels
Add a channel in Project Settings → Notifications. One channel can serve several drains, notifications, and approval requests at the same time.
| Channel | Delivers | Availability |
|---|---|---|
| Webhook | A JSON array per request. With a signing secret, each request carries X-Connic-Timestamp and X-Connic-Signature, the hex HMAC-SHA256 of {timestamp}.{raw body}. | Available |
| Datadog | Datadog Logs, using an API key and your Datadog site. | Available |
| OpenTelemetry | OTLP/HTTP with JSON encoding to any collector or vendor endpoint, with optional headers. | Available |
| Slack | Messages in a Slack channel through an incoming webhook. | On request |
| Microsoft Teams | Messages in a Teams channel through an incoming webhook. | On request |
Channel types marked On request appear in the channel picker and are enabled when you ask Connic support for them.
Datadog
Each entry becomes a log with source connic, the agent name as service (connic when there is none), an env tag from the environment name, and a status from level. Connic sends each entry in full; Datadog itself truncates a single log larger than 1 MB. Every other field is a log attribute, so log monitors and log-based metrics can use values such as @run_status or @duration_ms directly:
| Signal | Log query | Measure |
|---|---|---|
| Failed runs per agent | source:connic @type:run @run_status:failed | count by service |
| Run duration | source:connic @type:run | @duration_ms |
| Queue depth per environment | source:connic service:connic @type:environment_health | @queued_runs |
| Longest-running run per agent | source:connic @type:environment_health -service:connic | @oldest_running_seconds |
OpenTelemetry
Set the endpoint to the receiver's base URL, such as https://otel.example.com:4318. Connic sends spans to /v1/traces and every other entry to /v1/logs. Headers use the key=value,key2=value2 format of OTEL_EXPORTER_OTLP_HEADERS.
The resource carries service.name (the agent name) and deployment.environment.name. Log records for a run use the run ID as their trace ID, so a backend can show a run's logs next to its trace.
Private Endpoints
Webhook and OpenTelemetry channels can deliver through a Connic Bridge. Select the bridge on the channel to reach an endpoint inside your network, including plain HTTP. For example, point an OpenTelemetry channel at the OTLP receiver of a Datadog Agent in your cluster, such as http://datadog-agent.datadog.svc.cluster.local:4318, to get traces in Datadog APM. If the bridge agent has ALLOWED_HOSTS configured, it must include the endpoint's host:port.
Run Drains
A run drain forwards everything by default: run events with their inputs, outputs, and context, traces, and environment health. Turn off the parts you don't need, or select environments. Filters for run status, trigger source, and entry expressions apply only to run events and traces. Nothing is truncated.
Run Events
One entry per finished run, including deployment test runs (trigger_source deploy_gate). The drain checks every 15 seconds and picks up a run about a minute after it ends, so runs committed late are not skipped. duration_ms is active execution time; queue_wait_ms is the time between queueing and the start of processing. input and output are the run's full input and output, parsed when they are JSON, and context holds the context keys set by middleware. Turn off Inputs and outputs or Run context to leave them out.
{
"type": "run",
"id": "a1b2c3d4-...",
"timestamp": "2026-10-12T09:41:07.512000Z",
"level": "error",
"run_id": "a1b2c3d4-...",
"run_status": "failed",
"agent_name": "invoice-processor",
"environment_id": "e7f8a9b0-...",
"environment_name": "production",
"project_id": "9840b2b8-...",
"deployment_id": "d4c3b2a1-...",
"connector_id": "5e6f7a8b-...",
"parent_run_id": null,
"trigger_source": "connector",
"test_name": null,
"queued_at": "2026-10-12T09:38:55.104000Z",
"started_at": "2026-10-12T09:38:57.880000Z",
"ended_at": "2026-10-12T09:41:07.512000Z",
"queue_wait_ms": 2776.0,
"duration_ms": 129632.0,
"token_usage": { "input_tokens": 18234, "output_tokens": 2210, "total_tokens": 20444 },
"error": "TimeoutError",
"context": { "request_type": "invoice_validation", "customer_id": "C-1042" },
"input": { "payload": { "invoice_id": "INV-7731" } },
"output": null,
"channel_id": "c5d4e3f2-...",
"channel_name": "Datadog production"
}Environment Health
At the start of every minute, one snapshot per environment with the current number of queued, running, and approval-waiting runs, plus the age of the oldest queued and running run. Each agent that has active runs gets its own snapshot as well. Environment totals have no agent_name. Alert on oldest_running_seconds to find stuck runs and on queued_runs for a growing backlog.
{
"type": "environment_health",
"id": "0f9e8d7c-...",
"timestamp": "2026-10-12T09:42:00.031000Z",
"level": "info",
"environment_id": "e7f8a9b0-...",
"environment_name": "production",
"agent_name": "invoice-processor",
"queued_runs": 4,
"running_runs": 2,
"awaiting_approval_runs": 1,
"oldest_queued_seconds": 312.4,
"oldest_running_seconds": 2950.1
}Traces
Every span of each matching run is forwarded, with the run ID as trace ID and the parent links, timing, status, and attributes from the run's trace. Each span also carries its full input, output, and reasoning unless Inputs and outputs is off. Spans are sent in batches of up to 1,000.
OpenTelemetry channels receive them as OTLP traces, with input, output, and reasoning in connic.input, connic.output, and connic.thoughts. Webhook and Datadog channels receive one entry per span. To see traces in Datadog APM, send them through an OpenTelemetry channel to the Datadog Agent.
{
"type": "span",
"id": "7c1d2e3f-...",
"timestamp": "2026-10-12T09:39:01.204000Z",
"level": "info",
"run_id": "a1b2c3d4-...",
"trace_id": "a1b2c3d4e5f6...",
"span_id": "9f8e7d6c5b4a3921",
"parent_span_id": "0011223344556677",
"name": "tool:lookup_invoice",
"span_status": "ok",
"start_time": "2026-10-12T09:39:01.204000Z",
"end_time": "2026-10-12T09:39:02.118000Z",
"duration_ms": 914.0,
"attributes": { "tool.name": "lookup_invoice" },
"input": "{\"invoice_id\": \"INV-7731\"}",
"output": "{\"total\": 1840.5}",
"thoughts": null,
"agent_name": "invoice-processor",
"trigger_source": "connector",
"environment_id": "e7f8a9b0-...",
"environment_name": "production",
"project_id": "9840b2b8-..."
}Log Drains
A log drain forwards the lines that tools, middleware, hooks, and guardrails print or log, about every 15 seconds. To keep structured fields, log them as a json.dumps(...) string; the receiver can parse message.
{
"type": "log",
"id": "3f1c7a9e-...",
"timestamp": "2026-10-12T09:41:07.512000Z",
"level": "warning",
"source": "log:tool:lookup_customer",
"message": "cache miss for C-1042",
"run_id": "a1b2c3d4-...",
"agent_name": "invoice-processor",
"trigger_source": "connector",
"environment_id": "e7f8a9b0-...",
"environment_name": "production",
"project_id": "9840b2b8-..."
}Audit Drains
Audit entries use "type": "audit_log" with action, resource_type, resource_id, user_id, user_email, changes, ip_address, and created_at. Guardrail blocks arrive as guardrail.block with the guardrail type in changes.after.
Filter Expressions
An entry expression is evaluated against each log entry, audit entry, or run event as entry, and only matching entries are forwarded. For run drains, it also decides which runs are traced, and entry.context, entry.input, and entry.output are available even when they are not forwarded. Examples:
entry.agent_name == 'invoice-processor' and entry.duration_ms > 60000entry.context.request_type == 'tco_mapping'entry.level == 'error'
See Expressions for the syntax.
- A new drain forwards entries created after it was added. Earlier entries are not backfilled.
- Any response below 400 counts as delivered. Failed deliveries of run events, traces, logs, and audit entries are retried on the next cycle. Failed health snapshots are replaced by fresh snapshots the next minute.
- An entry can arrive more than once. Deduplicate on
id, which is the run ID for run events. - Disabling a channel pauses its drains. Turn them back on from the drain list once the channel is enabled.