Running Tests & CI

Running Tests Ad-Hoc

# Discover tests/*.yaml, build a one-shot test image,
# run every case in your default standard env, and exit
# with status 0 if all cases passed.
connic test

connic test picks the default standard environment's test_environment_id if one is set (see Environments), and falls back to the env itself otherwise. Override or filter as needed:

# Run only cases whose name contains the substring
connic test --filter adds_two_numbers

# Pick a specific environment to execute against
connic test --env <environment-id>

# Machine-readable output for CI
connic test --json

# Print local per-agent coverage and exit (no backend call)
connic test --coverage

As cases finish, the CLI prints them in a results table and surfaces a clickable dashboard link to the throwaway deployment that backed the run. Drill in there for per-case agent runs, traces, tool calls, and outputs.

Exit code is 0 when every case passed, 1 on failure, 2 on infrastructure error. Drop connic test straight into your CI pipeline.

Coverage Report

connic test --coverage is a static, offline report. It reads agents/ and tests/ from disk and tells you which agents have tests and which tools those tests actually exercise. No backend call, no credentials, no test container. Safe to wire into a pre-commit hook or a doc-style CI job on every PR.

The model is intentionally simple:

• Every agent counts equally. One of ten agents fully covered is 10% overall, regardless of tool count. This keeps the headline number honest when one agent has 20 tools and another has 2.
• Per-agent score = covered tools / total tools. A tool counts as covered if it appears at least once in any of that agent's expected_tool_calls entries (bare or mapping form) or expected_tool_call_order. To hit 100% on an agent, every one of its tools needs to show up in at least one case.
• No test file → 0%. An agent without a corresponding tests/<agent>.yaml contributes 0 to the average.
• Tool-less agents → 100% if a test file exists. Sequential agents and orchestrators have nothing to cover at the tool level, so a single test file is enough.
• A/B variants are skipped. Test variants like support-test-fast share the base agent's tools and are excluded from the count.
• Discoverable tools count too. Both tools and discoverable_tools are part of the denominator.

Uncovered tools are listed beneath the table. For search-agent above: web.fetch, web.summarize.

Pair it with --json to get a machine-readable report ({overall, agents: [{name, type, has_tests, tools_total, tools_covered, uncovered_tools, percent, parse_error}]}) you can pipe into a CI gate. For example, fail the build if overall coverage drops below a threshold, or if any agent reports a parse_error. Unlike connic test, coverage is a report, not a gate; the human-readable command exits non-zero only when the local coverage inputs cannot be parsed.

PR Testing

Run your suite against pull-request code before it merges. When a PR opens or its head updates, Connic checks out the head commit, runs the same pipeline as connic test against the environment whose branch matches the PR base, and reports the result as a commit status.

PR Testing currently supports GitHub and GitLab and requires at least one YAML file in tests/.

The Deploy Gate

Every deploy, whether triggered by connic deploy or by a git push to your connected branch, runs as a two-phase pipeline:

• 1. Build image: package your project into a runner image.
• 2. Run tests: spin up a one-shot test container from the just-built image, execute every case in tests/, capture results.
• 3. Deploy to {env name}: only if every case passed. Otherwise the deployment is marked FAILED and nothing ships.

You can watch this happen live on the deployment detail page in the dashboard. Each step shows pending → in progress → done; the test step expands to show the per-case list with per-invocation pills you can click to open the run drawer.

Test environment override

By default tests run in the deploy environment. To isolate them so a release can't hit real billing APIs, point Settings → Git & Environments → Test environment at a sibling environment with stub credentials and stage-only connectors. The deploy gate will use that env's vars and connectors for the test phase, then promote the image into the real environment if everything passes.

Skipping the gate

# Force a deploy through even if tests fail (or you have none yet).
# Available only on the CLI -- git auto-deploys never set this.
connic deploy --skip-tests

--skip-tests is CLI-only and intended as an escape hatch, for example, getting a hotfix out while a flaky test is being debugged. Git auto-deploys never expose it.

Where Results Show Up

• Deployments list: every row carries a Tests column with the suite's pass/fail/skipped status. Ad-hoc connic test runs appear with a purple Test run badge so they don't clutter your real release history.
• Deployment detail → Pipeline: the Build → Tests → Deploy timeline with live per-step status.
• Tests step (expanded): one row per case, with status, success ratio, threshold, and a clickable list of agent run IDs.
• Run history: every test invocation lands in the runs table for the env it executed in, tagged with a purple badge so you can filter them out (or drill in).

Best Practices

Patterns that hold up once a suite has more than a couple of cases. How to set up the test env, keep external state from leaking, and write assertions that fail when something regresses.

Set up a dedicated test environment

Create a sibling environment in Settings → Git & Environments (e.g. staging-test) and point your standard env's Test environment dropdown at it. The deploy gate picks it up automatically; ad-hoc connic test resolves it via the standard env's test_environment_id. Each environment is a fresh slate. DB rows, agent sessions, knowledge base content, env vars, and connectors are all keyed by environment, so a test run can't see or mutate production state. See Configure environments.

For one-off CLI runs, pass --env <environment-id> explicitly rather than relying on the default. The fallback is fine for CI but easy to forget when iterating locally. Default new cases to runs: 1 so deploys stay quick, and only raise runs (with a sub-100 success_threshold) on cases where stochastic LLM behavior actually matters.

Real tools, scoped credentials

Connic agents call real tools at test time. Nothing is auto-mocked, by design, so a passing suite is meaningful rather than a stubbed simulation. Point the test env's env vars at sandbox credentials for every external service the agent touches (Stripe test keys, sandboxed email, throwaway S3 bucket), and swap connectors to stage-only instances so production data is never read or written during a run.

For state that needs to exist before the agent runs (a row in your own API, an inbox message, a webhook fixture), use a dynamic payload builder. build() provisions the fixture and returns the agent input; the runner re-imports the module per invocation so fixtures don't bleed across cases. See Builder reference.

Clean up after dynamic builders

Per-env isolation covers Connic-side state (DB, sessions, KB), but anything the builder creates in your own API or a third-party sandbox is your responsibility to delete. Always implement cleanup(run, context, builder_args) alongside build() and DELETE the resource using the id you stashed in context. cleanup() always runs, even on agent timeout or crash, so it's a safe place for teardown. Returning False also fails the case, useful for Python-level checks that don't fit the YAML expression DSL.

Thread fixture state through to assertions

The context dict the builder mutates is also bound as context.<key> inside expected_result and expected_tool_calls. Use it to verify the agent threaded the exact id the builder minted, not just some id of the right shape. Without that check, the agent can hallucinate a real-looking uuid and the case still passes. The canonical pattern is to stash an id in build, reference it in the payload, and pin both sides of the round-trip:

tests:
  - name: fetches_the_row_we_just_inserted
    builder: insert_then_query
    expected_result: output.row.id == context.test_uuid
    expected_tool_calls:
      - db.fetch_row: params.uuid == context.test_uuid and invocations == 1

See Reference context from yaml assertions for the matching builder.

Pin both branches of conditional tool selection

When an agent picks tool A or tool B based on input, write two cases: one asserting expected_tool_calls: [A] with expected_no_tool_calls: [B], and the mirror. Without the negative side, both cases pass as long as some tool got called, so a regression that swaps A and B goes undetected. The same pattern works for approval rules, conditional middleware, or any branch where you need to assert which path ran, not just that the agent called something.

Treat coverage as a separate PR check

connic test --coverage --json is offline and has no backend dependency, so wiring it into the deploy gate is pointless. Run it as a separate PR-only CI job that parses the overall field and any parse_error values, then fails if coverage drops below a threshold (e.g. 60%) or a local suite is malformed. That keeps you from merging untested new agents or tools, without slowing every PR down with the full test container path. See Read the coverage report shape.