Write Your First Test

A test is a YAML file under tests/ that invokes an agent one or more times with a static payload or dynamic builder and asserts on the output, tool calls, and triggered agents. Tests use the selected Connic environment.

Two ways to run them:

• Ad-hoc: connic test from your terminal. Useful while iterating.
• Deploy gate: every connic deploy and git auto-deploy runs the suite before activation. A failing case aborts the deploy.

Both paths accept the same suites and show case and invocation results in the dashboard. A successful deploy-gate run also allows the deployment to activate.

Use judges to score runs and A/B tests to compare live variants with exploratory metrics or a statistically planned confidence experiment.

Create the test case

Create tests/stress-tester.yaml. The filename targets the stress-tester agent automatically.

version: "1.0"

tests:
  - name: completes_cleanly
    payload: "ping"
    expected_result: status == "completed"

Run it

# Run every suite under tests/
connic test

# Or run only this case while iterating
connic test --filter completes_cleanly

The case passes when the run reaches completed. From here, add output and tool-call assertions, or jump to CLI, coverage, and deploy-gate behavior.

File Layout

YAML suites are discovered recursively under the project-root tests/ directory. Unless the file sets agent:, its filename stem is the target agent name.

So tests/stress-tester.yaml contains tests for the agent named stress-tester. If you want to split a large suite for one agent across multiple files (e.g. smoke vs. load tests), set the top-level agent: field on each file so they all target the same agent. See the test YAML reference.