Write Your First Test

A test is a YAML file under tests/ that invokes an agent N times with a fixed payload and asserts on the output and the tools it called. Tests run in a real runner container against a real Connic environment, the same code path your production traffic hits, so a passing suite is meaningful, not a stubbed simulation.

Two ways to run them:

• Ad-hoc: connic test from your terminal. Useful while iterating.
• Deploy gate: every connic deploy and git auto-deploy runs the suite before the new image is shipped. A failing case aborts the deploy.

Both paths share the same discovery, the same execution model, and the same dashboard surfacing. The only difference is whether a successful run also promotes a deployment to active.

After deployment, use judges to score production runs and A/B tests to compare live variants with exploratory metrics or a statistically planned confidence experiment.

Create the test case

Create tests/stress-tester.yaml. The filename targets the stress-tester agent automatically.

version: "1.0"

tests:
  - name: completes_cleanly
    payload: "ping"
    expected_result: status == "completed"

Run it

# Run every suite under tests/
connic test

# Or run only this case while iterating
connic test --filter completes_cleanly

The case passes when the run reaches completed. From here, add output and tool-call assertions, or jump to CLI, coverage, and deploy-gate behavior.

File Layout

Test files live at the project root in a flat tests/ directory, the same convention as middleware/. The filename stem is the agent the suite targets.

So tests/stress-tester.yaml contains tests for the agent named stress-tester. If you want to split a large suite for one agent across multiple files (e.g. smoke vs. load tests), set the top-level agent: field on each file so they all target the same agent. See the test YAML reference.