Skip to main content
connic
Back to BlogProduct spotlight

connic tests: Catch Agent Regressions Before They Reach Production

A YAML-driven testing framework built for non-deterministic AI agents. Repeated-run pass thresholds, expression-based assertions, custom-code mocking, multimodal fixtures, and a deploy gate that blocks failed checks by default.

May 6, 2026(last updated: November 20, 2026)9 min readAuthor: connic engineering

A prompt changed, a model was swapped, or a tool was refactored. Did anything break? With traditional code, a test suite provides an answer in seconds. With AI agents, many teams ship a change, watch the dashboard, and hope for the best. That's not engineering. That's gambling against a non-deterministic system.

Today we're spotlighting connic tests, a testing framework built specifically for the way agents actually behave. It treats non-determinism as a first-class citizen, supports assertions across the entire trace, including the final string, and becomes the gate between a green commit and a production deploy.

Why Agents Break Traditional Test Frameworks

Pytest, Jest, JUnit. They all assume the same input always produces the same output. That assumption is the foundation of traditional assertions. Agents shred that assumption. The same prompt can return different tokens, take different tool paths, or surface a flaky failure that disappears on retry. Test suites that pretend the output is deterministic end up either useless (passing every red build) or unusable (failing every green one).

Failure modes ordinary test frameworks miss:

Flaky-by-design Behavior
The agent gets the right answer 7 out of 10 times. A unit test assertion either always passes or always fails. Both are wrong. The test needs a way to express “this is good enough at scale.”
Silent Tool-Call Drift
A refactor still produces a plausible-looking final answer, but the agent quietly stopped calling the calculator and started guessing. The output looks fine. The behavior is broken.
Forbidden Side Effects
A subtle prompt change makes the agent helpfully send an email it should never have sent. The test must assert on both the actions an agent took and those it was forbidden to take.
Live-Data Dependencies
Tests that need a real Stripe charge, a real database row, or a real S3 object can't be shoehorned into a static fixture file. These tests need fixtures that build state and tear it down.

connic tests was designed around exactly these problems.

Test agents before release

Write assertions against real agent runs and catch regressions before they reach production.

The 30-Second Tour

A test suite is just a YAML file at tests/<agent-name>.yaml. It uses the same flat layout as middleware/ and requires neither scaffolding nor boilerplate. Each file declares one or more test cases for the agent of the same name.

tests/stress-tester.yaml
version: "1.0"
agent: stress-tester        # optional, defaults to the filename stem

defaults:
  runs: 10                  # invoke the agent 10 times per case
  success_threshold: 90     # 9 out of 10 must pass
  timeout_s: 60

tests:
  - name: returns_id_10
    payload: '{"a": 4, "b": 6}'
    expected_result: output.id == 10
    expected_tool_calls:
      - math.calculator.add                  # called at least once
      - math.calculator.add: invocations >= 5  # ...or with an expression
    expected_no_tool_calls:
      - email.send                           # must NOT be called

This production-ready test defines ten invocations, a 90% pass threshold, an output assertion, two positive tool-call expectations, and a forbidden side effect without extra fixtures, runners, or glue code.

Built for Non-Determinism

The first thing that makes connic tests feel different is runs and success_threshold. Together they encode repeated-run expectations directly in the test contract.

runs (1–100)
How many independent agent invocations to execute for this case. Pick 1 for cheap deterministic checks. Pick 50 when measuring flakiness on a model that runs hot.
success_threshold (1–100)
Percent of those runs that must pass. A threshold of 100 means every run in the batch must pass; 90 means at least 9 of 10 must pass.
The result: repeatable checks, traceable failures
The same case can act as a smoke test (runs: 1) or a regression sentinel (runs: 50, success_threshold: 95). Tune per case so cheap cases stay cheap and critical cases get broader repeated-run coverage. Every invocation is recorded with its agent run ID and pass/fail outcome, so a case that fails on 3 of 50 runs links straight to the three offending traces. No log spelunking required.
The Run tests phase of a deployment with every test case passing, each case showing a green check and the agent run IDs of its individual invocations
The Run tests phase of a deployment: each case passes as a green row, and every invocation is recorded by its agent run ID. adds_two_numbers ran five times here.

Assertions That Match How Agents Actually Work

A string-equality check on the agent's final output is almost never the right assertion. connic tests provides four assertion shapes, and each one is an expression, not a hardcoded comparator.

expected_result: output expressions
Evaluated against the bindings output, error, and status. Write output.total > 0, output.status == "refunded", or any other safe expression. JSON payloads are parsed automatically.
expected_tool_calls: positive trace assertions
A bare tool name asserts the tool was called at least once. A mapping form like math.calculator.add: invocations >= 5 asserts on the call count. Catches the silent regression where an agent stops using a tool required by the workflow.
expected_no_tool_calls: negative trace assertions
A list of tools that must not be called. The most underrated assertion in agent testing. This proves that a refund flow never sends a confirmation email or a reader agent never tries to write.
Terminal status (implicit)
If expected_result is omitted, the run still has to reach the completed status to pass. A free, baseline “did it crash?” check on every case.
Why expressions, not matchers?
Hardcoded matchers (assertEquals, assertContains) cap the possible questions. Expressions do not. Anything computable over the output object (numeric comparisons, length checks, nested keys, set membership) is already a valid assertion. Same engine that powers connic's trace filters, evaluated safely server-side.

Multimodal Fixtures Without the Setup

Half of the agents people deploy today read PDFs, parse invoices, OCR receipts, or describe images. Their tests should too. Drop a binary in tests/files/ and reference it by name. The runner base64-encodes it and delivers a multimodal payload to the agent.

tests/invoice-extractor.yaml
tests:
  - name: extract_invoice_total
    payload: "Extract the total amount from this invoice."
    files:
      - invoice_a.pdf
      - invoice_b.pdf
    runs: 5
    success_threshold: 100
    expected_result: output.total > 0 and output.currency == "EUR"

Five runs, each receiving both PDFs alongside the prompt, with a strict expectation that every single one returns a positive Euro total. No mocking. No stubs. Real model calls against real fixtures.

Dynamic Builders for Stateful Tests

Some tests cannot be static. A refund agent test must create a real charge first. An updater test must seed a real row. Dynamic builders are small Python modules in tests/builders/ that produce the test payload at run time and tear down the fixture afterwards.

tests/builders/create_charge_then_refund.py
import stripe

def build(context, builder_args, test_name, payload, files):
    """Create a real Stripe charge, then ask the agent to refund it."""
    charge = stripe.Charge.create(
        amount=builder_args["amount_cents"],
        currency="usd",
        source="tok_visa",
    )
    # stash state for cleanup() to read back
    context["charge_id"] = charge.id
    return f"Please refund charge {charge.id} for the customer."

def cleanup(run, context, builder_args):
    """Runs after the agent finishes (pass OR fail)."""
    charge_id = context.get("charge_id")
    if charge_id:
        # ensure the fixture is gone even if the agent forgot
        try:
            stripe.Refund.create(charge=charge_id)
        except stripe.error.InvalidRequestError:
            pass  # already refunded by the agent under test
    return None  # returning False would mark the case failed
tests/billing-agent.yaml
tests:
  - name: refunds_a_real_charge
    builder: create_charge_then_refund
    builder_args:
      amount_cents: 4200
    expected_result: output.status == "refunded"
    expected_tool_calls:
      - stripe.refund.create
    expected_no_tool_calls:
      - email.send

The builder creates state, the agent operates on it, the assertion verifies behavior, and the cleanup hook tears down the fixture, even if the test failed. Builders run inside the same sandbox as the agent, so they see the same environment, the same secrets, and the same tools.

Cleanup runs unconditionally
Returning False from cleanup() additionally fails the case. This is useful when teardown itself reveals a bug (e.g. the agent didn't actually refund the charge it claimed to). Every other return value is treated as success.

Mock Side Effects, Keep the Reasoning

Builders test against real state. Mocks do the opposite: they exercise the agent's reasoning without letting selected custom code actually run. No row written, no email sent, no third-party charge. Drop a Python module in tests/mocks/ and point a case at it with the mocks field.

tests/mocks/customer_mocks.py
from connic import GuardrailResult

# Replace one custom file tool result.
def mock_data_customer_add_customer(tool_name, params, context):
    return {"id": "cust_test_1", "name": params.get("name")}

# Replace middleware and one exact before-hook phase.
def middleware_before(content, context):
    return content

def mock_data_customer_add_customer_hook_before(tool_name, params, context):
    return params

# Replace only this custom input guardrail.
def guardrail_input_domain_check(content, context):
    return GuardrailResult(passed=True)
tests/customer-agent.yaml
tests:
  - name: adds_a_customer_without_touching_the_db
    payload: '{"name": "Ada"}'
    mocks: customer_mocks
    strict_mocks: true
    strict_hook_mocks: true
    strict_middleware_mocks: true
    strict_guardrail_mocks: true
    # The add_customer call is served by the mock but still recorded,
    # so the assertion holds while nothing ever reaches a real datastore.
    expected_tool_calls:
      - data.customer.add_customer: params.name == "Ada"
    expected_result: output.id == "cust_test_1"

Tool replacements use a hierarchy from the exact function (mock_data_customer_add_customer), to its module (mock_data_customer), to its namespace (mock_data), to a catch-all mock. The same module can replace middleware_before / middleware_after, hierarchical hook phases ending in _hook_before / _hook_after, and custom guardrails. A matching function replaces that phase; by default, no match runs the real code. Built-in guardrails, predefined tools, and api: tool implementations always run for real.

Mocked, but still observed
The substituted call is recorded in the trace exactly like a real one, so expected_tool_calls and expected_no_tool_calls keep asserting on what the agent reached for, while the configured return value stands in as the result.
Strict mocks as contracts
Set strict_mocks: true (per file or per case) and the run fails before an unmocked custom file tool executes. This contract is tool-only. Turn on strict_hook_mocks, strict_middleware_mocks, or strict_guardrail_mocks independently to fail before an unmatched configured eligible real lifecycle phase executes. All three default to false; missing phases and built-in guardrails are exempt.

Tests Are a Deployment Gate, Not a Vibe Check

Tests that run on a developer's laptop and nowhere else are a comfort, not a control. connic tests is wired directly into the deployment pipeline:

git push→Build→Test Phase→Promote→Production

By default, connic discovers the test suites in the Project on every normal deployment, expands each case according to its runs count, and executes them in an isolated runner. If any case misses its threshold, the deployment stops before production. connic deploy --skip-tests is a deliberate CLI hotfix escape hatch; Git auto-deploys cannot bypass the gate.

A failed deployment whose Run tests step failed, listing the failing test cases with their invalid-JSON reason and per-run pass/fail results, blocking the rollout
A blocked deploy: the test phase failed, so production was never promoted. Each red case points straight at the offending runs.
Built into the deployment pipeline
No CI plumbing to write. The test phase runs by default in connic's deployment pipeline, and every test run is associated with the deployment that triggered it for full audit history.
Per-case visibility
Each test case becomes its own row in the deployment timeline, with the count of runs attempted, runs passed, and the underlying agent run IDs. Click any failing case to land directly on the failed traces.
Run on demand from the CLI
A fix can be verified without redeploying: connic test executes the same framework against the current environment and streams results back to the terminal.
Flake budgets, not flake denial
A 90% threshold isn't lowering the bar. It's the bar, stated honestly. The deployment passes when the observed batch meets the production-quality policy and fails when it does not.

Faster Deployments with Changed Tests Only

As a suite grows and the important cases run 10 or 50 times each, every deployment waits for the full batch, even when the change touched a single agent. A new prompt for the support agent is no reason to rerun every case of the invoice parser. With Changed Tests Only, connic runs only the cases that depend on something that changed since they last passed in the target environment and reuses the earlier pass for the rest. Reused cases take no time and make no model calls. If nothing changed for any case, the test phase passes right away.

The setting is off by default. Turn it on per environment under Settings → Git & Environments. It applies to deployments to that environment and to PR tests that target its branch. Passes from PR tests count too, so cases a pull request already passed don't run again when the merged change deploys, unless something they depend on changed in between. Ad-hoc connic test runs still execute every case.

What counts as a change
The case itself with its files, builder, and mocks. The agent's configuration (including _defaults.yaml), its API spec tools, and the code of its tools, middleware, hooks, and custom guardrails, along with every project module that code imports. Agents it runs in sequence and agents its test cases list under expected_child_agents count too, with everything they depend on. After an edit to one tool module, only the cases that depend on it run again. A change to requirements.txt or assets/, a different test environment, or an update to connic's agent runtime reruns every case.
Every reused pass is linked
Reused cases show up as unchanged in the deployment's Run tests step, each linked to the deployment where it passed. The deployment log reports how many cases were reused and how many ran, and the PR status counts unchanged cases separately from the ones that ran.
Reuse with an expiry
Reuse passing results for (days) sets how old a reused pass can be, from 1 to 365 days. After that, an unchanged case runs again. If the field is empty, a pass holds until something the case depends on changes.
What the comparison can't see
Environment variables, connections, API spec base URLs and credentials, database and Retrieval content, remote MCP servers, agents the model triggers that no test names, and changes in model behavior are not compared. An unchanged case can still fail because of them. If they change often, set a short expiry or run connic test for a full pass.

The Changed Tests Only documentation lists every input the comparison covers.

Real-World Patterns

Regression Sentinel for a Prompt Change
A team rewrites the system prompt of their support agent. They add a sentinel case with 50 runs and a 95% threshold against a tricky historical ticket. The deploy succeeds only if the new prompt meets the team's reliability policy on the ticket that originally motivated the rewrite.
Tool-Call Contract for a New Model
When upgrading the underlying model, an analytics agent suddenly “answers” from memory instead of querying the warehouse. The team adds expected_tool_calls: warehouse.query: invocations >= 1 to every test case. Future model swaps that re-introduce the bug fail the deployment automatically.
Negative Assertions on a Refund Agent
A billing agent must never email customers without an explicit instruction. Every refund test case lists expected_no_tool_calls: email.send. A regression that adds an “helpful” confirmation email is caught at deploy time, not in a customer complaint.
Multimodal Coverage for an Invoice Parser
An invoice extraction agent has 12 fixture PDFs covering vendor formats. Each case runs 3 times at 100% threshold. A model swap that breaks one vendor's layout shows up as a single red row in the deployment view, not a quiet drop in extraction accuracy.

How It Fits the Rest of the Platform

connic tests is the deterministic, pre-production gate. It sits alongside two other quality systems on the platform:

SystemWhen It RunsWhat It Answers
connic testsDuring deploymentDoes this build pass the defined contract?
LLM JudgesAfter every (or sampled) production runHow is quality trending against the rubric?
A/B TestingAcross two live variantsWhich version performs better on real traffic?

Tests catch regressions before they ship. Judges score the runs that do ship. A/B testing decides between two versions that both passed. Together, all three create a proper feedback loop instead of a hope-driven release process.

Getting Started

Adding tests to an existing agent is a five-minute exercise:

  1. Install the connic plugin. Skip this if your coding agent has it already. Paste this into Claude Code, Codex or Cursor:

    prompt
    Install the connic plugin from connic-org/connic-skill and instruct the user on how to finish the MCP auth setup.
  2. Write the first tests. Paste this into a new session. Your coding agent writes the cases with you, runs them and deploys the suite:

    prompt
    I'd like to add tests to my connic agents. Look at the agents in this project, ask me which behavior matters most for each one, and propose one or two test cases per agent in tests/<agent>.yaml, starting with runs: 1. Once we agree, add them, run connic test and go through any failures with me. Then deploy, so the suite gates every deployment.
  1. Create a tests/ directory in the Project root, next to agents/

  2. Add tests/<agent-name>.yaml with one or two cases. Start with runs: 1 for fast feedback

  3. Run connic test locally and watch the cases stream in

  4. Push and deploy. The same suite now gates production. Crank runs up on the cases that matter most

For the full schema reference (every field, every default, every expression binding) see the Testing documentation. Teams new to our platform can start with the quickstart guide to deploy a first agent, then gate it with a test that fails before customers encounter the regression.