Testing Framework

Asserting on Triggered Agents

When the agent under test calls trigger_agent (see trigger_agent), the test container runs the child agent in-process instead of dispatching to the live deployment. That gives you the same execution model as the parent for any agent the trigger reaches, so expected_child_agents can assert on output, tool calls, and further triggers exactly the way the top-level fields do.

The assertion stacks: each entry is keyed by the triggered agent's name and can carry its own expected_child_agents for whatever that child triggers in turn.

tests:
  # The dispatcher agent calls trigger_agent("summarizer", ...) with
  # wait_for_response=True. In the deploy-gate container the child runs
  # in-process, so its output and tool calls are captured here.
  - name: dispatches_to_summarizer
    payload: '{"text": "..."}'
    expected_child_agents:
      summarizer:
        expected_payload: payload.text != ""
        expected_result: output.summary != ""
        expected_tool_calls:
          - llm.complete: invocations >= 1
        expected_no_tool_calls:
          - email.send

  # Pin the trigger payload against builder context, so the test fails if
  # the agent forwards the wrong fixture id instead of the one it was
  # given. Works whether the parent passed a dict (payload.field) or a
  # string (substring via payload_raw).
  - name: forwards_charge_id_unchanged
    builder: create_charge_then_refund
    builder_args:
      amount_cents: 4200
    expected_child_agents:
      billing-refunder:
        expected_payload: payload.charge_id == context.charge_id

  # Recursive: assert on a grandchild that summarizer triggers in turn.
  # Same shape repeats at every depth -- agent name keys mapping to the
  # same assertion fields, plus its own expected_child_agents.
  - name: dispatches_summarizer_then_publisher
    payload: '{"text": "..."}'
    expected_child_agents:
      summarizer:
        expected_result: output.summary != ""
        expected_child_agents:
          publisher:
            expected_tool_calls:
              - kafka.publish: params.topic == "summaries"

  # Fire-and-forget triggers (wait_for_response=False) cannot have their
  # result inspected, but the payload is recorded at call time -- so
  # expected_payload still applies.
  - name: fans_out_telemetry
    payload: '{"event": "checkout"}'
    expected_child_agents:
      telemetry-writer:
        expected_triggered: 1
        expected_payload: payload.event == "checkout"

Two evaluation paths

• wait_for_response=True: the child runs synchronously inside the test container with its own tool-call collector, so expected_result, expected_tool_calls, expected_no_tool_calls, and nested expected_child_agents all apply.
• wait_for_response=False: fire-and-forget. The framework only knows the call happened and what payload it carried; use expected_triggered and expected_payload here. If a fire-and-forget trigger is the only match and the spec carries result / tool / nested assertions, the case fails with a clear reason telling you to wait for the response.

Asserting on the trigger payload

expected_payload uses the same expression grammar as expected_result, just with input-side bindings. Use payload.<key> when the parent passed a dict or a JSON string, and payload_raw for substring checks against a free-form string trigger. context.<key> is bound the same way the other assertions bind it, so you can pin a forwarded fixture id with payload.charge_id == context.charge_id. Because the payload is captured at call time, this assertion works on fire-and-forget triggers too — it's the one piece of every trigger record that's always observable.

trigger_agent_at in test mode is treated as fire-and-forget (the test container never waits for the scheduled time), so its triggered agents are matched by name and count too.

Matching semantics

• Per-trigger. Each trigger_agent call gets its own record, with its own captured tool calls and grandchildren — they don't leak back into the parent.
• At-least-one-must-pass. When the parent triggered the same child more than once, the assertion passes as soon as one waited trigger satisfies the spec.
• Builder context is shared. context.<key> in a child's expected_result or expected_tool_calls reads the same builder dict the top-level case uses, so a fixture id stashed in build() is reachable at every depth.
• Only inside the deploy-gate container. Production trigger_agent calls still route via the normal API path. The in-process dispatch is exclusive to tests so a deploy gate can never side-effect the live deployment.

File Attachments

Drop binary fixtures (PDFs, images, audio, anything the agent will receive in production) into tests/files/ and reference them by bare filename in the case's files: list. The runner reads each file, base64-encodes it, and attaches it under files. If payload is a JSON object (or comes from a builder returning a dict), its keys sit at the top level of context["payload"] next to files; otherwise the string is delivered as {message: payload}.

tests:
  - name: extracts_invoice_total
    payload: "extract the total amount as JSON"
    files:
      - invoice_acme.pdf
      - invoice_globex.pdf
    expected_result: output.total > 0

  # Files combine with a static payload (the prompt). They can also be
  # used with a builder -- attached files are merged with whatever the
  # builder returns.
  - name: classifies_receipt
    payload: "is this a meal or travel expense?"
    files:
      - receipt.jpg
    expected_result: 'output.category in ("meal", "travel")'

A few constraints worth knowing:

• Bare filenames only. No path separators, no ... The schema rejects anything that looks like a path.
• 25 MB upload budget. Code/config (everything outside tests/files/) is still capped at 5 MB; fixtures get the remaining headroom.
• Mime type is auto-detected from the extension via Python's mimetypes, falling back to application/octet-stream.
• Missing files fail fast. If a referenced fixture isn't in the upload, the deploy gate aborts before building the test image.

Dynamic Payload Builders

Some tests can't run against a static payload. You need a fresh fixture, a freshly minted database row, or a payload that references an ID that didn't exist a moment ago. Drop a Python module under tests/builders/ exposing build(context, builder_args, test_name, payload, files) (and optionally cleanup(run, context, builder_args)), point the case at it via the builder: field, and the runner will execute the pair once per invocation. build()'s return value (string or dict) becomes the agent input, replacing any static payload.

# tests/builders/create_charge_then_refund.py
import os, requests

def build(context, builder_args, test_name, payload, files):
    """Provision a fixture in your own API, then return the agent payload.

    context       dict       -- mutate to pass state to cleanup()
                                AND to expected_result / expected_tool_calls
    builder_args  dict       -- the yaml `builder_args`
    test_name     str        -- the yaml `name`
    payload       str | None -- the yaml `payload` (if any)
    files         list[str]  -- the yaml `files`
    """
    charge = requests.post(
        f"{os.environ['BILLING_API']}/charges",
        json={"amount_cents": builder_args["amount_cents"], "currency": "eur"},
    ).json()
    # Stash the id so cleanup() can tear the fixture down AND so the
    # yaml expressions can reference it via `context.charge_id`.
    context["charge_id"] = charge["id"]
    return {"charge_id": charge["id"], "instruction": "refund this charge"}


def cleanup(run, context, builder_args):
    """Tear down the fixture. Optionally add Python-level checks.

    Runs after every agent invocation -- pass, fail, or timeout --
    so external resources are always released.

    run["input"]    -- what the agent saw
    run["output"]   -- the agent's parsed output
    run["context"]  -- the run's run_context dict (run_id, agent_name,
                       connector_id, timestamp, plus anything middleware
                       or hooks added during the run)
    context         -- the dict you populated in build()
    builder_args    -- same dict that was passed to build()
    """
    requests.delete(f"{os.environ['BILLING_API']}/charges/{context['charge_id']}")
    # Return False to fail the case (in addition to yaml checks);
    # True/None to pass.
    return run["output"].get("refund_id") is not None

tests:
  - name: refunds_a_real_charge
    builder: create_charge_then_refund
    builder_args:
      amount_cents: 4200
    # `context` is the same dict build() mutated -- here it pins the
    # tool call to the exact charge_id the builder provisioned, so the
    # test fails if the agent invents an id or refunds the wrong charge.
    expected_result: output.status == "refunded" and output.charge_id == context.charge_id
    expected_tool_calls:
      - billing.refund: params.charge_id == context.charge_id and invocations == 1

Passing state from build to cleanup

Mutate the context dict inside build(). The same dict is delivered to cleanup() as its second argument. The canonical pattern is to stash an id you provisioned in build, then DELETE that resource in cleanup so the test never leaves residue behind.

Referencing context from yaml assertions

The same context dict is also bound as context in expected_result and expected_tool_calls expressions. That closes the loop on dynamic fixtures: the builder mints an id, the agent receives it via the payload, and the assertion can confirm the agent passed that exact id to the tool call instead of inventing one or grabbing the wrong row.

# tests/builders/insert_then_query.py
import os, uuid, requests

def build(context, builder_args, test_name, payload, files):
    test_uuid = str(uuid.uuid4())
    requests.post(
        f"{os.environ['DB_API']}/rows",
        json={"id": test_uuid, "value": "hello"},
    )
    context["test_uuid"] = test_uuid
    return f"Fetch the row with id {test_uuid} and tell me its value."

def cleanup(run, context, builder_args):
    requests.delete(f"{os.environ['DB_API']}/rows/{context['test_uuid']}")

tests:
  - name: fetches_the_row_we_just_inserted
    builder: insert_then_query
    expected_result: output.row.id == context.test_uuid
    expected_tool_calls:
      - db.fetch_row: params.uuid == context.test_uuid and invocations == 1

A few notes:

• Empty for builder-less cases. Tests with no builder get context = {}, so a missing key fails the predicate rather than raising.
• Read after build, before cleanup. Assertions evaluate against the dict as it stands when build() returns. cleanup() still sees the same reference and can mutate it for its own bookkeeping, but those changes can't reach the assertions, since cleanup runs afterwards.
• Same syntax both sides. context.foo.bar[0] works identically in expected_result and inside a params or invocations conjunct.

cleanup() return contract

• Return True or None to pass. Use this for plain teardown that has no opinion on the agent output.
• Return False to fail the case. Useful for Python-level assertions that aren't expressible as a yaml expression. The result is AND-ed with the yaml-defined checks (expected_result, expected_tool_calls, expected_no_tool_calls).
• Always runs. cleanup fires even when the agent timed out or crashed, so teardown is reliable. Exceptions raised by cleanup are caught and reported as a failure with the traceback in failure_reason.

Other notes

• One re-import per invocation. Builders are reloaded fresh every time, so module-level state (caches, counters) resets between calls. Use this for builders that POST a fixture, since you don't want stale IDs leaking between cases.
• Sync or async. If build or cleanup returns a coroutine the runner awaits it.
• Same env as the agent. Builders run inside the test container, with the same environment variables, connectors, and network reachability. Authenticating to your own API works exactly as it does from a tool.
• Combine with files. If both builder and files are set, attached fixtures are merged into the builder's output. If the builder returns a dict with its own files key, both lists concatenate.
• Missing builders fail fast. Same pre-build check as files: a typo aborts the deploy gate before the test container even starts.

Coverage Report

connic test --coverage is a static, offline report. It reads agents/ and tests/ from disk and tells you which agents have tests and which tools those tests actually exercise. No backend call, no credentials, no test container. Safe to wire into a pre-commit hook or a doc-style CI job on every PR.

The model is intentionally simple:

• Every agent counts equally. One of ten agents fully covered is 10% overall, regardless of tool count. This keeps the headline number honest when one agent has 20 tools and another has 2.
• Per-agent score = covered tools / total tools. A tool counts as covered if it appears at least once in any of that agent's expected_tool_calls entries (bare or mapping form). To hit 100% on an agent, every one of its tools needs to show up in expected_tool_calls in at least one case.
• No test file → 0%. An agent without a corresponding tests/<agent>.yaml contributes 0 to the average.
• Tool-less agents → 100% if a test file exists. Sequential agents and orchestrators have nothing to cover at the tool level, so a single test file is enough.
• A/B variants are skipped. Test variants like support-test-fast share the base agent's tools and are excluded from the count.
• Discoverable tools count too. Both tools and discoverable_tools are part of the denominator. The auto-injected search_tools / use_tool markers are not.

Uncovered tools are listed beneath the table. For search-agent above: web.fetch, web.summarize.

Pair it with --json to get a machine-readable report ({overall, agents: [{name, type, has_tests, tools_total, tools_covered, uncovered_tools, percent}]}) you can pipe into a CI gate. For example, fail the build if overall coverage drops below a threshold. Unlike connic test, --coverage always exits 0; it's a report, not a gate.