Fixtures & Mocks

File Attachments

Put agent input files such as PDFs, images, and audio in tests/files/ and reference them by bare filename in the case's files: list. Each file is base64-encoded and attached under files. If payload is a JSON object, or a builder returns a dict, its keys remain at the top level beside files. Other payloads are delivered as {message: payload}.

tests:
  - name: extracts_invoice_total
    payload: "extract the total amount as JSON"
    files:
      - invoice_acme.pdf
      - invoice_globex.pdf
    expected_result: output.total > 0

  # Files combine with a static payload (the prompt). They can also be
  # used with a builder -- attached files are merged with whatever the
  # builder returns.
  - name: classifies_receipt
    payload: "is this a meal or travel expense?"
    files:
      - receipt.jpg
    expected_result: 'output.category in ("meal", "travel")'

A few constraints worth knowing:

• Bare filenames only. No path separators, no ... The schema rejects anything that looks like a path.
• 25 MB upload budget. Code/config (everything outside tests/files/) is still capped at 5 MB; fixtures get the remaining headroom.
• Mime type is auto-detected from the extension via Python's mimetypes, falling back to application/octet-stream.
• Missing files stop the suite. Every filename in files: must exist before agent execution begins.

Dynamic Payload Builders

Use a builder when agent input depends on a fixture, database row, or generated identifier. Add a Python module under tests/builders/ with build(context, builder_args, test_name, payload, files) and optional cleanup(run, context, builder_args), then reference it with builder:. These functions run once per invocation. build()'s string or dict return value becomes the agent input and replaces any static payload.

import os, requests

def build(context, builder_args, test_name, payload, files):
    """Provision a fixture in your own API, then return the agent payload.

    context       dict       -- mutate to pass state to cleanup()
                                AND to expected_result / expected_tool_calls
    builder_args  dict       -- the yaml `builder_args`
    test_name     str        -- the yaml `name`
    payload       str | None -- the yaml `payload` (if any)
    files         list[str]  -- the yaml `files`
    """
    charge = requests.post(
        f"{os.environ['BILLING_API']}/charges",
        json={"amount_cents": builder_args["amount_cents"], "currency": "eur"},
    ).json()
    # Stash the id so cleanup() can tear the fixture down AND so the
    # yaml expressions can reference it via `context.charge_id`.
    context["charge_id"] = charge["id"]
    return {"charge_id": charge["id"], "instruction": "refund this charge"}


def cleanup(run, context, builder_args):
    """Tear down the fixture. Optionally add Python-level checks.

    Runs after every agent invocation -- pass, fail, or timeout --
    so external resources are always released.

    run["input"]    -- what the agent saw
    run["output"]   -- the agent's parsed output
    run["context"]  -- the run's run_context dict (run_id, agent_name,
                       connector_id, timestamp, plus anything middleware
                       or hooks added during the run)
    context         -- the dict you populated in build()
    builder_args    -- same dict that was passed to build()
    """
    requests.delete(f"{os.environ['BILLING_API']}/charges/{context['charge_id']}")
    # Return False to fail the case (in addition to yaml checks);
    # True/None to pass.
    return run["output"].get("refund_id") is not None

tests:
  - name: refunds_a_real_charge
    builder: create_charge_then_refund
    builder_args:
      amount_cents: 4200
    # `context` is the same dict build() mutated -- here it pins the
    # tool call to the exact charge_id the builder provisioned, so the
    # test fails if the agent invents an id or refunds the wrong charge.
    expected_result: output.status == "refunded" and output.charge_id == context.charge_id
    expected_tool_calls:
      - billing.refund: params.charge_id == context.charge_id and invocations == 1

Passing state from build to cleanup

Mutate the context dict inside build(). The same dict is delivered to cleanup() as its second argument. The canonical pattern is to stash an id you provisioned in build, then DELETE that resource in cleanup so the test never leaves residue behind.

Referencing context from yaml assertions

The same context dict is also bound as context in expected_result and expected_tool_calls expressions. That closes the loop on dynamic fixtures: the builder mints an id, the agent receives it via the payload, and the assertion can confirm the agent passed that exact id to the tool call instead of inventing one or grabbing the wrong row.

import os, uuid, requests

def build(context, builder_args, test_name, payload, files):
    test_uuid = str(uuid.uuid4())
    requests.post(
        f"{os.environ['DB_API']}/rows",
        json={"id": test_uuid, "value": "hello"},
    )
    context["test_uuid"] = test_uuid
    return f"Fetch the row with id {test_uuid} and tell me its value."

def cleanup(run, context, builder_args):
    requests.delete(f"{os.environ['DB_API']}/rows/{context['test_uuid']}")

tests:
  - name: fetches_the_row_we_just_inserted
    builder: insert_then_query
    expected_result: output.row.id == context.test_uuid
    expected_tool_calls:
      - db.fetch_row: params.uuid == context.test_uuid and invocations == 1

A few notes:

• Empty for builder-less cases. Tests with no builder get context = {}, so a missing key fails the predicate rather than raising.
• Read after build, before cleanup. Assertions evaluate against the dict as it stands when build() returns. cleanup() still sees the same reference and can mutate it for its own bookkeeping, but those changes can't reach the assertions, since cleanup runs afterwards.
• Same syntax both sides. context.foo.bar[0] works identically in expected_result and inside a params or invocations conjunct.

cleanup() return contract

• Return True or None to pass. Use this for plain teardown that has no opinion on the agent output.
• Return False to fail the case. Useful for Python-level assertions that aren't expressible as a yaml expression. The result is AND-ed with the yaml-defined checks (expected_result, expected_tool_calls, expected_no_tool_calls).
• Always runs. Cleanup runs after timeouts and agent errors. A cleanup exception fails the case and appears in failure_reason.

Other notes

• State resets per invocation. Module-level caches and counters do not persist between builder calls.
• Sync or async. build and cleanup may return a value or a coroutine.
• Selected test environment. Builders use the environment variables and network access configured for the agent invocation.
• Combine with files. If both builder and files are set, attached fixtures are merged into the builder's output. If the builder returns a dict with its own files key, both lists concatenate.
• Missing builders stop the suite. A referenced builder module must exist before agent execution begins.

Mocking Tools and Lifecycle Code

Tests execute configured custom tools, middleware, hooks, and guardrails unless a case replaces them. Automatic outbound connectors are suppressed. Calls to agent-tool and middleware outbound connectors are recorded for assertions but never delivered. To isolate other side effects, point mocks: at a Python module under tests/mocks/. That module can replace custom file tool results and individual lifecycle phases.

Replacement is opt-in per function. A matching replacement runs instead of the configured function. Without a match, the configured project code runs unless the corresponding strict mock flag requires a replacement.

Tool results

For a tool ref like data.customer.add_customer, the first defined name in this order is used:

Middleware phases

Define middleware_before(content, context) to replace the agent's before middleware, or middleware_after(response, context) to replace after. Return the same value the real phase would return.

Tool hook phases

Hook replacements use the tool hierarchy plus a phase suffix. For the same tool, before resolves mock_data_customer_add_customer_hook_before → mock_data_customer_hook_before → mock_data_hook_before → mock_hook_before. The after ladder uses the same names ending in _hook_after. Non-alphanumeric runs in each tool-ref segment normalize to _, so api:weather-v2.lookup resolves mock_api_weather_v2_lookup_hook_before. Before handlers use (tool_name, params, context) → params; after handlers use (tool_name, params, result, context) → result.

Custom guardrails

For an input custom guardrail named domain_check, resolution follows guardrail_input_domain_check → guardrail_input → guardrail. Output guardrails use the equivalent guardrail_output_* order. Each handler uses (content, context) → GuardrailResult. Names are normalized for Python, so domain-check also resolves to domain_check. Only type: custom guardrails are eligible; built-in guardrails are not replaced.

from connic import GuardrailResult

# Tool result replacements use hierarchical mock_* names.
def mock_data_customer_add_customer(tool_name, params, context):
    return {"id": "cust_test_1", "name": params.get("name")}

# Middleware replacements use the real phase signatures.
def middleware_before(content, context):
    context["customer_id"] = "cust_test_1"
    return content

def middleware_after(response, context):
    return response

# Hook replacements use the tool hierarchy plus the phase suffix.
def mock_data_customer_add_customer_hook_before(tool_name, params, context):
    params["name"] = params["name"].strip()
    return params

def mock_hook_after(tool_name, params, result, context):
    return result

# Only type: custom guardrails are replaceable.
def guardrail_input_domain_check(content, context):
    return GuardrailResult(passed=True)

tests:
  - name: adds_a_customer_without_touching_the_db
    payload: '{"name": "Ada"}'
    mocks: customer_mocks
    strict_mocks: true
    strict_hook_mocks: true
    strict_middleware_mocks: true
    strict_guardrail_mocks: true
    # The agent's add_customer call is served by the mock, but it's still
    # recorded -- so you can assert the agent reached for it with the right
    # argument while never writing to a real datastore.
    expected_tool_calls:
      - data.customer.add_customer: params.name == "Ada"
    expected_result: output.id == "cust_test_1"

• Tool contract. Tool mocks use (tool_name, params, context) → result. tool_name is the full ref, so a broad mock can branch on which tool it replaces.
• Tool implementation eligibility. Only custom file tool implementations can be replaced. Predefined tools (db_find, web_search, trigger_agent, …) and api: tool implementations run for real. Agent-tool outbound connector calls and middleware outbound connector calls made through send_connector are exceptions: tests record them as mocked and skip delivery.
• Replacement only. A lifecycle mock replaces an existing middleware, hook, or configured custom guardrail phase; it does not add a phase that the project does not define. Custom guardrail files must still load successfully.
• Hook replacements are independent. Wherever agent hooks normally run, a hook phase can be replaced whether the tool result is real or mocked. Without a matching hook replacement, the real hook runs by default; strict_hook_mocks fails before it executes instead. Remote MCP tools do not run agent hooks, so hook replacements do not apply to them.
• Parameter validation. Mocked arguments are validated against the tool's required arguments, types, and accepted names, so a malformed call fails the case. Defaulted parameters are optional.
• Tracing and assertions. A mocked call appears in the trace with a mocked pill and counts toward expected_tool_calls / expected_no_tool_calls.
• strict_mocks is tool-only. Set it on the case or in defaults to fail before an unmocked custom file tool executes. It does not govern middleware, hook, or guardrail replacements.
• Lifecycle strictness is explicit and independent. strict_hook_mocks, strict_middleware_mocks, and strict_guardrail_mocks each default to false and can be set in defaults or per case. Each fails before its configured eligible real phase executes without a matching replacement. A missing hook or middleware phase is exempt, as are missing guardrail phases and built-in guardrails.
• Uses the real context contract. Middleware, hook, and tool replacements receive the shared run context; custom guardrail replacements receive the same guardrail context view as the real check. Builder context is separate and is used by assertions, approval decisions, and cleanup.
• State resets per invocation. Module-level counters and caches do not persist between mock calls.
• Missing modules stop the suite. A module referenced by mocks: must exist before agent execution begins.