Fixtures & Mocks

File Attachments

Drop binary fixtures (PDFs, images, audio, anything the agent will receive in production) into tests/files/ and reference them by bare filename in the case's files: list. The runner reads each file, base64-encodes it, and attaches it under files. If payload is a JSON object (or comes from a builder returning a dict), its keys sit at the top level of context["payload"] next to files; otherwise the string is delivered as {message: payload}.

tests:
  - name: extracts_invoice_total
    payload: "extract the total amount as JSON"
    files:
      - invoice_acme.pdf
      - invoice_globex.pdf
    expected_result: output.total > 0

  # Files combine with a static payload (the prompt). They can also be
  # used with a builder -- attached files are merged with whatever the
  # builder returns.
  - name: classifies_receipt
    payload: "is this a meal or travel expense?"
    files:
      - receipt.jpg
    expected_result: 'output.category in ("meal", "travel")'

A few constraints worth knowing:

• Bare filenames only. No path separators, no ... The schema rejects anything that looks like a path.
• 25 MB upload budget. Code/config (everything outside tests/files/) is still capped at 5 MB; fixtures get the remaining headroom.
• Mime type is auto-detected from the extension via Python's mimetypes, falling back to application/octet-stream.
• Missing files fail fast. If a referenced fixture isn't in the upload, the deploy gate aborts before building the test image.

Dynamic Payload Builders

Some tests can't run against a static payload. You need a fresh fixture, a freshly minted database row, or a payload that references an ID that didn't exist a moment ago. Drop a Python module under tests/builders/ exposing build(context, builder_args, test_name, payload, files) (and optionally cleanup(run, context, builder_args)), point the case at it via the builder: field, and the runner will execute the pair once per invocation. build()'s return value (string or dict) becomes the agent input, replacing any static payload.

# tests/builders/create_charge_then_refund.py
import os, requests

def build(context, builder_args, test_name, payload, files):
    """Provision a fixture in your own API, then return the agent payload.

    context       dict       -- mutate to pass state to cleanup()
                                AND to expected_result / expected_tool_calls
    builder_args  dict       -- the yaml `builder_args`
    test_name     str        -- the yaml `name`
    payload       str | None -- the yaml `payload` (if any)
    files         list[str]  -- the yaml `files`
    """
    charge = requests.post(
        f"{os.environ['BILLING_API']}/charges",
        json={"amount_cents": builder_args["amount_cents"], "currency": "eur"},
    ).json()
    # Stash the id so cleanup() can tear the fixture down AND so the
    # yaml expressions can reference it via `context.charge_id`.
    context["charge_id"] = charge["id"]
    return {"charge_id": charge["id"], "instruction": "refund this charge"}


def cleanup(run, context, builder_args):
    """Tear down the fixture. Optionally add Python-level checks.

    Runs after every agent invocation -- pass, fail, or timeout --
    so external resources are always released.

    run["input"]    -- what the agent saw
    run["output"]   -- the agent's parsed output
    run["context"]  -- the run's run_context dict (run_id, agent_name,
                       connector_id, timestamp, plus anything middleware
                       or hooks added during the run)
    context         -- the dict you populated in build()
    builder_args    -- same dict that was passed to build()
    """
    requests.delete(f"{os.environ['BILLING_API']}/charges/{context['charge_id']}")
    # Return False to fail the case (in addition to yaml checks);
    # True/None to pass.
    return run["output"].get("refund_id") is not None

tests:
  - name: refunds_a_real_charge
    builder: create_charge_then_refund
    builder_args:
      amount_cents: 4200
    # `context` is the same dict build() mutated -- here it pins the
    # tool call to the exact charge_id the builder provisioned, so the
    # test fails if the agent invents an id or refunds the wrong charge.
    expected_result: output.status == "refunded" and output.charge_id == context.charge_id
    expected_tool_calls:
      - billing.refund: params.charge_id == context.charge_id and invocations == 1

Passing state from build to cleanup

Mutate the context dict inside build(). The same dict is delivered to cleanup() as its second argument. The canonical pattern is to stash an id you provisioned in build, then DELETE that resource in cleanup so the test never leaves residue behind.

Referencing context from yaml assertions

The same context dict is also bound as context in expected_result and expected_tool_calls expressions. That closes the loop on dynamic fixtures: the builder mints an id, the agent receives it via the payload, and the assertion can confirm the agent passed that exact id to the tool call instead of inventing one or grabbing the wrong row.

# tests/builders/insert_then_query.py
import os, uuid, requests

def build(context, builder_args, test_name, payload, files):
    test_uuid = str(uuid.uuid4())
    requests.post(
        f"{os.environ['DB_API']}/rows",
        json={"id": test_uuid, "value": "hello"},
    )
    context["test_uuid"] = test_uuid
    return f"Fetch the row with id {test_uuid} and tell me its value."

def cleanup(run, context, builder_args):
    requests.delete(f"{os.environ['DB_API']}/rows/{context['test_uuid']}")

tests:
  - name: fetches_the_row_we_just_inserted
    builder: insert_then_query
    expected_result: output.row.id == context.test_uuid
    expected_tool_calls:
      - db.fetch_row: params.uuid == context.test_uuid and invocations == 1

A few notes:

• Empty for builder-less cases. Tests with no builder get context = {}, so a missing key fails the predicate rather than raising.
• Read after build, before cleanup. Assertions evaluate against the dict as it stands when build() returns. cleanup() still sees the same reference and can mutate it for its own bookkeeping, but those changes can't reach the assertions, since cleanup runs afterwards.
• Same syntax both sides. context.foo.bar[0] works identically in expected_result and inside a params or invocations conjunct.

cleanup() return contract

• Return True or None to pass. Use this for plain teardown that has no opinion on the agent output.
• Return False to fail the case. Useful for Python-level assertions that aren't expressible as a yaml expression. The result is AND-ed with the yaml-defined checks (expected_result, expected_tool_calls, expected_no_tool_calls).
• Always runs. cleanup fires even when the agent timed out or crashed, so teardown is reliable. Exceptions raised by cleanup are caught and reported as a failure with the traceback in failure_reason.

Other notes

• One re-import per invocation. Builders are reloaded fresh every time, so module-level state (caches, counters) resets between calls. Use this for builders that POST a fixture, since you don't want stale IDs leaking between cases.
• Sync or async. If build or cleanup returns a coroutine the runner awaits it.
• Same env as the agent. Builders run inside the test container, with the same environment variables, connectors, and network reachability. Authenticating to your own API works exactly as it does from a tool.
• Combine with files. If both builder and files are set, attached fixtures are merged into the builder's output. If the builder returns a dict with its own files key, both lists concatenate.
• Missing builders fail fast. Same pre-build check as files: a typo aborts the deploy gate before the test container even starts.

Mocking Tools and Lifecycle Code

Tests normally execute the same custom tools, middleware, hooks, and guardrails as production. To isolate selected side effects, point a case at a Python module under tests/mocks/ with the mocks: field. That module can replace custom file tool results and individual lifecycle phases.

Replacement is opt-in per function. When a matching replacement exists, the runner calls it instead of the real function. By default, no match runs the real project code; enable the corresponding strict mock flag to fail before that eligible real function executes instead.

Tool results

For a tool ref like data.customer.add_customer the runner tries each name in order and uses the first one defined:

Middleware phases

Define middleware_before(content, context) to replace the agent's before middleware, or middleware_after(response, context) to replace after. Return the same value the real phase would return.

Tool hook phases

Hook replacements use the tool hierarchy plus a phase suffix. For the same tool, before resolves mock_data_customer_add_customer_hook_before → mock_data_customer_hook_before → mock_data_hook_before → mock_hook_before. The after ladder uses the same names ending in _hook_after. Non-alphanumeric runs in each tool-ref segment normalize to _, so api:weather-v2.lookup resolves mock_api_weather_v2_lookup_hook_before. Before handlers use (tool_name, params, context) → params; after handlers use (tool_name, params, result, context) → result.

Custom guardrails

For an input custom guardrail named domain_check, the runner tries guardrail_input_domain_check → guardrail_input → guardrail. Output guardrails use the equivalent guardrail_output_* ladder. Each handler uses (content, context) → GuardrailResult. Names are normalized for Python, so domain-check also resolves to domain_check. Only type: custom guardrails are eligible; built-in guardrails always run for real.

# tests/mocks/customer_mocks.py
from connic import GuardrailResult

# Tool result replacements use hierarchical mock_* names.
def mock_data_customer_add_customer(tool_name, params, context):
    return {"id": "cust_test_1", "name": params.get("name")}

# Middleware replacements use the real phase signatures.
def middleware_before(content, context):
    context["customer_id"] = "cust_test_1"
    return content

def middleware_after(response, context):
    return response

# Hook replacements use the tool hierarchy plus the phase suffix.
def mock_data_customer_add_customer_hook_before(tool_name, params, context):
    params["name"] = params["name"].strip()
    return params

def mock_hook_after(tool_name, params, result, context):
    return result

# Only type: custom guardrails are replaceable.
def guardrail_input_domain_check(content, context):
    return GuardrailResult(passed=True)

tests:
  - name: adds_a_customer_without_touching_the_db
    payload: '{"name": "Ada"}'
    mocks: customer_mocks
    strict_mocks: true
    strict_hook_mocks: true
    strict_middleware_mocks: true
    strict_guardrail_mocks: true
    # The agent's add_customer call is served by the mock, but it's still
    # recorded -- so you can assert the agent reached for it with the right
    # argument while never writing to a real datastore.
    expected_tool_calls:
      - data.customer.add_customer: params.name == "Ada"
    expected_result: output.id == "cust_test_1"

• Tool contract. Tool mocks use (tool_name, params, context) → result. tool_name is the full ref, so a broad mock can branch on which tool it replaces.
• Tool implementation eligibility. Only custom file tool implementations can be replaced. Predefined tools (db_find, web_search, trigger_agent, …) and api: tool implementations always run for real.
• Replacement only. A lifecycle mock replaces an existing middleware, hook, or configured custom guardrail phase; it does not add a phase that the project does not define. Custom guardrail files must still load successfully.
• Hook replacements are independent. Wherever agent hooks normally run, a hook phase can be replaced whether the tool result is real or mocked. Without a matching hook replacement, the real hook runs by default; strict_hook_mocks fails before it executes instead. Remote MCP tools do not run agent hooks, so hook replacements do not apply to them.
• Parameter validation. Mocked arguments are validated against the real tool's parameters — required arguments, types, and unknown arguments — so a malformed call fails the case. Defaulted parameters are optional.
• Tracing and assertions. A mocked call appears in the trace with a mocked pill and counts toward expected_tool_calls / expected_no_tool_calls.
• strict_mocks is tool-only. Set it on the case or in defaults to fail before an unmocked custom file tool executes. It does not govern middleware, hook, or guardrail replacements.
• Lifecycle strictness is explicit and independent. strict_hook_mocks, strict_middleware_mocks, and strict_guardrail_mocks each default to false and can be set in defaults or per case. Each fails before its configured eligible real phase executes without a matching replacement. A missing hook or middleware phase is exempt, as are missing guardrail phases and built-in guardrails.
• Uses the real context contract. Middleware, hook, and tool replacements receive the shared run context; custom guardrail replacements receive the same guardrail context view as the real check. Builder context is separate and is used by assertions, approval decisions, and cleanup.
• One re-import per invocation. Like builders, the mocks module is reloaded fresh every run, so module-level counters or caches reset between cases.
• Missing modules fail fast. A typo in mocks: aborts the deploy gate before the test container starts, the same pre-build check as files and builder.