Skip to main content
connic

Catch a bad agent change.
Before it reaches a customer.

Write down what your agent must do and must never do. connic checks it before every release and stops the release when a case fails.

~/help-agents
$ connic test
FAILoutage-triage::reply_gives_repair_time0/1
→ Failure details:
✗ outage-triage::reply_gives_repair_time: expected_result failed: '"11:30" in output'
Output: There's a known outage in 10437. Technicians are on site.
On deploy:Run tests1 failedDeploy to Productionskipped

Protect what your customers rely on, case by case.

Write one case for each thing customers would notice if it broke. connic runs every case before each release.

The agent confirms the outage, gives the repair time and pages the network team once. Another case fails if a slow Wi-Fi question pages anyone.

Home Internet · Help
help.example.net/contact
Home InternetMy contractBillsHelp
Disruption in Berlin 10437 since 06:52. Technicians are on site.
Support chat
Replies in a few seconds

No internet since 7:10, the router shows a red DSL light. Postcode 10437.

There's a known outage in 10437 since 06:52. Technicians are on site and expect it fixed by 11:30. I've added your line to the incident, so we'll text you when it's back.

PASSconfirmed_outage_pages_oncall_once5/5 runs
Write a message

A credit needs approval from billing first. One case rejects it on purpose and checks that the customer still gets a clear explanation.

Inbox
mail.example.org/inbox/48213
Compose
Inbox3
Starred
Sent
Archive
Your October bill and the second €23.40 charge
HI
Home Internet Billing <billing@example.net>Tue 14:26
to Sven Richter

Hi Sven,

our billing team checked your October bill. The second charge of €23.40 is the TV add-on booked on 12 October, so it isn't a double charge and we can't credit it.

If you didn't book the add-on, reply to this email and we'll cancel it and refund October free of charge.

Your Home Internet team

Reply
PASScredit_rejection_gets_an_explanation1/1

Before a new model goes live, real customer photos run five times each. Four of five runs have to pass.

9:41
Home Internet
Support

Which light is the problem?

The DSL light is red, so your line has no signal. Check that the grey cable sits firmly in the wall socket, then restart the router. Still red after 10 minutes? I can book a technician for you.

PASSphoto_red_dsl_light5/5
Message

Test with your real tools, minus the side effects.

Tests use your real tools unless you swap one for a test version. Swap the ones that page people or move money, and the trace shows what would have happened.

Test keys, not live ones
Run the tests in a staging environment with its own keys, connections and data.
Messages stay unsent
Messages the agent sends through your connectors are recorded during a test, not delivered.
Run Details
Completed
SpanData
LLM Agentoutage-triageok5214.6ms
Tool Calloknetwork.outage_status184.2ms
Tool Calloktickets.link_line96.8ms
Tool Callokmockedoncall.page0.4ms
Tool Callokmockedsms-updates0.3ms

Write down what must happen, and what must never happen.

A case is a few lines of YAML in tests/. The same file runs before every release, on pull requests and with connic test in your own CI.

tests/outage-triage.yaml
version: "1.0"

defaults:
  mocks: triage_mocks

tests:
  - name: confirmed_outage_pages_oncall_once
    payload: "No internet since 7:10, red DSL light. Postcode 10437"
    runs: 5
    success_threshold: 80
    expected_tool_calls:
      - network.outage_status: params.postcode == "10437"
      - oncall.page: invocations == 1

  - name: slow_wifi_is_not_an_outage
    payload: "Wi-Fi is slow in the bedroom, fine in the living room"
    expected_no_tool_calls:
      - oncall.page

  - name: credit_rejection_gets_an_explanation
    payload: "I was charged twice in October, invoice 2026-10-48213"
    approval_decisions:
      - tool: billing.issue_credit
        decision: reject
        reason: The second charge is the TV add-on
    expected_result: status == "completed" and "add-on" in output
  1. mocks: triage_mocks

    Side effects stay in the test

    tests/mocks/triage_mocks.py replaces oncall.page, so no real phone rings.

  2. runs: 5

    Five runs, four must pass

    For answers that vary, the case passes when at least 80% of the runs pass.

  3. expected_tool_calls

    The right tools with the right input

    Each listed tool must run. The expression checks its input and how often it ran.

  4. expected_no_tool_calls

    Tools the agent must leave alone

    One call to oncall.page fails the case.

  5. approval_decisions

    Scripted human decisions

    Approve, reject or time out a waiting tool, then check what the agent does next.

Frequently Asked Questions

Let a case run several times and set how many runs must pass. With five runs and a threshold of 80%, the case passes when four of the five pass.

Only where you let them. Tools run for real unless you swap them for a test version, and messages through your connectors are recorded instead of sent. Give the test environment test keys for every other service.

Changes pushed through Git always run the tests. People allowed to release by hand can skip them for a single manual release. You choose per environment which groups may release by hand, and project owners always can.

No. Every release from the command line or a Git push runs the tests first. On the pro and enterprise plans, connic also reports a check on pull and merge requests in GitHub and GitLab. If you want, the tests run in your own CI too.