Test a change on real requests.
Switch when quality and cost add up.
connic sends a share of real requests to a new version of your agent and compares answer quality, cost and speed with the current one. Set a failure limit, and a bad version pauses itself.
Try a new prompt, model or tool without betting the product on it.
A new version gets a share of real requests while the current one keeps the rest. Your users see one version, and you see how each one did.
Half of the help questions go to a prompt that names the exact menu path. A judge scores each answer, and the test says when the new prompt is better.
Every message in the same chat goes to the same version. A user never sees the tone change halfway through a conversation.
A new export tool handles 20% of export requests. When too many of its recent runs fail, the test pauses and every request goes back to the current agent.
Get a clear answer on quality, cost and speed.
The test page puts the current version and the new one side by side. In Confidence mode, connic recommends the new version, the current one or neither.
- Know the run count up front
- connic works out how many runs each side needs before the start and locks that plan.
- Check the page as often as you like
- Decisions happen only at planned checkpoints, so looking early doesn't skew the result.
A bad version stops itself.
Set a failure limit, a minimum judge score or both. When the new version crosses one, the test pauses and requests go back to the current agent.
- Pause on failures
- connic watches the latest finished runs and pauses once too many of them fail.
- Pause on quality
- connic averages the latest judge scores and pauses when they drop below your minimum.
Start a variant by copying one file.
Copy the base agent, change what you want to test and name the copy {base-agent}-test-{name}. Deploy both, then start the test from the base agent's page.
- The name does the wiring
- Everything before -test- must match an agent in the same deployment, or the deployment fails.
- Change one thing at a time
- Swap the model, the prompt or the tools, such as tools/export_v2.py. One change per variant keeps the result readable.
version: "1.0"
name: help-assistant
model: connic/gpt-5.6-terra
description: "Answers how-to questions inside the invoicing app"
system_prompt: |
Answer with the exact menu path and one example.
If a question is about a specific invoice, look it up first.
tools:
- help_center.search
- invoices.lookupversion: "1.0"
name: help-assistant-test-small-model
model: connic/gpt-5.6-luna
description: "Answers how-to questions inside the invoicing app"
system_prompt: |
Answer with the exact menu path and one example.
If a question is about a specific invoice, look it up first.
tools:
- help_center.search
- invoices.lookup