Skip to main content
connic

Test a change on real requests.
Switch when quality and cost add up.

connic sends a share of real requests to a new version of your agent and compares answer quality, cost and speed with the current one. Set a failure limit, and a bad version pauses itself.

small-modelVariant recommended
Control 70%
Variant 30%
Avg Judge Score
91.2%
control 91.5%
Average Token Cost
€0.0039
control €0.0112
P50 Duration
1.4s
control 3.2s
Judge quality effect-0.3 ppNon-inferiority Margin 3.0 ppEvidence threshold met · 1,184 control and 507 variant runs

Try a new prompt, model or tool without betting the product on it.

A new version gets a share of real requests while the current one keeps the rest. Your users see one version, and you see how each one did.

Half of the help questions go to a prompt that names the exact menu path. A judge scores each answer, and the test says when the new prompt is better.

Invoicing
app.example.com/invoices?status=overdue
InvoicingInvoicesCustomersReportsSettingsNS
Invoices
INV-2026-0412Bäckerei Lindner€1,240.00Overdue 9 days
INV-2026-0409Studio Kranz€380.50Overdue 3 days
INV-2026-0415Hofmann Logistik€2,915.75Due Nov 14
INV-2026-0416Nordlicht Café€96.20Paid
Ask Invoicing

How do I remind customers about overdue invoices?

Go to Settings → Reminders and turn on Overdue reminders. Set the first one to 7 days after the due date. You can also send one now: open INV-2026-0412 and click Send reminder.

exact-stepsVariant recommended
Judge score 92.4% vs 78.1% on the current prompt
Ask a question

Every message in the same chat goes to the same version. A user never sees the tone change halfway through a conversation.

Invoicing
app.example.com/customers/hofmann-logistik
InvoicingInvoicesCustomersReportsSettingsNS
Hofmann Logistik
Open
€2,915.75
Paid this year
€18,402.10
Payment terms
14 days
Billing contact: ap@hofmann-logistik.example.com
Ask Invoicing

Can I give Hofmann 30 days to pay?

Yes. Open the customer, then Payment terms, and choose 30 days. New invoices use it.

And the open invoice?

That one keeps 14 days. Edit INV-2026-0415 and change its due date to Dec 1.

short-answersRunning
Randomized by session: both runs of this chat went to the variant
Ask a question

A new export tool handles 20% of export requests. When too many of its recent runs fail, the test pauses and every request goes back to the current agent.

Invoicing
app.example.com/reports/export
InvoicingInvoicesCustomersReportsSettingsNS
Export for your accountant
PeriodMarch 2026
Invoices38
Send totax@example.com
Ask Invoicing

Send my March invoices to my accountant.

Done. 38 invoices from March went to tax@example.com as one ZIP file.

export-v2Paused
Auto-paused: variant failure rate 14% exceeded threshold 10%. Requests go to help-assistant.
Ask a question

Get a clear answer on quality, cost and speed.

The test page puts the current version and the new one side by side. In Confidence mode, connic recommends the new version, the current one or neither.

Know the run count up front
connic works out how many runs each side needs before the start and locks that plan.
Check the page as often as you like
Decisions happen only at planned checkpoints, so looking early doesn't skew the result.
small-modelPausedConclude
Decision StatusVariant recommended
-0.3 pp
Judge quality effect
Non-inferiority Margin 3.0 pp
Completed looks 3 of 4

Evidence threshold met

A planned look reached the efficacy boundary.

Statistics
1,184
Completed Control Runs
507
Completed Variant Runs
MetricControlVariant
Average Token Cost€0.0112€0.0039
Avg Duration3.4s1.6s
P50 Duration3.2s1.4s
P95 Duration6.8s3.2s
Avg Judge Score91.5%91.2%
Success Rate99.4%99.1%

A bad version stops itself.

Set a failure limit, a minimum judge score or both. When the new version crosses one, the test pauses and requests go back to the current agent.

Pause on failures
connic watches the latest finished runs and pauses once too many of them fail.
Pause on quality
connic averages the latest judge scores and pauses when they drop below your minimum.
export-v2PausedStartConclude

Paused by failure guardrail

Auto-paused: variant failure rate 14% exceeded threshold 10%

Configuration
AnalysisExploratory
Variant Agenthelp-assistant-test-export-v2
Traffic20%
Min Sample Size100
Failure Auto-PauseEnabled
Failure Threshold10%
Rolling Window50 runs
Quality Auto-PauseEnabled
Quality JudgeAnswer quality
Quality Floor80%
Quality Window10 of 20 evaluations

Start a variant by copying one file.

Copy the base agent, change what you want to test and name the copy {base-agent}-test-{name}. Deploy both, then start the test from the base agent's page.

The name does the wiring
Everything before -test- must match an agent in the same deployment, or the deployment fails.
Change one thing at a time
Swap the model, the prompt or the tools, such as tools/export_v2.py. One change per variant keeps the result readable.
agents/help-assistant.yaml
version: "1.0"

name: help-assistant
model: connic/gpt-5.6-terra
description: "Answers how-to questions inside the invoicing app"
system_prompt: |
  Answer with the exact menu path and one example.
  If a question is about a specific invoice, look it up first.
tools:
  - help_center.search
  - invoices.lookup
agents/help-assistant-test-small-model.yaml
version: "1.0"

name: help-assistant-test-small-model
model: connic/gpt-5.6-luna
description: "Answers how-to questions inside the invoicing app"
system_prompt: |
  Answer with the exact menu path and one example.
  If a question is about a specific invoice, look it up first.
tools:
  - help_center.search
  - invoices.lookup

Frequently Asked Questions

No. In Confidence mode, connic plans the test before it starts and then recommends the new version, the current one or neither. Exploratory mode shows the same numbers without a recommendation, for a quick look.

It depends on how big a difference you want to detect, how much traffic the new version gets and how much the results vary. connic shows the number of runs each side needs before you start the test.

Yes. A judge can be the main measure of a confidence test or pause a version when its scores drop. In both cases the judge must score every run automatically.

Exploratory tests stay editable while they run or are paused. A confidence test locks its plan, traffic split and safety settings when it first starts, and connic keeps its records as evidence.