Score real answers against your own rubric.
Know when quality slips.
A judge model scores your agent's real runs on your criteria and writes down why. Averages show which part slipped, and an alert goes out when quality drops below your threshold.
Check the answers that matter, the way your best reviewer would.
Write down what a good answer looks like. A judge applies it to real runs and leaves a score and a reason on each one.
A writing assistant drafts comments on student essays. A judge checks a quarter of them against the marking scheme and flags feedback a student couldn't act on.
Parents email the admissions office, and the agent answers from the admissions policy. A judge compares each reply with the policy, so a wrong deadline shows up in the score and the alert.
The teaching team scores last week's 312 tutor chats with a new judge first. Once its scores agree with their own spot checks, they let it score new chats automatically.
See which part of an answer went wrong, and why.
Every criterion gets its own score and a written reason. A low total points straight to the part that slipped, right on the run.
- Your criteria, your words
- Give each criterion a name, a maximum and a sentence on what good looks like.
- The judge reads the whole run
- What came in, what went out, and what tools and retrieval returned on the way. The agent's instructions too, if you turn that on.
Hear about a drop as soon as the average crosses your line.
Set a threshold and a window, such as the last 50 runs. Your team gets one alert in the app and by email, and the next one only after the average recovers.
Point a judge at the runs that matter.
Set up a judge in the dashboard or from an AI client connected to connic. It belongs to one agent in one environment, and you decide which of its runs it scores.
- Only the runs you care about
- A filter expression over
context,inputandoutput, such ascontext.course == 'algebra-1', picks the runs. - Score a share of them
- An automatic judge scores 1 to 100% of the matching runs, so the sample rate also sets what judging costs.
- Manual first, automatic later
- On Manual, score a single run or every completed run in a timeframe, up to 1,000 at a time. Switch to Automatic once the scores match your own review.