Skip to main content
connic

Score real answers against your own rubric.
Know when quality slips.

A judge model scores your agent's real runs on your criteria and writes down why. Averages show which part slipped, and an alert goes out when quality drops below your threshold.

Tutor qualityAvg Score64%
Alert under 70%
#5c1e07ab live
Oct 16Oct 23Oct 30
Guides, doesn't solve 0/511/20

Hands over the full solution in the first reply.

How do I solve 3x + 7 = 22?

Subtract 7, then divide by 3: x = 5.

Check the answers that matter, the way your best reviewer would.

Write down what a good answer looks like. A judge applies it to real runs and leaves a score and a reason on each one.

A writing assistant drafts comments on student essays. A judge checks a quarter of them against the marking scheme and flags feedback a student couldn't act on.

English 10 · Essays
learn.example.org/english-10/essays/mila-k
Mila K. · submitted Tue 21:47 · 742 words
Should school start later?

Schools should start at 9:00, not 7:45, because teenagers learn better when they have slept. Most students in my year say they feel tired in the first lesson.

A sleep study found that students who started later slept 34 minutes more and their grades in the first lesson went up. Sports practice and clubs would have to move, but that can be planned.

Feedback draftNot sent
¶ 1
Your claim is clear in the first sentence. That's what band 4 asks for under Argument.
¶ 2
The sleep study supports your point. Name who ran it and when, so the reader can judge it.
Overall
Add more evidence to make your essay stronger.
Feedback quality9/15
Next step 1/5 · doesn't say which paragraph needs evidence

Parents email the admissions office, and the agent answers from the admissions policy. A judge compares each reply with the policy, so a wrong deadline shows up in the score and the alert.

Admissions
admissions.example.org/inbox/APP-3318
Admissions inbox
Elena Popescu08:12
Year 7 deadline for 2027
Tom Becker07:58
Open day times
Aylin DemirMon
Sibling place, Year 5
Year 7 deadline for 2027
Elena Popescu · 07:41
Hello, when do we have to apply for Year 7 starting in September 2027? We moved here from Cluj in August.
Sent by the admissions assistant · 08:12
Dear Ms Popescu, welcome! Applications for Year 7 starting in September 2027 close on 15 March 2027. You can apply online with your daughter's last school report and proof of address.
Admissions replies9/15
Matches the policy 0/5 · says 15 March, the admissions policy says 1 March

The teaching team scores last week's 312 tutor chats with a new judge first. Once its scores agree with their own spot checks, they let it score new chats automatically.

Tutor review
staff.example.org/tutor-review/week-43
Tutor review, week 43
New criterion: Guides, doesn't solve · judge on Manual
Ready to switch on
312
chats scored
50
checked by teachers
47
verdicts agree
ChatTopicTeacherJudgeAgree
4be1c0d7a93fFractionsGood hint5/5
9f03a6e15c28Linear equationsGave the answer0/5
d27b8e4f0a61PercentagesGood hint4/5
a61c93e0b7d2Area of a circleToo much help4/5
3e7f0c2d9b14RatiosGave the answer1/5

Disagreements go to Friday's team meeting.

See which part of an answer went wrong, and why.

Every criterion gets its own score and a written reason. A low total points straight to the part that slipped, right on the run.

Your criteria, your words
Give each criterion a name, a maximum and a sentence on what good looks like.
The judge reads the whole run
What came in, what went out, and what tools and retrieval returned on the way. The agent's instructions too, if you turn that on.
Run Details
7c41e9a02d5b Completed9/15
Judge Evaluations (1)
Admissions repliescompleted
60%(9/15)
Criteria Scores
Matches the policy0/5

Says applications close on 15 March 2027. The admissions policy returned by retrieval says 1 March 2027.

Answers the question5/5

Gives the deadline for the year group and start date the parent asked about.

Next step for the family4/5

Names the documents to upload but doesn't link the application form.

Overall Assessment

Friendly and complete, but the deadline is two weeks late. A family that trusts it would miss the real one.

Hear about a drop as soon as the average crosses your line.

Set a threshold and a window, such as the last 50 runs. Your team gets one alert in the app and by email, and the next one only after the average recovers.

Notifications

Low Judge Score: Tutor quality

Average dropped to 64% (threshold: 70%)

Online School · 2 minutes ago

Judge Alert: Tutor quality average dropped to 64% - Online School

Tutor quality average score dropped below threshold

Average: 64% (threshold: 70%)

Judge
Tutor quality
Agent
math-tutor
Average Score
12.8/20.0 (64%)
Threshold
70%
Total Evaluated
50
View Judge

Point a judge at the runs that matter.

Set up a judge in the dashboard or from an AI client connected to connic. It belongs to one agent in one environment, and you decide which of its runs it scores.

Only the runs you care about
A filter expression over context, input and output, such as context.course == 'algebra-1', picks the runs.
Score a share of them
An automatic judge scores 1 to 100% of the matching runs, so the sample rate also sets what judging costs.
Manual first, automatic later
On Manual, score a single run or every completed run in a timeframe, up to 1,000 at a time. Switch to Automatic once the scores match your own review.
Create Judge
Name
Tutor quality
Agent
math-tutor
Model
connic/claude-sonnet-5
System Prompt (optional)
Students are 13 to 15. Be strict about giving away answers, lenient about wording.
Scoring CriteriaAdd
Math is correct10
Every number and step in the reply is mathematically right.
Guides, doesn't solve5
Gives one step or a question, never the final answer, unless the student already found it.
Fits the unit5
Uses the methods of the current unit, such as checking by substitution in unit 4.
Filter Expression
context.course == 'algebra-1'
Trigger ModeAutomatic
Sample Rate25%

Frequently Asked Questions

The run's input, output, errors, context, status and duration, plus what its tools and retrieval returned. If you turn it on, it also reads the agent's instructions. It returns a score and a reason for each criterion and an overall assessment.

Each completed evaluation costs the same as starting an agent run, paid from your project credit. Failed evaluations cost nothing. The judge model's tokens come on top. Score only a share of the runs to keep both low.

Yes. Score a single past run, or every completed run of the agent in a timeframe, up to 1,000 at a time. That way you can check a new rubric against your own judgement before it scores new runs automatically.

Guardrails check a run while it happens and can block it. Tests run fixed cases before a release goes live. Judges score real runs after they finish and never change the result, so they track quality over time.