Evaluation report — expense-triage

Customer

northwind

Suite

expense-triage

This run

staging — 2026-09-08T07:43:12.821659+00:00

Reference

staging — 2026-09-08T07:42:54.701842+00:00

Code version

b44ad6ffbf22cc9dabdf54ec7e453c95e8259d93-dirty — uncommitted changes were present, so this run cannot be reproduced from the repository

Did it get worse? No

Nothing got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

regressed 0
errored 0
new 0
missing 0
improved 2
unchanged 30

Overall

Measure	Result	What happened	Why
`precision`	0.800000 / 0.600000	20 counted · 0 suspended · 0 not judged	precision 0.800000 = 8/10 (20 counted, 0 suspended, 0 could not be judged)
`precision[group=hotel]`	1.000000 / 0.600000	3 counted · 0 suspended · 0 not judged	precision 1.000000 = 2/2 (3 counted, 0 suspended, 0 could not be judged)
`precision[group=meal]`	1.000000 / 0.600000	7 counted · 0 suspended · 0 not judged	precision 1.000000 = 4/4 (7 counted, 0 suspended, 0 could not be judged)
`precision[group=taxi]`	1.000000 / 0.600000	4 counted · 0 suspended · 0 not judged	precision 1.000000 = 1/1 (4 counted, 0 suspended, 0 could not be judged)
`precision[group=tools]`	0.500000 / 0.600000	3 counted · 0 suspended · 0 not judged	precision 0.500000 = 1/2 (3 counted, 0 suspended, 0 could not be judged)
`precision[group=travel]`	0.000000 / 0.600000	3 counted · 0 suspended · 0 not judged	precision 0.000000 = 0/1 (3 counted, 0 suspended, 0 could not be judged)
`accuracy`	0.900000 / 0.700000	20 counted · 0 suspended · 0 not judged	accuracy 0.900000 = 18/20 (20 counted, 0 suspended, 0 could not be judged)
`accuracy[group=hotel]`	1.000000 / 0.700000	3 counted · 0 suspended · 0 not judged	accuracy 1.000000 = 3/3 (3 counted, 0 suspended, 0 could not be judged)
`accuracy[group=meal]`	1.000000 / 0.700000	7 counted · 0 suspended · 0 not judged	accuracy 1.000000 = 7/7 (7 counted, 0 suspended, 0 could not be judged)
`accuracy[group=taxi]`	1.000000 / 0.700000	4 counted · 0 suspended · 0 not judged	accuracy 1.000000 = 4/4 (4 counted, 0 suspended, 0 could not be judged)
`accuracy[group=tools]`	0.666667 / 0.700000	3 counted · 0 suspended · 0 not judged	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)
`accuracy[group=travel]`	0.666667 / 0.700000	3 counted · 0 suspended · 0 not judged	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)

Some measures are below their threshold and are still not reported as having got worse: they were below it in the reference too. The threshold says the system does not meet the bar; the comparison says it has not moved. Both are true, and only movement decides the answer above.

What got worse (0)

Nothing in this section.

What could not be judged (0)

Nothing in this section.

What is set aside (0)

Nothing in this section.

What was added or removed (0)

Nothing in this section.

What got better (2)

Case	Check	What happened	Why
`dinner_client`	`agrees_with_mark`	Went from failing to passing (0.000000 → 0.600000).	mean of 5 samples (1.000000, 1.000000, 0.000000, 1.000000, 0.000000)
	`precision[group=meal]`	Score rose from 0.800000 to 1.000000 — beyond the noise of this check (0.666667–0.800000 across 5 samples).	precision 1.000000 = 4/4 (7 counted, 0 suspended, 0 could not be judged)

What stayed the same (30)

Case	Check	What happened	Why
`amount_over_cap`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`book_technical`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`coffee_two`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`courier_urgent`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`dinner_solo`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`duplicate_claim`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`hotel_capped`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`hotel_over`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`late_taxi`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`lunch_team`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`meal_no_note`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`monitor_home`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`no_receipt`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`parking_airport`	`agrees_with_mark`	Score unchanged at 0.000000.	mean of 5 samples (0.000000, 0.000000, 0.000000, 0.000000, 0.000000)
`software_seat`	`agrees_with_mark`	Score unchanged at 0.000000.	mean of 5 samples (0.000000, 0.000000, 0.000000, 0.000000, 0.000000)
`taxi_long`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`taxi_short`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`train_standard`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`weekend_bar`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
	`accuracy[group=tools]`	Score unchanged at 0.666667.	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)
	`precision[group=travel]`	Score unchanged at 0.000000.	precision 0.000000 = 0/1 (3 counted, 0 suspended, 0 could not be judged)
	`precision[group=tools]`	Score unchanged at 0.500000.	precision 0.500000 = 1/2 (3 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=hotel]`	Score unchanged at 1.000000.	accuracy 1.000000 = 3/3 (3 counted, 0 suspended, 0 could not be judged)
	`precision`	Score unchanged at 0.800000.	precision 0.800000 = 8/10 (20 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=meal]`	Score unchanged at 1.000000.	accuracy 1.000000 = 7/7 (7 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=taxi]`	Score unchanged at 1.000000.	accuracy 1.000000 = 4/4 (4 counted, 0 suspended, 0 could not be judged)
	`precision[group=hotel]`	Score unchanged at 1.000000.	precision 1.000000 = 2/2 (3 counted, 0 suspended, 0 could not be judged)
	`precision[group=taxi]`	Score unchanged at 1.000000.	precision 1.000000 = 1/1 (4 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=travel]`	Score unchanged at 0.666667.	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)
	`accuracy`	Score unchanged at 0.900000.	accuracy 0.900000 = 18/20 (20 counted, 0 suspended, 0 could not be judged)