Did it get worse? No
Nothing got worse compared with the reference. Every case could be judged. 1 case is suspended. The suite is unchanged from the reference. The system under test answered under the same configuration as the reference.
Configured as in the reference.
| Parameter | This run | Reference |
|---|---|---|
model | support-desk-2 | support-desk-2 |
provider | northwind-inhouse | northwind-inhouse |
temperature | 0.0 | 0.0 |
Nothing in this section.
Nothing in this section.
| Case | Why |
|---|---|
refund-status | the refund API is down, ticket 412 |
Nothing in this section.
Nothing in this section.
| Case | Check | What happened | Why |
|---|---|---|---|
how-do-i-return | starts_capitalised | Score unchanged at 1.000000. | pattern /^[A-Z]/ matched |
how-do-i-return | cost_budget | Score unchanged at 0.826446. | 0.004200 USD against a 0.020000 cap (within budget) |
how-do-i-return | latency_budget | Score unchanged at 0.816327. | 180.000 ms against a 800.000 cap (within budget) |
how-do-i-return | llm_rubric | Score unchanged at 0.894700. | mean of 3 samples (0.901700, 0.860300, 0.922100) |
how-do-i-return | contains | Score unchanged at 1.000000. | 'Northwind Support' found in the output |
how-do-i-return | length | Score unchanged at 1.000000. | 142 characters, within bounds (at most 400) |
is-it-waterproof | starts_capitalised | Score unchanged at 1.000000. | pattern /^[A-Z]/ matched |
is-it-waterproof | cost_budget | Score unchanged at 0.826446. | 0.004200 USD against a 0.020000 cap (within budget) |
is-it-waterproof | latency_budget | Score unchanged at 0.816327. | 180.000 ms against a 800.000 cap (within budget) |
is-it-waterproof | llm_rubric | Score unchanged at 0.910400. | mean of 3 samples (0.936900, 0.924900, 0.869400) |
is-it-waterproof | contains | Score unchanged at 1.000000. | 'Northwind Support' found in the output |
is-it-waterproof | length | Score unchanged at 1.000000. | 119 characters, within bounds (at most 400) |
where-is-my-order | starts_capitalised | Score unchanged at 1.000000. | pattern /^[A-Z]/ matched |
where-is-my-order | cost_budget | Score unchanged at 0.826446. | 0.004200 USD against a 0.020000 cap (within budget) |
where-is-my-order | latency_budget | Score unchanged at 0.816327. | 180.000 ms against a 800.000 cap (within budget) |
where-is-my-order | llm_rubric | Score unchanged at 0.891267. | mean of 3 samples (0.930900, 0.872900, 0.870000) |
where-is-my-order | contains | Score unchanged at 1.000000. | 'Northwind Support' found in the output |
where-is-my-order | length | Score unchanged at 1.000000. | 145 characters, within bounds (at most 400) |