Did it get worse? No
Nothing got worse compared with the reference. Every case could be judged. 1 case is suspended. The suite is unchanged from the reference. The system under test answered under the same configuration as the reference.
Configured as in the reference.
| Parameter | This run | Reference |
|---|---|---|
max_tokens | 512 | 512 |
model | gpt-4o-mini | gpt-4o-mini |
provider | openai | openai |
temperature | 0.0 | 0.0 |
Nothing in this section.
Nothing in this section.
| Case | Why |
|---|---|
refund-status | the refund API is down, ticket 412 |
Nothing in this section.
Nothing in this section.
| Case | Check | What happened | Why |
|---|---|---|---|
how-do-i-return | starts_capitalised | Score unchanged at 1.000000. | pattern /^[A-Z]/ matched |
how-do-i-return | cost_budget | Score unchanged at 0.989609. | 0.000210 USD against a 0.020000 cap (within budget) |
how-do-i-return | latency_budget | Score unchanged at 0.677966. | 380.000 ms against a 800.000 cap (within budget) |
how-do-i-return | contains | Score unchanged at 1.000000. | 'Northwind Support' found in the output |
how-do-i-return | length | Score unchanged at 1.000000. | 142 characters, within bounds (at most 400) |
is-it-waterproof | starts_capitalised | Score unchanged at 1.000000. | pattern /^[A-Z]/ matched |
is-it-waterproof | cost_budget | Score unchanged at 0.989609. | 0.000210 USD against a 0.020000 cap (within budget) |
is-it-waterproof | latency_budget | Score unchanged at 0.677966. | 380.000 ms against a 800.000 cap (within budget) |
is-it-waterproof | contains | Score unchanged at 1.000000. | 'Northwind Support' found in the output |
is-it-waterproof | length | Score unchanged at 1.000000. | 119 characters, within bounds (at most 400) |
where-is-my-order | starts_capitalised | Score unchanged at 1.000000. | pattern /^[A-Z]/ matched |
where-is-my-order | cost_budget | Score unchanged at 0.989609. | 0.000210 USD against a 0.020000 cap (within budget) |
where-is-my-order | latency_budget | Score unchanged at 0.677966. | 380.000 ms against a 800.000 cap (within budget) |
where-is-my-order | contains | Score unchanged at 1.000000. | 'Northwind Support' found in the output |
where-is-my-order | length | Score unchanged at 1.000000. | 145 characters, within bounds (at most 400) |