Dataset · Evaluation CLI · 120 tasks (77 travel + 43 e-commerce) · pass@3
Weighted average across all 120 tasks (77 travel + 43 e-commerce)
Travel planning tasks — hotel, transport, attraction
English shopping & consumption tasks — gear, food, electronics, lifestyle
Submission history — newest first
E-Commerce Last Exam evaluates LLM agents on real-world tool-use tasks. Each task runs in an isolated Docker container with domain-specific CLI tools and SQLite databases. Agents must search, analyze, and produce structured recommendations.
Travel config 77 tasks — Hotel booking, transport routing, attraction planning across 30+ cities. Tools: search_hotel_list, search_traffic, search_poi_and_ticket.
E-Commerce config 43 tasks — Shopping decisions, price comparison, product research across travel gear, food, electronics, lifestyle. Tools: hotel_search, taobao_search, general_search.
Scoring: Each task yields a reward in [0, 1] combining hard checks (rubrics.py) and LLM soft judge (judge.py).
pip install flyai-bench # Run evaluation flyai-bench run --dataset-config travel # Package results flyai-bench submit --model your-model --provider your-provider
Then open a PR to ecommerce_last_exam with your results in experiments/evaluation/<config>/<slug>/.
CI validates format automatically. Once merged, the leaderboard updates.
| File | Description |
|---|---|
metadata.yaml | Model/agent info + evaluation stats |
scores.jsonl | Per-instance: {instance_id, domain, reward, duration_sec} |
summary.json | Aggregate statistics (auto-generated if missing) |