Metadata-Version: 2.4 Name: actbench Version: 0.0.1a4 Summary: A framework for evaluating web automation agents and LAM systems. Project-URL: Homepage, https://github.com/raccoonaihq/actbench Project-URL: Repository, https://github.com/raccoonaihq/actbench Author-email: Raccoon AI License-Expression: MIT License-File: LICENSE Keywords: AI,LAM systems,agent evaluation,benchmarking,web automation Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: MacOS Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: OS Independent Classifier: Operating System :: POSIX Classifier: Operating System :: POSIX :: Linux Classifier: Programming Language :: Python :: 3.12 Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Typing :: Typed Requires-Python: >=3.12 Requires-Dist: click~=8.1.8 Requires-Dist: langchain-openai~=0.3.7 Requires-Dist: langchain~=0.3.19 Requires-Dist: pydantic~=2.10.6 Requires-Dist: pyfiglet~=1.0.2 Requires-Dist: raccoonai~=0.1.0a6 Requires-Dist: rich~=13.9.4 Description-Content-Type: text/markdown ``` __ __ __ ____ ______/ /_/ /_ ___ ____ _____/ /_ / __ `/ ___/ __/ __ \/ _ \/ __ \/ ___/ __ \ / /_/ / /__/ /_/ /_/ / __/ / / / /__/ / / / \__,_/\___/\__/_.___/\___/_/ /_/\___/_/ /_/ ``` [![PyPI version](https://img.shields.io/pypi/v/actbench.svg?logo=pypi&&logoColor=white&&color=5d5fef&&cacheSeconds=10)](https://pypi.org/project/actbench/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Requires: Python 3.12+](https://img.shields.io/badge/Python-3.12+-blue.svg)](https://www.python.org/downloads/release/python-3120/) ## Overview **actbench** is a extensible framework designed to evaluate the performance and capabilities of web automation agents and LAM systems. ## Installing actbench CLI **actbench** requires Python 3.12 or higher. We recommend using `pipx` for a clean, isolated installation: ```bash pipx install actbench ``` ## Usage ### 1. Setting API Keys Before running benchmarks, you need to set API keys for the agents you want to use. ```bash actbench set-key --agent raccoonai ``` You can list the supported agents and check which API keys are stored: ```bash actbench agents list ``` ### 2. Listing Available Tasks **actbench** provides a [built-in dataset](https://github.com/raccoonaihq/actbench/blob/master/dataset.jsonl) of web automation tasks, crafted by merging and refining tasks from the [webarena](https://github.com/web-arena-x/webarena/blob/main/config_files/test.raw.json) and [webvoyager](https://github.com/MinorJerry/WebVoyager/blob/main/data/WebVoyager_data.jsonl) datasets.
Duplicate tasks have been stripped out, and the queries have been refreshed to align with the most recent information.
If you want to explore how the tasks have been modified, you can trace their IDs back to the original datasets for a side-by-side comparison.
To see all the tasks currently available, just run this command: ```bash actbench tasks list ``` ### 3. Running Benchmarks The `run` command is the heart of **actbench**. It allows you to execute tasks against specified agents. #### Basic Usage ```bash actbench run --agent raccoonai --task 256 --task 424 ``` This command runs tasks with IDs `256` and `424` using the `raccoonai` agent. #### Running All Tasks ```bash actbench run --agent raccoonai --all-tasks ``` This runs all available tasks using the `raccoonai` agent. #### Running Random Tasks ```bash actbench run --agent raccoonai --random 5 ``` This runs a random sample of 5 tasks using the `raccoonai` agent. #### Running with All Agents ```bash actbench run --all-agents --all-tasks ``` This runs all tasks with all configured agents (for which API keys are stored). #### Controlling Parallelism ```bash actbench run --agent raccoonai --all-tasks --parallel 4 ``` This runs all tasks using the `raccoonai` agent, executing up to 4 tasks concurrently. #### Setting Rate Limiting ```bash actbench run --agent raccoonai --all-tasks --rate-limit 0.5 ``` This adds a 0.5-second delay between task submissions. #### Disabling Scoring ```bash actbench run --agent raccoonai --all-tasks --no-scoring ``` This disables the LLM powered scoring, and gives all tasks a score of -1. #### Combined Options You can combine these options for more complex benchmark configurations: ```bash actbench run --agent raccoonai --agent anotheragent --task 1 --task 2 --random 3 --parallel 2 --rate-limit 0.2 ``` This command runs tasks 1 and 2, plus 3 random tasks, using both `raccoonai` and `anotheragent` (assuming API keys are set), with a parallelism of 2 and a rate limit of 0.2 seconds. ### 4. Viewing Results The `results` command group allows you to manage and view benchmark results. #### Listing Results ```bash actbench results list ``` You can filter results by agent or run ID: ```bash actbench results list --agent raccoonai actbench results list --run-id ``` #### Exporting Results You can export results to JSON or CSV files: ```bash actbench results export --format json --output results.json actbench results export --format csv --output results.csv --agent raccoonai ``` #### Here's a complete table detailing the `actbench` CLI commands, their flags (options) and explanations: | Command | Flag(s) / Option(s) | Explanation | |:-------------------------------|:-----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------| | `actbench run` | `--task` / `-t` | Specifies one or more task IDs to run. Can be used multiple times. If omitted, other task selection flags (`--random`, `--all-tasks`) must be used. | | | `--agent` / `-a` | Specifies one or more agents to use. Can be used multiple times. If omitted, `--all-agents` must be used. | | | `--random` / `-r` | Runs a specified number of random tasks. Takes an integer argument (e.g., `--random 5`). | | | `--all-tasks` | Runs all available tasks. | | | `--all-agents` | Runs with all configured agents (for which API keys have been set). | | | `--parallel` / `-p` | Sets the number of tasks to run concurrently. Takes an integer argument (e.g., `--parallel 4`). Defaults to 1 (no parallelism). | | | `--rate-limit` / `-l` | Sets the delay (in seconds) between task submissions. Takes a float argument (e.g., `--rate-limit 0.5`). Defaults to 0.1. | | | `--no-scoring` / `-ns` | Disables LLM-based scoring. Results will have a score of -1. | | `actbench tasks list` | *None* | Lists all available tasks in the dataset, showing their ID, query, URL, complexity, and whether they require login. | | `actbench set-key` | `--agent` / `-a` | Sets the API key for a specified agent. Prompts the user to enter the key securely. Example: `actbench set-key --agent raccoonai` | | `actbench agents list` | *None* | Lists all supported agents, and shows which agents have API Keys stored. | | `actbench results list` | `--agent` / `-a` | Filters the results to show only those for a specific agent. | | | `--run-id` / `-r` | Filters the results to show only those for a specific run ID. | | `actbench results export` | `--agent` / `-a` | Filters the results to be exported to a specific agent. | | | `--run-id` / `-r` | Filters the results to be exported for a specific run ID. | | | `--format` / `-f` | Specifies the export format. Must be one of `json` or `csv`. Defaults to `json`. | | | `--output` / `-o` | Specifies the output file path. Required. | | `actbench` | *None* | Prints the help message for the CLI. | | `actbench --version` | *None* | Prints the actbench version number. | ## Extending actbench ### Adding New Agents 1. **Create a new client class:** Create a new Python file in the `actbench/clients/` directory (e.g., `my_agent.py`). 2. **Implement the `BaseClient` interface:** Your class should inherit from `actbench.clients.BaseClient` and implement the `set_api_key()` and `run()` methods. 3. **Register your client:** Add your client class to the `_CLIENT_REGISTRY` in `actbench/clients/__init__.py`. ### Adding New Datasets 1. **Create a new dataset class:** Create a new Python file in the `actbench/datasets/` directory (e.g., `my_dataset.py`). 2. **Implement the `BaseDataset` interface:** Your class should inherit from `actbench.datasets.BaseDataset` and implement the `load_task_data()`, `get_all_task_ids()`, and `get_all_tasks()` methods. 3. **Provide your dataset file:** Place your dataset file (e.g., `my_dataset.jsonl`) in the `src/actbench/dataset/` directory. 4. **Update `_DATASET_INSTANCE`**: If you want to use this dataset by default, update the `_DATASET_INSTANCE` variable in `src/actbench/datasets/__init__.py`. ### Adding New Evaluation Metrics You can customize the evaluation process by modifying the `Evaluator` class in `actbench/executor/evaluator.py` or by creating a new evaluator and integrating it into the `TaskExecutor`. ## Contributing Contributions are welcome! Please follow these simple guidelines: 1. Fork the repository. 2. Create a new branch for your feature or bug fix. 3. Write clear and concise code with appropriate comments. 4. Submit a pull request.