Metadata-Version: 2.4
Name: selenium-execution-engine
Version: 1.0.0
Summary: A flexible, JSON-driven execution engine for Selenium with BeautifulSoup-powered extraction.
Author-email: Ricardo Martinez <rimtzg@gmail.com>
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: selenium>=4.10.0
Requires-Dist: jsonschema>=4.18.0
Requires-Dist: beautifulsoup4>=4.12.0
Dynamic: license-file

# Selenium Execution Engine

A flexible, JSON-driven execution engine for Selenium. This engine allows you to define navigation steps, interactions, and data extraction rules using a simple JSON configuration. It can be used as a standalone Command Line Interface (CLI) tool or imported as a Python library.

## Installation

1. Clone this repository.
2. Install the package in editable mode (which also installs all dependencies):
   ```bash
   pip install -e .
   ```
3. Make sure you have the appropriate WebDriver installed (e.g., ChromeDriver for Google Chrome) and available in your system's PATH.

## Usage as CLI

Once installed, the CLI tool `selenium-engine` will be available in your PATH. You can run it directly:

```bash
selenium-engine config.json
```

To save the extracted results to a file, use the `-o` or `--output` flag:

```bash
selenium-engine config.json -o results.json
```

To override the number of iterations when using loop mode, use the `-i` or `--iterations` flag:

```bash
selenium-engine config.json -i 5
```

## Usage as a Library

You can integrate the engine into your Python scripts.

```python
import json
from selenium_engine import execute_scraping

config_json = """
{
  "target_url": "https://example.com",
  "actions": [
    {"type": "wait", "selector": "h1", "timeout": 5}
  ],
  "extraction_schema": {
    "items": {
      "fields": {
        "title": {"selector": "h1", "type": "text"}
      }
    }
  }
}
"""

# You can pass the JSON string directly
results = execute_scraping(config_json)
print(json.dumps(results, indent=2))

# Or you can pass a parsed dictionary
config_dict = json.loads(config_json)
results_dict = execute_scraping(config_dict)
print(json.dumps(results_dict, indent=2))
```

## JSON Configuration Schema

The engine uses a JSON schema to validate your configuration. Here is an overview of the options:

### Basic Structure

```json
{
  "target_url": "https://surtido.app/productos",
  "selenium_config": {
    "driver": "chrome",
    "headless": true
  },
  "actions": [
    ...
  ],
  "extraction_schema": {
    ...
  }
}
```

### Selenium Configuration (`selenium_config`)
- `driver`: Browser to use (`chrome`, `firefox`, `edge`, `safari`). Defaults to `chrome`.
- `headless`: Run in headless mode (`true`/`false`). Defaults to `true`.
- `arguments`: Array of string arguments to pass to the driver.
- `page_load_timeout`: Timeout in seconds.

### Execution Mode (`execution_mode`)
Allows running the engine in a loop instead of a single linear execution.
- `type`: Either `linear` or `loop`.
- `iterations`: Number of times to loop (requires `target_url` at the root).
- `loop_urls`: Array of URLs to loop through.

*Note: If you use `execution_mode`, you can omit `target_url` at the root if you provide `loop_urls`.*

### Actions (`actions`)
An array of steps to execute sequentially on the page.

#### Wait Action
Waits for an element to be present in the DOM.
```json
{"type": "wait", "selector": ".product-list", "timeout": 10}
```

#### Click Action
Clicks on an element.
```json
{"type": "click", "selector": "#submit-btn"}
```

#### Scroll Action
Scrolls the page.
```json
{"type": "scroll", "target": "bottom"}
// OR
{"type": "scroll", "amount": 500}
```

#### Input Action
Types text into an input field.
```json
{"type": "input", "selector": "#search-box", "value": "laptops"}
```

### Extraction Schema (`extraction_schema`)
Defines how to extract data from the page after actions are completed.

#### Extracting a Single Item
```json
"extraction_schema": {
  "items": {
    "fields": {
      "title": {"selector": "h1.title", "type": "text"}
    }
  }
}
```

#### Extracting a List of Items
Specify a `container` selector to loop through multiple elements.
```json
"extraction_schema": {
  "items": {
    "container": ".product-card",
    "fields": {
      "title": {"selector": "h2.title", "type": "text"},
      "price": {"selector": ".price-tag", "type": "text", "transform": "float"},
      "stock": {"selector": ".stock-status", "type": "attribute", "attr": "data-qty"}
    }
  }
}
```

**Field Configurations:**
- `selector`: CSS selector. Set to `"self"` (or leave empty `""`) to select the container element itself.
- `type`: `text` (inner text) or `attribute` (element attribute).
- `attr`: Required if type is `attribute`. Name of the attribute.
- `transform`: Optional. Cast the value to `float`, `int`, `string`, or `bool`.

### BeautifulSoup Extraction

Under the hood, once actions are completed, the engine retrieves the page source and uses **BeautifulSoup** to parse and extract the configured fields. This yields high-performance queries and allows flexible scraping features like referencing the container element itself.

## Modularity

To add new actions, simply create a new `.py` file in the `selenium_engine/actions/` directory. It should inherit from `BaseAction` and implement the `execute(self, driver, action_config)` method. Then, register it in `selenium_engine/actions/__init__.py` and update the `schema.json` to allow the new action type.
