Metadata-Version: 2.4
Name: adaro-rl
Version: 0.3.0
Summary: A Unified Framework of Adversarial Attacks for Robustness of Deep Reinforcement Learning
Author-email: Lucas SCHOTT <lucas.schott@irt-systemx.fr>
License-Expression: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy==1.26.4
Requires-Dist: scipy==1.14.0
Requires-Dist: pandas==2.2.2
Requires-Dist: torch==2.4.0
Requires-Dist: tensorboard==2.17.0
Requires-Dist: tqdm==4.66.4
Requires-Dist: rich==13.7.1
Requires-Dist: opencv-python==4.10.0.84
Requires-Dist: gymnasium==1.0.0
Requires-Dist: stable_baselines3==2.4.1
Requires-Dist: adversarial-robustness-toolbox==1.18.2
Requires-Dist: mujoco==2.3.3
Requires-Dist: gymnasium[mujoco]
Requires-Dist: ale_py==0.10.1
Requires-Dist: gymnasium[atari]
Requires-Dist: huggingface_hub==0.29.3
Dynamic: license-file

<div align="center">
    <h1 style="font-size: large; font-weight: bold;">Adaro-RL</h1>
</div><div align="center">
	<a href="#">
        <img src="https://img.shields.io/badge/Python-3.10%20--%203.11-blue">
  </a>
  <a href="#">
        <img src="https://img.shields.io/badge/Licence-Apache%202.0-blue">
    </a>
	<a href="_static/pylint/pylint.txt">
        <img src="_static/pylint/pylint.svg" alt="Pylint Score">
    </a>
    <a href="_static/flake8/index.html">
        <img src="_static/flake8/flake8.svg" alt="Flake8 Report">
    </a>
	<a href="_static/coverage/index.html">
        <img src="_static/coverage/coverage.svg" alt="Coverage report">
    </a>
</div>

<br>
<div align="center">
    <a href="https://github.com/IRT-SystemX/adaro-rl">
        <img src="https://img.shields.io/badge/GitHub-Repository-181717?logo=github" alt="GitHub">
    </a>
    <a href="https://irt-systemx.github.io/adaro-rl/">
        <img src="https://img.shields.io/badge/Online%20Documentation-available-0A66C2?logo=readthedocs&logoColor=white" alt="Docs">
    </a>
    <a href="https://pypi.org/project/adaro-rl/">
        <img src="https://img.shields.io/pypi/v/adaro-rl?color=blue&label=PyPI&logo=pypi&logoColor=white" alt="PyPI">
    </a>
</div>

<br>



# ADARO-RL

A unified framework of Adversarial Attacks for Robust Reinforcement Learning.

## 📚 Table of Contents

1. [📂 Content of the Repository](#content-of-the-repository)
2. [💾 Installation](#installation)
3. [📦 Content of the Library](#content-of-the-library)
4. [🛡️ Adversarial Attacks](#adversarial-attacks)
   - [ Available Methods](#available-methods)
     - [ Random Attacks](#random-attacks-)
     - [ Fast Gradients Method Based Attacks](#fast-gradients-method-based-attacks-)
     - [ Fast Gradients Sign Method Based Attacks](#fast-gradients-sign-method-based-attacks-)
   - [ Parameters of the Attacks](#parameters-of-the-attacks)
5. [🛠️ Usage](#usage)
   - [ Usage of Attacks Only](#usage-of-attacks-only)
   - [ Usage with the Scripts](#usage-with-the-scripts)
     - [ Command Lines](#command-lines)
     - [ Python](#python)

## 💾 Installation

Use `python 3.10`

```sh
git clone git@github.com:IRT-SystemX/adaro-rl.git
pip install -e adaro-rl
```

## 📦 Content of the Library

```shell
adaro_rl/
├── attacks/
├── pipelines/
├── wrappers/
├── zoo/
└── __init__.py
```

- `src/adaro_rl/attacks/` contains the implementation of many adversarial attacks in a unified framework.
- `src/adaro_rl/pipelines/` contains the implementation of scripts to train, test, attack, and adversarially train agents.
- `src/adaro_rl/wrappers/` contains the implementations of wrappers of gymnasium environments to apply attacks from different perspectives (the agent or the adversary) and with different supports for perturbation (observations or environment).
- `src/adaro_rl/zoo/` contains the implementation of tools and functionality to run the scripts.

## 🛡️ Adversarial Attacks

###  Available Methods

####  Random Attacks

- `RandomUniformAttack` or `RUA`: generates perturbation with random uniform sampling.
- `RandomNormalAttack` or `RNA`: generates perturbation with random normal sampling.
- `RandomSignedAttack` or `RSA`: generates perturbation with random sampling between -1 and 1.

####  Fast Gradients Method Based Attacks

- `FGM_D` for `FastGradientMethod_DiscreteAction`: generates perturbation with fast gradient method to attack discrete action policies.
- `FGM_C` for `FastGradientMethod_ContinuousAction`: generates perturbation with fast gradient method to attack continuous action policies.
- `FGM_V` for `FastGradientMethod_V_Critic`: generates perturbation with fast gradient method to attack value critics.
- `FGM_QC` for `FastGradientMethod_Q_Critic`: generates perturbation with fast gradient method to attack the q critic of a q actor-critic agent.
- `FGM_QAC` for `FastGradientMethod_Q_ActorCritic`: generates perturbation with fast gradient method to attack both models of a q actor-critic agent.

####  Fast Gradients Method Based Attacks

- `FGSM_D` for the sign version of `FGM_D`.
- `FGSM_C` for the sign version of `FGM_C`.
- `FGSM_V` for the sign version of `FGM_V`.
- `FGSM_QC` for the sign version of `FGM_QC`.
- `FGSM_QAC` for the sign version of `FGM_QAC`.


###  Parameters of the Attacks

The common parameters for all attacks are the following:

- `eps` defines the amount of perturbation applied with an attack.
- `norm` defines the norm of the delta. Options: 0, 1, 2, ... inf.
- `max_eps` defines the maximum amount allowed on each dimension of the perturbation. It can be a number, the same for each dimension. It can also be an array, defining a specific amount of perturbation for each dimension. By default, max_eps is set as delta.
- `is_proportional_mask` is an array of booleans that defines for each dimension, if the perturbation has to be applied proportionally to the value of the support on which the perturbation is applied. By default, is_prop_perturb_mask is None.

Other parameters for gradient attacks are the following:

- `target` defines the target followed by the attack. Options: 'untargeted', 'targeted', 'min', 'max' or 'target_fct'. Default is 'untargeted'.

## 🛠️ Usage

###  Usage of Attacks Only

The first way to use the **ADARO-RL** library is to directly apply the attack within your infrastructure with your training and testing scripts.

```python
from adaro_rl.attacks import make_attack
attack = make_attack(attack_name, **attack_kwargs)
perturbation = attack.generate_perturbation(observation)
adv_obs = observation + perturbation
```
- Here, use one of the `attack_name` listed in `adaro_rl.attacks.attack_names`.
- Add the `attack_kwargs` needed for the attack_name specified.
- Use an `observation` of your environment|dataset.

###  Usage with the Scripts

The second way to use the **ADARO-RL** library is to use the application available in `adaro_rl` in command line. This enables to train, test, attack, and adversarial train agents, it is needed to choose a configuration of an existing use case provided in the *adaro_rl.zoo*, add your configurations in the zoo, or provide a new zoo module to use.

####  Command Lines

Here are some basic command line examples :

```bash
adaro_rl train --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo'
adaro_rl test --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo'
adaro_rl online_attack --config 'HalfCheetah-v5' --zoo 'adaro_rl.zoo' --attack-name 'FGM_D' --eps 0.01 --norm "2"
```

The different pipelines that can be executed by the application are the following:

- `train`
- `test`
- `online_attack`
- `adversarial_train`

Use any given script with the option `--help` or check the source codes available in `adaro-rl/pipelines` to see how to use them.


####  Python

You can also use the scripts directly in Python. Here are some examples equivalent to the ones presented above in command lines:

```python
import adaro_rl.zoo as zoo
config = zoo.configs['HalfCheetah-v5']
adaro_rl.train(config=config)
adaro_rl.test(config=config)
adaro_rl.online_attack(config=config, attack_name='FGM_D', eps=0.1, norm=2)
```


####  Examples

Example scripts and jupyter notebooks are available in ```examples``` folder of the repository. You will find there :
- [Attacks examples](./examples/attacks.ipynb) and `examples/attacks.py`: a jupyter notebook and a python script showing the usages of the attacks available.
- [Pipelines examples](./examples/pipelines.ipynb) and `examples/pipelines.sh`: a jupyter notebook and a bash script showing the usages of the high level pipelines available.

## 🛠️ Documentation
- `docs/` contains the documentation the library

To build the HTML documentation:

```bash
cd docs
make html
```

The output will be in:

```
docs/build/html/index.html
```

Open this file in a browser to view the docs locally.
