### Navigation

- index
- modules |
- next |
- previous |
- Maja Machine Learning Framework v1.0 documentation »
- Auto-generated API-documentation »

# Agents

Module for MMLF interface for agents

This module contains the AgentBase class that specifies the interface that all MMLF agents have to implement.

class `agents.agent_base.``AgentBase`(config, baseUserDir, *args, **kwargs)

  MMLF interface for agents

  Each agent that should be used in the MMLF needs to be derived from this class and implements the following methods:

  Interface Methods

  > setStateSpace: | : Informs the agent of the environment’s state space
  > setActionSpace: | : Informs the agent of the environment’s action space
  > setState: | : Informs the agent of the environment’s current state
  > giveReward: | : Provides a reward to the agent
  > getAction: | : Request the next action the agent want to execute
  > nextEpisodeStarted:
  > | : Informs the agent that the current episode has terminated and a new one has started.

  `getAction`()

    Request the next action the agent want to execute

  `getGreedyPolicy`()

    Returns the optimal, greedy policy the agent has found so far

  `giveReward`(reward)

    Provides a reward to the agent

  `nextEpisodeStarted`()

    Informs the agent that a new episode has started.

  `setActionSpace`(actionSpace)

    Informs the agent about the action space of the environment

    More information about action spaces can be found in State and Action Spaces

  `setState`(state)

    Informs the agent of the environment’s current state

    More information about (valid) states can be found in State and Action Spaces

  `setStateSpace`(stateSpace)

    Informs the agent about the state space of the environment

    More information about state spaces can be found in State and Action Spaces

  `storePolicy`(filePath, optimal=True)

    Stores the agent’s policy in the given file by pickling it.

    Pickles the agent’s policy and stores it in the file filePath. If the agent is based on value functions, a value function policy wrapper is used to obtain a policy object which is then stored.

    If optimal==True, the agent stores the best policy it has found so far, if optimal==False, the agent stores its current (exploitation) policy.

## Actor-critic

Agent that learns based on the actor-critic architecture.

This module contains an agent that learns based on the actor critic architecture. It uses standard TD(lambda) to learn the value function of the critic and updates the preferences of the actor based on the TD error.

class `agents.actor_critic_agent.``ActorCriticAgent`(*args, **kwargs)

  Agent that learns based on the actor-critic architecture.

  This agent learns based on the actor critic architecture. It uses standard TD(lambda) to learn the value function of the critic. For this reason, it subclasses TDLambdaAgent. The main difference to TD(lambda) is the means for action selection. Instead of deriving an epsilon-greedy policy from its Q-function, it learns an explicit stochastic policy. To this end, it maintains preferences for each action in each state. These preferences are updated after each action execution according to the following rule:

  where delta is the TD error

  Action selection is based on a Gibbs softmax distribution:

  where tau is a temperature parameter.

  Note that the preferences are stored in a function approximator such that in principle, action preferences can be generalized over the state space.

  New in version 0.9.9: Added Actor-Critic agent

  CONFIG DICT

    gamma: | : The discount factor for computing the return given the rewards
    lambda: | : The eligibility trace decay rate
    tau: | : Temperature parameter used in the Gibbs softmax distribution for action selection
    minTraceValue: | : The minimum value of an entry in a trace that is considered to be relevant. If the eligibility falls below this value, it is set to 0 and the entry is thus no longer updated
    update_rule: | : Whether the learning is on-policy or off-policy.. Can be either “SARSA” (on-policy) or “WatkinsQ” (off-policy)
    defaultStateDimDiscretizations:
    | : The default “resolution” the agent uses for every dimension
    defaultActionDimDiscretizations:
    | : Per default, the agent discretizes a continuous action space in this number of discrete actions.
    function_approximator:
    | : The function approximator used for representing the Q value function
    preferences_approximator:
    | The function approximator used for representing the action preferences (i.e. the policy)

## Direct Policy Search (DPS)

Agent that performs direct search in the policy space to find a good policy

This agent uses a black-box optimization algorithm to optimize the parameters of a parametrized policy such that the accumulated (undiscounted) reward of the the policy is maximized.

class `agents.dps_agent.``DPS_Agent`(*args, **kwargs)

  Agent that performs direct search in the policy space to find a good policy

  This agent uses a black-box optimization algorithm to optimize the parameters of a parametrized policy such that the accumulated (undiscounted) reward of the the policy is maximized.

  CONFIG DICT

    policy_search: | : The method used for search of an optimal policy in the policy space. Defines policy parametrization and internally used black box optimization algorithm.

## Dyna TD

The Dyna-TD agent module

This module contains the Dyna-TD agent class. It uses temporal difference learning along with learning a model of the environment and is based on the Dyna architecture.

class `agents.dyna_td_agent.``DynaTDAgent`(*args, **kwargs)

  Agent that learns based on the DYNA architecture.

  Dyna-TD uses temporal difference learning along with learning a model of the environment and doing planning in it.

  CONFIG DICT

    gamma: | : The discount factor for computing the return given the rewards
    epsilon: | : Exploration rate. The probability that an action is chosen non-greedily, i.e. uniformly random among all available actions
    lambda: | : The eligibility trace decay rate
    minTraceValue: | : The minimum value of an entry in a trace that is considered to be relevant. If the eligibility falls below this value, it is set to 0 and the entry is thus no longer updated
    update_rule: | : Whether the learning is on-policy or off-policy.. Can be either “SARSA” (on-policy) or “WatkinsQ” (off-policy)
    defaultStateDimDiscretizations:
    | : The default “resolution” the agent uses for every dimension
    defaultActionDimDiscretizations:
    | : Per default, the agent discretizes a continuous action space in this number of discrete actions
    planner: | : The algorithm used for planning, i.e. for optimizing the policy based on a learned model
    model: | : The algorithm used for learning a model of the environment
    function_approximator:
    | : The function approximator used for representing the Q value function

## Fitted R-Max

Fitted R-Max agent

Fitted R-Max is a model-based RL algorithm that uses the RMax heuristic for exploration control, uses a fitted function approximator (even though this can be configured differently), and uses Dynamic Programming (boosted by prioritized sweeping) for deriving a value function from the model. Fitted R-Max learns usually very sample-efficient (meaning that a good policy is learned with only a few interactions with the environment) but requires a huge amount of computational resources.

class `agents.fitted_r_max_agent.``FittedRMaxAgent`(*args, **kwargs)

  Fitted R-Max agent

  Fitted R-Max is a model-based RL algorithm that uses the RMax heuristic for exploration control, uses a fitted function approximator (even though this can be configured differently), and uses Dynamic Programming (boosted by prioritized sweeping) for deriving a value function from the model. Fitted R-Max learns usually very sample-efficient (meaning that a good policy is learned with only a few interactions with the environment) but requires a huge amount of computational resources.

  See also

  Nicholas K. Jong and Peter Stone, “Model-based function approximation in reinforcement learning”, in “Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems” Honolulu, Hawaii: ACM, 2007, 1-8, http://portal.acm.org/citation.cfm?id=1329125.1329242.

  CONFIG DICT

    gamma: | : The discount factor for computing the return given the rewards#
    min_exploration_value:
    | : The agent explores in a state until the given exploration value (approx. number of exploratory actions in proximity of state action pair) is reached for all actions
    RMax: | : An upper bound on the achievable return an agent can obtain in a single episode
    planner: | : The algorithm used for planning, i.e. for optimizing the policy based on a learned model
    model: | : The algorithm used for learning a model of the environment
    function_approximator:
    | : The function approximator used for representing the Q value function
    defaultActionDimDiscretizations:
    | : Per default, the agent discretizes a continuous action space in this number of discrete actions

## Model-based Direct Policy Search (MBDPS)

The Model-based Direct Policy Search agent

This module contains an agent that uses the state-action-reward-successor_state transitions to learn a model of the environment. It performs than direct policy search (similar to the direct policy search agent using a black-box optimization algorithm to optimize the parameters of a parameterized policy) in the model in order to optimize a criterion defined by a fitness function. This fitness function can be e.g. the estimated accumulated reward obtained by this policy in the model environment. In order to enforce exploration, the model is wrapped for an RMax-like behavior so that it returns the reward RMax for all states that have not been sufficiently explored.

class `agents.mbdps_agent.``MBDPS_Agent`(*args, **kwargs)

  The Model-based Direct Policy Search agent

  An agent that uses the state-action-reward-successor_state transitions to learn a model of the environment. It performs direct policy search (similar to the direct policy search agent using a black-box optimization algorithm to optimize the parameters of a parameterized policy) in the model in order to optimize a criterion defined by a fitness function. This fitness function can be e.g. the estimated accumulated reward obtained by this policy in the model environment. In order to enforce exploration, the model is wrapped for an RMax-like behavior so that it returns the reward RMax for all states that have not been sufficiently explored. RMax should be an upper bound to the actual achievable return in order to enforce optimism in the face of uncertainty.

  CONFIG DICT

    gamma: | : The discount factor for computing the return given the rewards
    planning_episodes:
    | : The number internally simulated episodes that are performed in one planning step
    policy_search: | : The method used for search of an optimal policy in the policy space. Defines policy parametrization and internally used black box optimization algorithm.
    model: | : The algorithm used for learning a model of the environment

## Monte-Carlo

Monte-Carlo learning agent

This module defines an agent which uses Monte Carlo policy evaluation to optimize its behavior in a given environment

class `agents.monte_carlo_agent.``MonteCarloAgent`(*args, **kwargs)

  Agent that learns based on monte-carlo samples of the Q-function

  An agent which uses Monte Carlo policy evaluation to optimize its behavior in a given environment.

  CONFIG DICT

    gamma: | : The discount factor for computing the return given the rewards
    epsilon: | : Exploration rate. The probability that an action is chosen non-greedily, i.e. uniformly random among all available actions
    visit: | : Whether first (“first”) or every visit (“every”) is used in Monte-Carlo updates
    defaultQ: | : The initially assumed Q-value for each state-action pair. Allows to control initial exploration due to optimistic initialization

## Option

class `agents.option_agent.``OptionAgent`(*args, **kwargs)
  CONFIG DICT

    temporalAbstractions:
    | : Whether temporal abstractions are used to augment (“augment”) or to abstract (“abstract”) the action space.
    gamma: | : The discount factor for computing the return given the rewards
    epsilon: | : Exploration rate. The probability that an action is chosen non-greedily, i.e. uniformly random among all available actions
    learningRate: | : The learning rate used internally for updates of the options’ value functions
    experienceReplaySteps:
    | : The number of experience replay steps used for initializing a newly added option
    initDistance: | : The maximal distance a state may have from a subgoal to allow that the ReachSubgoal option is initialized
    goalReachedDistance:
    | : The maximal distance a state may have from a subgoal to allow that the ReachSubgoal option terminates successfully when reaching that state
    skill_discovery:
    | : The algorithm used for skill discovery (i.e. for identifying new options that may be useful)
    defaultStateDimDiscretizations:
    | : The default “resolution” the agent uses for every dimension
    defaultActionDimDiscretizations:
    | : Per default, the agent discretizes a continuous action space in this number of discrete actions

## Random

MMLF agent that acts randomly

This module defines a simple agent that can interact with an environment. It chooses all available actions with the same probability.

This module deals also as an example of how to implement an MMLF agent.

class `agents.random_agent.``RandomAgent`(*args, **kwargs)

  Agent that chooses uniformly randomly among the available actions.

## Temporal Difference Learning

### TD(0)

Agent based on temporal difference learning

This module defines a base agent for all kind of agents based on temporal difference learning. Most of these agents can reuse most methods of this agents and have to modify only small parts.

Note: The TDAgent cannot be instantiated by itself, it is a abstract base class!

class `agents.td_agent.``TDAgent`(*args, **kwargs)

  A base agent for all kind of agents based on temporal difference learning. Most of these agents can reuse most methods of this agents and have to modify only small parts

  Note: The TDAgent cannot be instantiated by itself, it is a abstract base class!

### TD(lambda) - Eligibility Traces

Agent based on temporal difference learning with eligibility traces

This module defines an agent that uses temporal difference learning (e.g. Sarsa) with eligibility traces and function approximation (e.g. linear tile coding CMAC) to optimize its behavior in a given environment

class `agents.td_lambda_agent.``TDLambdaAgent`(*args, **kwargs)

  Agent that implements TD(lambda) RL

  An agent that uses temporal difference learning (e.g. Sarsa) with eligibility traces and function approximation (e.g. linear tile coding CMAC) to optimize its behavior in a given environment

  CONFIG DICT

    update_rule: | : Whether the learning is on-policy or off-policy.. Can be either “SARSA” (on-policy) or “WatkinsQ” (off-policy)
    gamma: | : The discount factor for computing the return given the rewards
    epsilon: | : Exploration rate. The probability that an action is chosen non-greedily, i.e. uniformly random among all available actions
    epsilonDecay: | : Decay factor for the exploration rate. The exploration rate is multiplied with this value after each episode.
    lambda: | : The eligibility trace decay rate
    minTraceValue: | : The minimum value of an entry in a trace that is considered to be relevant. If the eligibility falls below this value, it is set to 0 and the entry is thus no longer updated
    replacingTraces:
    | : Whether replacing or accumulating traces are used.
    defaultStateDimDiscretizations:
    | : The default “resolution” the agent uses for every dimension
    defaultActionDimDiscretizations:
    | : Per default, the agent discretizes a continuous action space in this number of discrete actions
    function_approximator:
    | : The function approximator used for representing the Q value function

### Navigation

- index
- modules |
- next |
- previous |
- Maja Machine Learning Framework v1.0 documentation »
- Auto-generated API-documentation »