 [2407.18913] SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments




























  








Skip to main content





Grab your spot at the free arXiv Accessibility Forum
Forum Schedule

We gratefully acknowledge support fromthe Simons Foundation, Stockholm University, and all contributors. Donate





 > cs > arXiv:2407.18913
  





Help | Advanced Search




All fields
Title
Author
Abstract
Comments
Journal reference
ACM classification
MSC classification
Report number
arXiv identifier
DOI
ORCID
arXiv author ID
Help pages
Full text




Search















open search






GO



open navigation menu


quick links

Login
Help Pages
About












Computer Science > Machine Learning


arXiv:2407.18913 (cs)
    




  [Submitted on 26 Jul 2024]
Title:SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments
Authors:Shu Ishida, João F. Henriques View a PDF of the paper titled SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments, by Shu Ishida and Jo\~ao F. Henriques
View PDF

Abstract:This work compares ways of extending Reinforcement Learning algorithms to Partially Observed Markov Decision Processes (POMDPs) with options. One view of options is as temporally extended action, which can be realized as a memory that allows the agent to retain historical information beyond the policy's context window. While option assignment could be handled using heuristics and hand-crafted objectives, learning temporally consistent options and associated sub-policies without explicit supervision is a challenge. Two algorithms, PPOEM and SOAP, are proposed and studied in depth to address this problem. PPOEM applies the forward-backward algorithm (for Hidden Markov Models) to optimize the expected returns for an option-augmented policy. However, this learning approach is unstable during on-policy rollouts. It is also unsuited for learning causal policies without the knowledge of future trajectories, since option assignments are optimized for offline sequences where the entire episode is available. As an alternative approach, SOAP evaluates the policy gradient for an optimal option assignment. It extends the concept of the generalized advantage estimation (GAE) to propagate option advantages through time, which is an analytical equivalent to performing temporal back-propagation of option policy gradients. This option policy is only conditional on the history of the agent, not future actions. Evaluated against competing baselines, SOAP exhibited the most robust performance, correctly discovering options for POMDP corridor environments, as well as on standard benchmarks including Atari and MuJoCo, outperforming PPOEM, as well as LSTM and Option-Critic baselines. The open-sourced code is available at this https URL.
    



Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Cite as:
arXiv:2407.18913 [cs.LG]


 
(or 
arXiv:2407.18913v1 [cs.LG] for this version)
          





Submission history From: Shu Ishida [view email]       [v1]
        Fri, 26 Jul 2024 17:59:55 UTC (10,882 KB)



 

Full-text links:
Access Paper:


View a PDF of the paper titled SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments, by Shu Ishida and Jo\~ao F. HenriquesView PDFTeX SourceOther Formats


view license


 
    Current browse context: cs.LG


< prev

  |   
next >


new
 | 
recent
 | 2024-07

    Change to browse by:
    
cs
cs.AI




References & Citations

NASA ADSGoogle Scholar
Semantic Scholar




a
export BibTeX citation
Loading...




BibTeX formatted citation
×


loading...


Data provided by: 




Bookmark





 




Bibliographic Tools

Bibliographic and Citation Tools






Bibliographic Explorer Toggle



Bibliographic Explorer (What is the Explorer?)







Litmaps Toggle



Litmaps (What is Litmaps?)







scite.ai Toggle



scite Smart Citations (What are Smart Citations?)








Code, Data, Media

Code, Data and Media Associated with this Article






Links to Code Toggle



CatalyzeX Code Finder for Papers (What is CatalyzeX?)







DagsHub Toggle



DagsHub (What is DagsHub?)







GotitPub Toggle



Gotit.pub (What is GotitPub?)







Links to Code Toggle



Papers with Code (What is Papers with Code?)







ScienceCast Toggle



ScienceCast (What is ScienceCast?)











Demos

Demos






Replicate Toggle



Replicate (What is Replicate?)







Spaces Toggle



Hugging Face Spaces (What is Spaces?)







Spaces Toggle



TXYZ.AI (What is TXYZ.AI?)








Related Papers

Recommenders and Search Tools






Link to Influence Flower



Influence Flower (What are Influence Flowers?)







Connected Papers Toggle



Connected Papers (What is Connected Papers?)







Core recommender toggle



CORE Recommender (What is CORE?)

 




IArxiv recommender toggle



IArxiv Recommender
(What is IArxiv?)






Author
Venue
Institution
Topic














        About arXivLabs
      



arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.










Which authors of this paper are endorsers? |
    Disable MathJax (What is MathJax?)
    












About
Help





contact arXivClick here to contact arXiv
 Contact


subscribe to arXiv mailingsClick here to subscribe
 Subscribe











Copyright
Privacy Policy




Web Accessibility Assistance


arXiv Operational Status 
                    Get status notifications via
                    email
                    or slack





 





