From Toy to Target: Investigating Representation Transfer for Reinforcement Learning with Implications for Cyber-Physical Systems
Author(s)
Advisor(s)
Editor(s)
Collections
Supplementary to:
Permanent Link
Abstract
As Cyber-Physical Systems (CPS) become more common and more complex, training reinforcement learning (RL) agents to perform well in these large-scale environments remains both challenging and computationally expensive. This paper proposes the Toy Transfer Method (TTM) as a potential approach for leveraging knowledge acquired in small-scale toy environments to expedite agent learning in larger environments. The key idea is that an RL agent trained in a well-structured toy environment may learn useful representations that can be transferred to a more complex target environment, expediting training and improving agent efficacy. The Toy Transfer Method is evaluated using OpenAI's Taxi environment as a case study, transferring knowledge from a 5x5 grid world to multiple 7x7 grid worlds with a roughly twice as large state space. The results demonstrate that the TTM-enhanced Deep Q-Network (DQN) agent consistently outperforms a baseline DQN agent trained from scratch, achieving faster convergence and higher average rewards. Furthermore, the TTM-enhanced agent often leads to convergence in cases where the baseline agent fails to converge to a successful policy. These results suggest that environment abstraction and transfer learning may be viable strategies for improving RL efficiency in CPS, especially when the toy and target environments are structurally similar.
Sponsor
SIEMENS
Date
2025-05-06
Extent
Resource Type
Text
Resource Subtype
Rights Statement
Unless otherwise noted, all materials are protected under U.S. Copyright Law and all rights are reserved