Organizational Unit:

Mobile Robot Laboratory

Permanent Link

https://hdl.handle.net/1853/70779

Parent Organization

Organizational Unit

College of Computing

ArchiveSpace Name Record

https://finding-aids.library.gatech.edu/agents/corporate_entities/1110

Full item page

Publication Search Results

Now showing 1 - 10 of 10

A New Heuristic Approach for Dual Control

(Georgia Institute of Technology, 1997) Ram, Ashwin ; Santamaria, Juan Carlos

Autonomous agents engaged in a continuous interaction with an incompletely known environment face the problem of dual control [Fel’dbaum 1965]. Simply stated, actions are necessary not only for studying the environment, but also for making progress on the task. In other words, actions must bear a “dual” character: They must be investigators to some degree, but also directors to some degree. Because the number of variables involved in the solution of the dual control problem increases with the number of decision stages, the exact solution of the dual control problem is computationally intractable except for a few special cases. This paper provides an overview of dual control theory and proposes a heuristic approach towards obtaining a near-optimal dual control method that can be implemented. The proposed algorithm selects control actions taking into account the information contained in past observations as well as the possible information that future observations may reveal. In short, the algorithm anticipates the fact that future learning is possible and selects the control actions accordingly. The algorithm uses memory-based methods to associate long-term benefit estimates to belief states and actions, and selects the actions to execute next according to such estimates. The algorithm uses the outcome of every experience to progressively refine the long-term benefit estimates so that it can make better, improved decisions as it progresses. The algorithm is tested on a classical simulation problem.
Experiments With Reinforcement Learning in Problems With Continuous State and Action Spaces

(Georgia Institute of Technology, 1996) Ram, Ashwin ; Santamaria, Juan Carlos ; Sutton, Richard S.

A key element in the solution of reinforcement learning problems is the value function. The purpose of this function is to measure the long-term utility or value of any given state and it is important because an agent can use it to decide what to do next. A common problem in reinforcement learning when applied to systems having continuous states and action spaces is that the value function must operate with a domain consisting of real-valued variables, which means that it should be able to represent the value of infinitely many state and action pairs. For this reason, function approximators are used to represent the value function when a close-form solution of the optimal policy is not available. In this paper, we extend a previously proposed reinforcement learning algorithm so that it can be used with function approximators that generalize the value of individual experiences across both, state and action spaces. In particular, we discuss the benefits of using sparse coarse-coded function approximators to represent value functions and describe in detail three implementations: CMAC, instance-based, and case-based. Additionally, we discuss how function approximators having different degrees of resolution in different regions of the state and action spaces may influence the performance and learning efficiency of the agent. We propose a simple and modular technique that can be used to implement function approximators with non-uniform degrees of resolution so that it can represent the value function with higher accuracy in important regions of the state and action spaces. We performed extensive experiments in the double integrator and pendulum swing up systems to demonstrate the proposed ideas.
Using Genetic Algorithms to Learn Reactive Control Parameters for Autonomous Robotic Navigation

(Georgia Institute of Technology, 1994) Ram, Ashwin ; Arkin, Ronald C. ; Boone, Gary Noel ; Pearce, Michael

This paper explores the application of genetic algorithms to the learning of local robot navigation behaviors for reactive control systems. Our approach evolves reactive control systems in various environments, thus creating sets of "ecological niches" that can be used in similar environments. The use of genetic algorithms as an unsupervised learning method for a reactive control architecture greatly reduces the effort required to configure a navigation system. Unlike standard genetic algorithms, our method uses a floating point gene representation. The system is fully implemented and has been evaluated through extensive computer simulations of robot navigation through various types of environments.
A Multistrategy Case-Based and Reinforcement Learning Approach to Self-Improving Reactive Control Systems for Autonomous Robotic Navigation

(Georgia Institute of Technology, 1993) Ram, Ashwin ; Santamaria, Juan Carlos

This paper presents a self-improving reactive control system for autonomous robotic navigation. The navigation module uses a schema-based reactive control system to perform the navigation task. The learning module combines case-based reasoning and reinforcement learning to continuously tune the navigation system through experience. The case-based reasoning component perceives and characterizes the system’s environment, retrieves an appropriate case, and uses the recommendations of the case to tune the parameters of the reactive control system. The reinforcement learning component refines the content of the cases based on the current experience. Together, the learning components perform on-line adaptation, resulting in improved performance as the reactive control system tunes itself to the environment, as well as on-line learning, resulting in an improved library of cases that capture environmental regularities necessary to perform on-line adaptation. The system is extensively evaluated through simulation studies using several performance metrics and system configurations.
Knowledge Compilation and Speedup Learning in Continuous Task Domains

(Georgia Institute of Technology, 1993) Ram, Ashwin ; Santamaria, Juan Carlos

Many techniques for speedup learning and knowledge compilation focus on the learning and optimization of macro-operators or control rules in task domains that can be characterized using a problem-space search paradigm. However, such a characterization does not fit well the class of task domains in which the problem solver is required to perform in a continuous manner. For example, in many robotic domains, the problem solver is required to monitor real-valued perceptual inputs and vary its motor control parameters in a continuous, on-line manner to successfully accomplish its task. In such domains, discrete symbolic states and operators are difficult to define. To improve its performance in continuous problem domains, a problem solver must learn, modify, and use “continuous operators” that continuously map input sensory information to appropriate control outputs. Additionally, the problem solver must learn the contexts in which those continuous operators are applicable. We propose a learning method that can compile sensorimotor experiences into continuous operators, which can then be used to improve performance of the problem solver. The method speeds up the task performance as well as results in improvements in the quality of the resulting solutions. The method is implemented in a robotic navigation system, which is evaluated through extensive experimentation.
Multistrategy Learning in Reactive Control Systems for Autonomous Robotic Navigation

(Georgia Institute of Technology, 1993) Ram, Ashwin ; Santamaria, Juan Carlos

This paper presents a self-improving reactive control system for autonomous robotic navigation. The navigation module uses a schema-based reactive control system to perform the navigation task. The learning module combines case-based reasoning and reinforcement learning to continuously tune the navigation system through experience. The case-based reasoning component perceives and characterizes the system's environment, retrieves an appropriate case, and uses the recommendations of the case to tune the parameters of the reactive control system. The reinforcement learning component refines the content of the cases based on the current experience. Together, the learning components perform on-line adaptation, resulting in improved performance as the reactive control system tunes itself to the environment, as well as on-line case learning, resulting in an improved library of cases that capture environmental regularities necessary to perform on-line adaptation. The system is extensively evaluated through simulation studies using several performance metrics and system configurations.
A Case-Based Approach to Reactive Control for Autonomous Robots

(Georgia Institute of Technology, 1992) Moorman, Kenneth ; Ram, Ashwin

We propose a case-based method of selecting behavior sets as an addition to traditional reactive robotic control systems. The new system (ACBARR - A Case BAsed Reactive Robotic system) provides more flexible performance in novel environments, as well as overcoming a standard "hard" problem for reactive systems, the box canyon. Additionally, ACBARR is designed in a manner which is intended to remain as close to pure reactive control as possible. Higher level reasoning and memory functions are intentionally kept to a minimum. As a result, the new reasoning does not significantly slow the system down from pure reactive speeds.
Learning Momentum: On-Line Performance Enhancement for Reactive Systems

(Georgia Institute of Technology, 1992) Arkin, Ronald C. ; Clark, Russell J. ; Ram, Ashwin

We describe a reactive robotic control system which incorporates aspects of machine learning to improve the system's ability to successfully navigate in unfamiliar environments. This system overcomes limitations of completely reactive systems by exercising on-line performance enhancement without the need for high level planning. The results of extensive simulation studies using the learning enhanced reactive controller are presented.
The Learning of Reactive Control Parameters Through Genetic Algorithms

(Georgia Institute of Technology, 1992) Arkin, Ronald C. ; Pearce, Michael ; Ram, Ashwin

This paper explores the application of genetic algorithms to the learning of local robot navigation behaviors for reactive control systems. Our approach is to train a reactive control system in various types of environments, thus creating a set of "ecological niches" that can be used in similar environments. The use of genetic algorithms as an unsupervised learning method for a reactive control architecture greatly reduces the effort required to configure a navigation system. Findings from computer simulations of robot navigation through various types of environments are presented.
Case-Based Reactive Navigation: A Cased Based Method for On-Line Selection and Adaptation of Reactive Control Parameters in Autonomous Robotics Systems

(Georgia Institute of Technology, 1992) Ram, Ashwin ; Arkin, Ronald C. ; Clark, Russell J. ; Moorman, Kenneth

This article presents a new line of research investigating on-line learning mechanisms for autonomous intelligent agents. We discuss a case-based method for dynamic selection and modification of behavior assemblages for a navigational system. The case-based reasoning module is designed as an addition to a traditional reactive control system, and provides more flexible performance in novel environments without extensive high-level reasoning that would otherwise slow the system down. The method is implemented in the ACBARR (A Case-BAsed Reactive Robotic) system, and evaluated through empirical simulation of the system on several different environments, including "box canyon" environments known to be problematic for reactive control systems in general.