Towards a Theory of Representation Learning for Reinforcement Learning

Agarwal, Alekh

Title:

Towards a Theory of Representation Learning for Reinforcement Learning

dc.contributor.author	Agarwal, Alekh
dc.contributor.corporatename	Georgia Institute of Technology. Machine Learning	en_US
dc.contributor.corporatename	Google	en_US
dc.date.accessioned	2021-09-27T14:48:24Z
dc.date.available	2021-09-27T14:48:24Z
dc.date.issued	2021-09-15
dc.description	Presented online via Bluejeans Events on September 15, 2021 at 12:15 p.m.	en_US
dc.description	Alekh Agarwal is a researcher who works on theoretical foundations of machine learning, spanning many areas including large-scale and distributed optimization, high-dimensional statistics, online learning, and most recently reinforcement learning. He focuses on designing theoretically sound methods which lend themselves to practice, and his work at Microsoft has resulted in the creation of a new Azure service (https://aka.ms/personalizer) that operationalizes some of his reinforcement learning research.	en_US
dc.description	Runtime: 61:09 minutes	en_US
dc.description.abstract	Provably sample-efficient reinforcement learning from rich observational inputs remains a key open challenge in research. While impressive recent advances have allowed the use of linear modelling while carrying out sample-efficient exploration and learning, the handling of more general non-linear models remains limited. In this talk, we study reinforcement learning using linear models, where the features underlying the linear model are learned, rather than apriori specified. While the broader question of representation learning for useful embeddings of complex data has seen tremendous progress, doing so in reinforcement learning presents additional challenges: good representations cannot be discovered without adequate exploration, but effective exploration is challenging in the absence of good representations. Concretely, we study this question in the context of low-rank MDPs [Jiang et al., 2017, Jin et al., 2019, Yang and Wang, 2019], where the features underlying a state-action pair are not assumed to be known, unlike most prior works. We develop two styles of methods, model-based and model-free. For the model-based method, we learn an approximate factorization of the transition model, plan within the model to obtain a fresh exploratory policy and then update our factorization with additional data. In the model-free technique, we learn features so that quantities such as value functions at subsequent states can be predicted linearly in those features. In both approaches, we address the intricate coupling between exploration and representation learning, and provide sample complexity guarantees. More details can be found at https://arxiv.org/abs/2006.10814 and https://arxiv.org/abs/2102.07035. [Based on joint work with Jingling Chen, Nan Jiang, Sham Kakade, Akshay Krishnamurthy, Aditya Modi and Wen Sun]	en_US
dc.format.extent	61:09 minutes
dc.identifier.uri	http://hdl.handle.net/1853/65363
dc.language.iso	en_US	en_US
dc.relation.ispartofseries	Machine Learning @ Georgia Tech (ML@GT) Seminar Series
dc.subject	Reinforcement learning	en_US
dc.subject	Representation learning	en_US
dc.title	Towards a Theory of Representation Learning for Reinforcement Learning	en_US
dc.type	Moving Image
dc.type.genre	Lecture
dspace.entity.type	Publication
local.contributor.corporatename	Machine Learning Center
local.contributor.corporatename	College of Computing
local.relation.ispartofseries	ML@GT Seminar Series
relation.isOrgUnitOfPublication	46450b94-7ae8-4849-a910-5ae38611c691
relation.isOrgUnitOfPublication	c8892b3c-8db6-4b7b-a33a-1b67f7db2021
relation.isSeriesOfPublication	9fb2e77c-08ff-46d7-b903-747cf7406244