Title:
Integrating New Knowledge into a Neural Network without Catastrophic Interference: Computational and Theoretical Investigations in a Hierarchically Structured Environment

dc.contributor.author McClelland, James L.
dc.contributor.corporatename Georgia Institute of Technology. Neural Engineering Center en_US
dc.contributor.corporatename Stanford University. Dept. of Psychology en_US
dc.date.accessioned 2019-04-24T14:44:52Z
dc.date.available 2019-04-24T14:44:52Z
dc.date.issued 2019-04-15
dc.description Presented on April 15, 2019 at 11:15 a.m. in the Krone Engineered Biosystems Building, Room 1005. en_US
dc.description James L. McClelland is the Lucie Stern Professor in the Social Sciences and the Director, Center for Mind, Brain and Computation in the Department of Psychology at Stanford University. His research addresses a broad range of topics in cognitive science and cognitive neuroscience, including perception and perceptual decision making; learning and memory; language and reading; semantic and mathematical cognition; and cognitive development. en_US
dc.description Runtime: 63:03 minutes en_US
dc.description.abstract According to complementary learning systems theory, integrating new memories into a multi-layer neural network without interfering with what is already known depends on interleaving presentation of the new memories with ongoing presentations of items previously learned. This putative dependence is both costly for machine learning and biologically implausible for real brains which are unlikely to have sufficient time for such massive interleaving, even during sleep. We use deep linear neural networks in hierarchically structured environments previously analyzed by Saxe, McClelland, and Ganguli () to gain new insights into how integration of new knowledge might be made more efficient. For this type of environment, its content can be described by the singular value decomposition (SVD) of the environment's input-output covariance matrix, in which each successive dimension corresponds to categorical split in the hierarchical environment. Prior work showed that deep linear networks are sufficient to learn the content of the environment, and they do so in a stage-line way, with each dimension strength rising from near-zero to its maximum strength after a delay inversely proportional to the strength of the dimension, as previously demonstrated by Saxe et al capturing patterns previously observed in deeper non-linear neural networks by Rogers and McClelland (2004). Several observations are then accessible when we consider learning a new item previously not encountered in the micro-environment. (1) The item can be examined in terms of its projection onto the existing structure, and the degree to which it adds a new categorical split. (2) To the extent the item projects onto existing structure, including it in the training corpus leads to the rapid adjustment of the representation of the categories involved, and effectively no adjustment occurs to categories onto which the new item does not project at all. (3) Learning a new split, however, is slow, and its learning dynamics show the same delayed rise to maximum that depends on the dimension's strength. These observations then motivate the development of ideas about how the new information might be acquired efficiently, combining interleaved learning with other strategies. en_US
dc.format.extent 63:03 minutes
dc.identifier.uri http://hdl.handle.net/1853/61001
dc.language.iso en_US en_US
dc.relation.ispartofseries GT Neuro Seminar Series
dc.subject Deep linear neural networks en_US
dc.subject Learning systems en_US
dc.subject Memory en_US
dc.title Integrating New Knowledge into a Neural Network without Catastrophic Interference: Computational and Theoretical Investigations in a Hierarchically Structured Environment en_US
dc.type Moving Image
dc.type.genre Lecture
dspace.entity.type Publication
local.contributor.corporatename Neural Engineering Center
local.relation.ispartofseries GT Neuro Seminar Series
relation.isOrgUnitOfPublication c2e26044-257b-4ef6-8634-100dd836a06c
relation.isSeriesOfPublication 608bde12-7f29-495f-be22-ac0b124e68c5
Files
Original bundle
Now showing 1 - 4 of 4
No Thumbnail Available
Name:
mcclelland.mp4
Size:
286.39 MB
Format:
MP4 Video file
Description:
Download video
No Thumbnail Available
Name:
mcclelland_videostream.html
Size:
1.01 KB
Format:
Hypertext Markup Language
Description:
Streaming video
No Thumbnail Available
Name:
transcript.txt
Size:
51.07 KB
Format:
Plain Text
Description:
Transcription
Thumbnail Image
Name:
thumbnail.jpg
Size:
60.05 KB
Format:
Joint Photographic Experts Group/JPEG File Interchange Format (JFIF)
Description:
Thumbnail
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
3.13 KB
Format:
Item-specific license agreed upon to submission
Description:
Collections