Continual Learning and Catastrophic Forgetting

Share this & earn $10
Paul Hand
Published at : 09 Nov 2021
4651 views
184
1

A lecture that discusses continual learning and catastrophic forgetting in deep neural networks. We discuss the context, methods for evaluating algorithms, and algorithms based on regularization, dynamic architectures, and Complementary Learning Systems. Specifically, we discuss data permutation tasks, incremental task learning, multimodal learning, the Learning without Forgetting algorithm, Elastic Weight Consolidation, Progressive Neural Networks, and Generative replay.

This lecture is from Northeastern University’s CS 7150 Summer 2020 class on Deep Learning, taught by Paul Hand.

The notes are available at: http://khoury.northeastern.edu/home/hand/teaching/cs7150-summer-2020/Continual_Learning_and_Catastrophic_Forgetting.pdf

References:

Parisi et al. 2019:

Parisi, German I., Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. "Continual lifelong learning with neural networks: A review." Neural Networks (2019).

Chen and Liu 2018:

Chen, Zhiyuan, and Bing Liu. "Lifelong machine learning." Synthesis Lectures on Artificial Intelligence and Machine Learning 12, no. 3 (2018): 1-207. Chapter 4.

Kemker et al. 2017:

Kemker, Ronald, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. "Measuring catastrophic forgetting in neural networks." In Thirty-second AAAI conference on artificial intelligence. 2018.

van de Ven and Tolias 2019:

van de Ven, Gido M., and Andreas S. Tolias. "Three scenarios for continual learning." arXiv preprint arXiv:1904.07734 (2019).

Li and Hoiem 2018:

Li, Zhizhong, and Derek Hoiem. "Learning without forgetting." IEEE transactions on pattern analysis and machine intelligence 40, no. 12 (2017): 2935-2947.

Kirkpatrick et al. 2017:

Kirkpatrick, James, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114, no. 13 (2017): 3521-3526.

Rusu et al. 2016:

Rusu, Andrei A., Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. "Progressive neural networks." arXiv preprint arXiv:1606.04671 (2016).

Shin et al. 2017:

Shin, Hanul, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. "Continual learning with deep generative replay." In Advances in Neural Information Processing Systems, pp. 2990-2999. 2017.

ContinualLearningCatastrophic

Waking Up In 2044, They’re The Only 2 Men Left On Earth As Males Have Long Been Extinct

Waking Up In 2044, They’re The Only 2 Men Left On Earth As Males Have Long Been Extinct

H.E.R. - Process (Audio)

H.E.R. - Process (Audio)

Mr. Vegas - Identify My Love (Official Video)

Mr. Vegas - Identify My Love (Official Video)

Biden: "I'm a nice guy.. I'm a decent man.. Biden is a good guy."

Biden: "I'm a nice guy.. I'm a decent man.. Biden is a good guy."

Types Of Pizza Lovers | @Akshay Nayar  | @Kanishk Priyadarshi  | @Aakansha Pushp| Satyam | Zomato

Types Of Pizza Lovers | @Akshay Nayar | @Kanishk Priyadarshi | @Aakansha Pushp| Satyam | Zomato

Minato Kunai Modeling Tutorial #Kai_Free_Models ✔

Minato Kunai Modeling Tutorial #Kai_Free_Models ✔

Impartial (The Battle) - The Clockwork Dolls

Impartial (The Battle) - The Clockwork Dolls

Young Girl Trapped Inside A Room And Forced To Watch Her Boyfriend Cheating On Her

Young Girl Trapped Inside A Room And Forced To Watch Her Boyfriend Cheating On Her

Farécla - Stone Chip Removal Kit

Farécla - Stone Chip Removal Kit

OtherView - What You Want | Official Video Clip

OtherView - What You Want | Official Video Clip

SUPERHEROES SNEAK FOOD INTO CLASS | I Became a Superheroes And Help My Friends! by 123GO! SCHOOL

SUPERHEROES SNEAK FOOD INTO CLASS | I Became a Superheroes And Help My Friends! by 123GO! SCHOOL

Epic Games Is Building The Oasis! First Oculus Quest 120hz Game, Pico Neo 3, VR Skating & More!

Epic Games Is Building The Oasis! First Oculus Quest 120hz Game, Pico Neo 3, VR Skating & More!

Vice Ganda gives additional cash prize to the Songbayanan | Everybody GuesSing | Everybody Sing

Vice Ganda gives additional cash prize to the Songbayanan | Everybody GuesSing | Everybody Sing

1. Introduction, Financial Terms and Concepts

1. Introduction, Financial Terms and Concepts

Free Time Activities in English

Free Time Activities in English

Apocalyptica - 'Path' (Official Video)

Apocalyptica - 'Path' (Official Video)

Hang Massive - Goa Garden Live Concert

Hang Massive - Goa Garden Live Concert

Kids vocabulary - Body - parts of the body - Learn English for kids - English educational video

Kids vocabulary - Body - parts of the body - Learn English for kids - English educational video

Introduction to Glue

Introduction to Glue

Differences Between Products and Services

Differences Between Products and Services

Besides the Fact

Besides the Fact

Traveling to the Happiest Country in the World!!

Traveling to the Happiest Country in the World!!

PRETTYMUCH - Lying (Official Video) ft. Lil Tjay

PRETTYMUCH - Lying (Official Video) ft. Lil Tjay

Figuring out - Koryin - 1 hour

Figuring out - Koryin - 1 hour

Reamonn - Sometimes

Reamonn - Sometimes

Colin Powell Dies Of Complications From COVID-19

Colin Powell Dies Of Complications From COVID-19

Paolo Nutini - Bear Me In Mind - Zermatt 2017

Paolo Nutini - Bear Me In Mind - Zermatt 2017

Warzone Moments that will probably offend you

Warzone Moments that will probably offend you

Price Wars! What to Do If Your Competitors Reduce the Price

Price Wars! What to Do If Your Competitors Reduce the Price

The Art of Effortless Living (Taoist Documentary)

The Art of Effortless Living (Taoist Documentary)

YOU WILL NOT BE DEFEATED - Motivational Speech 2021

YOU WILL NOT BE DEFEATED - Motivational Speech 2021

Lil Mosey - Noticed [Clean]

Lil Mosey - Noticed [Clean]

Πώς να αναθερμάνω την ερωτική μου σχέση | Familiar 18/11/2021 | OPEN TV

Πώς να αναθερμάνω την ερωτική μου σχέση | Familiar 18/11/2021 | OPEN TV

Some Feeling

Some Feeling

B-Dynamitze - Overall Feat. LetoDie (CLIPE OFICIAL)

B-Dynamitze - Overall Feat. LetoDie (CLIPE OFICIAL)

7 Signs You Lack Confidence

7 Signs You Lack Confidence

Konola - Battling (Official Audio)

Konola - Battling (Official Audio)

210312 Store køer for gratis munnbind

210312 Store køer for gratis munnbind

Talk to Troubled Lianne | Genshin Impact

Talk to Troubled Lianne | Genshin Impact

Exploring Oslo, Norway/Sorenga/Bjorvika/Bonding Time with Friends

Exploring Oslo, Norway/Sorenga/Bjorvika/Bonding Time with Friends

Halsey - Castle

Halsey - Castle

I Customized 100 iPhones, Then Giving Them To People In Need!

I Customized 100 iPhones, Then Giving Them To People In Need!

A NEW MEMBER FOR THE PLUMBING TEAM!

A NEW MEMBER FOR THE PLUMBING TEAM!

Capture More Leads With A Chatbot (and Live Chat) Solution - Continually Walkthrough and Review

Capture More Leads With A Chatbot (and Live Chat) Solution - Continually Walkthrough and Review

Noam Chomsky: On Power and Ideology | The New School

Noam Chomsky: On Power and Ideology | The New School

Tony Junior Live @ Revealed Night ADE 2021

Tony Junior Live @ Revealed Night ADE 2021

Top 10 Places To Visit In 2021 (If We Can Travel)

Top 10 Places To Visit In 2021 (If We Can Travel)

Fit - Ne Mogu Bez Nje (Official Music Video)

Fit - Ne Mogu Bez Nje (Official Music Video)

Hozier - Would That I (Audio)

Hozier - Would That I (Audio)