XiaorongLi (李晓蓉)

Staff research engineer, Baidu
Beijing, China

Research: My research interests include continual learning, online learning and image recognition, etc.

Biography: I got my Ph.D. degree in the school of mathematics and statistics of Xi’an Jiaotong University, supervised by Prof. Zongben Xu and co-supervised by Prof. Jian Sun. I received the Bachelor degree in school of science from Nanjing University of Science and Technology, China, in 2018.

news

Jan 12, 2024	We won the second prize of Guangdong-Hong Kong-Macau Greater Bay Area (Huangpu) International Algorithm Case Competition.
Aug 1, 2023	The code of our VDFD method accepted by TPAMI is released. Welcome to use.

selected publications [full list]

(*) denotes equal contribution

TPAMI

Variational Data-Free Knowledge Distillation for Continual Learning

Xiaorong Li, Shipeng Wang, Jian Sun, and Zongben Xu

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract PDF Supp Code Poster

Deep neural networks suffer from catastrophic forgetting when trained on sequential tasks in continual learning. Various methods rely on storing data of previous tasks to mitigate catastrophic forgetting, which is prohibited in real-world applications considering privacy and security issues. In this paper, we consider a realistic setting of continual learning, where training data of previous tasks are unavailable and memory resources are limited. We contribute a novel knowledge distillation-based method in an information-theoretic framework by maximizing mutual information between outputs of previously learned and current networks. Due to the intractability of computation of mutual information, we instead maximize its variational lower bound, where the covariance of variational distribution is modeled by a graph convolutional network. The inaccessibility of data of previous tasks is tackled by Taylor expansion, yielding a novel regularizer in network training loss for continual learning. The regularizer relies on compressed gradients of network parameters. It avoids storing previous task data and previously learned networks. Additionally, we employ self-supervised learning technique for learning effective features, which improves the performance of continual learning. We conduct extensive experiments including image classification and semantic segmentation, and the results show that our method achieves state-of-the-art performance on continual learning benchmarks.
PR

Memory Efficient Data-Free Distillation for Continual Learning

Xiaorong Li*, Shipeng Wang*, Jian Sun, and Zongben Xu

Pattern Recognition

Abstract PDF Supp Code

Deep neural networks suffer from the catastrophic forgetting phenomenon when trained on sequential tasks in continual learning, especially when data from previous tasks are unavailable. To mitigate catastrophic forgetting, various methods either store data from previous tasks, which may raise privacy concerns, or require large memory storage. Particularly, the distillation-based methods mitigate catastrophic forgetting by using proxy datasets. However, proxy datasets may not match the distributions of the original datasets of previous tasks. To address these problems in a setting where the full training data of previous tasks are unavailable and memory resources are limited, we propose a novel data-free distillation method. Our method encodes knowledge of previous tasks into network parameter gradients by Taylor expansion, deducing a regularizer relying on gradients in network training loss. To improve memory efficiency, we design an approach to compressing the gradients in the regularizer. Moreover, we theoretically analyze the approximation error of our method. Experimental results on multiple datasets demonstrate that our proposed method outperforms the existing approaches in continual learning.