Continual Training of Language Models for Few-Shot Learning

Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. Adapting or posttraining an LM using an unlabeled ___domain corpus can produce even better performance for end-tasks in the ___domain. This paper proposes the problem of continually extending an LM by incrementally post-train the LM with a sequence of unlabeled ___domain corpora to expand its knowledge without forgetting its previous skills. The goal is to improve the few-shot end-task learning in these domains. The resulting system is called CPT (Continual PostTraining), which to our knowledge, is the first continual post-training system. Experimental results verify its effectiveness.