Volume 10, Number 2

Extractive Summarization with Very Deep Pretrained Language Model

  Authors

Yang Gu¹ and Yanke Hu², ¹Suning USA, USA and ²Humana, USA

  Abstract

Recent development of generative pretrained language models has been proven very successful on a wide range of NLP tasks, such as text classification, question answering, textual entailment and so on.In this work, we present a two-phase encoder decoder architecture based on Bidirectional Encoding Representation from Transformers(BERT) for extractive summarization task. We evaluated our model by both automatic metrics and human annotators, and demonstrated that the architecture achieves the state-of-the-art comparable result on large scale corpus - CNN/Daily Mail¹. As the best of our knowledge, this is the first work that applies BERT based architecture to a text summarization task and achieved the state-of-the-art comparable result.

  Keywords

BERT, AI, Deep Learning, Summarization