Volume 15, Number 1

An Improved mT5 Model for Chinese Text Summary Generation

  Authors

Fuping Ren², Jian Chen¹, and Defu Zhang¹, ¹Xiamen University, China, ²Shenzhen Comtech Technology Co. Ltd, China

  Abstract

Complicated policy texts require a lot of effort to read, so there is a need for intelligent interpretation of Chinese policies. To better solve the Chinese Text Summarization task, this paper utilized the mT5 model as the core framework and initial weights. Additionally, In addition, this paper reduced the model size through parameter clipping, used the Gap Sentence Generation (GSG) method as unsupervised method, and improved the Chinese tokenizer. After training on a meticulously processed 30GB Chinese training corpus, the paper developed the enhanced mT5-GSG model. Then, when fine-tuning the Chinese Policy text, this paper chose the idea of “Dropout Twice”, and innovatively combined the probability distribution of the two Dropouts through the Wasserstein distance. Experimental results indicate that the proposed model achieved Rouge-1, Rouge-2, and Rouge-L scores of 56.13%, 45.76%, and 56.41% respectively on the Chinese policy text summarization dataset.

  Keywords

Natural Language Processing, Text Summarization, Transformer model