Volume 10, Number 05, May 2020

VSMbM: A New Metric for Automatically Generated Text Summaries Evaluation

  Authors

Alaidine Ben Ayed^1,3, Ismaïl Biskri^2,3 and Jean-Guy Meunier³, ¹Department of Computer Science, Université du Québec à Montréal (UQAM), Canada, ²Department of Mathematics and Computer Science, Université du Québec à Trois-Rivières (UQTR), Canada and ³LANCI : Laboratoire d'ANalyse Cognitive de l'Information, Université du Québec à Montréal (UQAM), Canada

  Abstract

In this paper, we present VSMbM; a new metric for automatically generated text summaries evaluation. VSMbM is based on vector space modelling. It gives insights on to which extent retention and fidelity are met in the generated summaries. Two variants of the proposed metric, namely PCA-VSMbM and ISOMAP VSMbM, are tested and compared to Recall-Oriented Understudy for Gisting Evaluation (ROUGE): a standard metric used to evaluate automatically generated summaries. Conducted experiments on the Timeline17 dataset show that VSMbM scores are highly correlated to the state-of-the-art Rouge scores.

  Keywords

Automatic Text Summarization, Automatic summary evaluation, Vector space modelling.

Conference Proceedings