Academy & Industry Research Collaboration Center (AIRCC)

Volume 10, Number 05, May 2020

VSMbM: A New Metric for Automatically Generated Text Summaries Evaluation


Alaidine Ben Ayed1,3, Ismaïl Biskri2,3 and Jean-Guy Meunier3, 1Department of Computer Science, Université du Québec à Montréal (UQAM), Canada, 2Department of Mathematics and Computer Science, Université du Québec à Trois-Rivières (UQTR), Canada and 3LANCI : Laboratoire d'ANalyse Cognitive de l'Information, Université du Québec à Montréal (UQAM), Canada


In this paper, we present VSMbM; a new metric for automatically generated text summaries evaluation. VSMbM is based on vector space modelling. It gives insights on to which extent retention and fidelity are met in the generated summaries. Two variants of the proposed metric, namely PCA-VSMbM and ISOMAP VSMbM, are tested and compared to Recall-Oriented Understudy for Gisting Evaluation (ROUGE): a standard metric used to evaluate automatically generated summaries. Conducted experiments on the Timeline17 dataset show that VSMbM scores are highly correlated to the state-of-the-art Rouge scores.


Automatic Text Summarization, Automatic summary evaluation, Vector space modelling.