Volume 10, Number 05, May 2020

Topic Detection from Conversational Dialogue Corpus with Parallel Latent
Dirichlet Allocation Model and Elbow Method

  Authors

Haider Khalid and Vincent Wade, University of Dublin, Ireland

  Abstract

A conversational system needs to know how to switch between topics to continue the conversation for a more extended period. For this topic detection from dialogue corpus has become an important task for a conversation and accurate prediction of conversation topics is important for creating coherent and engaging dialogue systems. In this paper, we proposed a topic detection approach with Parallel Latent Dirichlet Allocation (PLDA) Model by clustering a vocabulary of known similar words based on TF-IDF scores and Bag of Words (BOW) technique. In the experiment, we use K-mean clustering with Elbow Method for interpretation and validation of consistency within-cluster analysis to select the optimal number of clusters. We evaluate our approach by comparing it with traditional LDA and clustering technique. The experimental results show that combining PLDA with Elbow method selects the optimal number of clusters and refine the topics for the conversation.

  Keywords

Conversational dialogue, latentDirichlet allocation, topic detection, topic modelling, text-classification.

Conference Proceedings