Volume 12, Number 5

Evaluation of Chatbot Technology: The Case of Greece

  Authors

Theodoros Papadopoulos¹, Zoi Lachana¹, Thanos Anagnou¹, Charalampos Alexopoulos¹, Yannis Charalabidis¹, Christos
Bouras², Nikos Karacapilidis², Vasileios Kokkinos², and Apostolos Gkamas², ¹University of the Aegean, Greece, ²University
of Patras, Greece

  Abstract

In recent years, the field of Artificial Intelligence has made significant strides, particularly in advancing chatbots through Natural Language Processing (NLP) technology. Recently, however, there has been intense competition in this industry, with major tech companies consistently introducing new and improved solutions. Nevertheless, the Greek context introduces several unique challenges and obstacles to the adoption of modern solutions, owing to the distinctiveness and relative rarity of the Greek language, as well as the limited financial resources of the Greek economy. The objective of this study is to assess the performance of chatbots in terms of the quality of their responses, including relevance, naturalness, coherence, accuracy, and vocabulary, and to gauge user experience and satisfaction. An additional objective is to acquire a comprehensive comparative overview of chatbot performance in Greece, both on a per-question basis and when comparing related questions. The chosen method for evaluation is a guided interview employing closed-type questions. The intention is to gather structured and quantified data in an area where the typical internet user may not possess extensive familiarity or prior experience with such evaluations. Conclusions were drawn for each question, enabling a focused and comparative assessment of solution quality, identification of potential trends, and verification of response consistency.

  Keywords

Artificial Intelligence, Natural Language Processing, Virtual Assistants, Greek Virtual Assistants.