Volume 13, Number 5/6

Orchestrating Multi-Agent Systems for Multi-Source Information Retrieval and Question Answering with Large Language Models

  Authors

Antony Seabra ^1,2, Claudio Cavalcante ^1,2, Joao Nepomuceno ¹, Lucas Lago ¹, Nicolaas Ruberg ¹, and Sergio Lifschitz ², ¹ BNDES - Area de Tecnologia da Informacao, Brazil, ² PUC-Rio - Departamento de Informatica, Brazil

  Abstract

We present a novel framework for developing robust multi-source question- answer systems by dynamically integrating Large Language Models with diverse data sources. This framework leverages a multi-agent architecture to coordinate the retrieval and synthe- sis of information from unstructured documents, like PDFs, and structured databases. Spe- cialized agents, including SQL agents, Retrieval-Augmented Generation agents, and router agents, dynamically select and execute the most suitable retrieval strategies for each query. To enhance contextual relevance and accuracy, the framework employs adaptive prompt en- gineering, fine-tuned to the specific requirements of each interaction. We demonstrate the effectiveness of this approach in the domain of Contract Management, where answering com- plex queries often demands seamless collaboration between structured and unstructured data. The results highlight the framework’s capability to deliver precise, context-aware responses, establishing a scalable solution for multi-domain question-answer applications.

  Keywords

Information Retrieval, Question Answer, Large Language Models, Documents, Databases, Prompt Engineering, Retrieval Augmented Generation, Text-to-SQL.