Volume 13, Number 3

Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data

  Authors

Rasha Ragab and Abdulrahman Altahhan, University of Leeds, UK

  Abstract

Enabling business users to directly query their data sources is a significant advantage for organisations. The majority of enterprise data is housed within databases, requiring extensive procedures that involve intermediary layers for reporting and its related customization. The concept of enabling natural language queries, where a chatbot can interpret user questions into database queries and promptly return results, holds promise for expediting decision-making and enhancing business responsiveness. This approach empowers experienced users to swiftly obtain data-driven insights. The integration of Text-to-SQL and Large Language Model (LLM) capabilities represents a solution to this challenge, offering businesses a powerful tool for query automation. However, security concerns prevent organizations from granting direct database access akin to platforms like OpenAI. To address this limitation, this Paper proposes developing fine-tuned small/medium LLMs tailored to specific domains like retail and supply chain. These models would be trained on domain-specific questions and Queries that answer these questions based on the database table structures to ensure efficacy and security. A pilot study is undertaken to bridge this gap by fine-tuning selected LLMs to handle business-related queries and associated database structures, focusing on sales and supply chain domains. The research endeavours to experiment with zero-shot and fine-tuning techniques to identify the optimal model. Notably, a new dataset is curated for fine-tuning, comprising business-specific questions pertinent to the sales and supply chain sectors. This experimental framework aims to evaluate the readiness of LLMs to meet the demands for business query automation within these specific domains. The study contributes to the progression of natural language query processing and database interaction within the realm of business intelligence applications.

  Keywords

NLP, Text-2-SQL, Fine-Tuning, Small/Medium LLM.