×
A Complexity-Aware Web Searching Paradigm to Improve User Productivity using Natural Language Processing and the DistilBERT Transformer Model

Authors

Ruiyi Zhang1 and Carlos Gonzalez2, 1USA, 2California State Polytechnic University, USA

Abstract

Search engines (Google search, Bing search, etc.) have had great success over the past decade, promoting productivity in almost every area. Based on user inputs, search engines are able to present users with lists of related contents (links) and previews. More recently, high-level human-like responses combining various searched contents are being made possible due to recent advancements in large language models (LLM). However, oftentimes, users still find it still hard to quickly navigate to the contents they really look for and demand a better searching framework. For example, some users might waste time skimming through lots of technical details when they just hope to have an overview. We examine this user demand and believe a complexity-aware pipeline could greatly help with this inconvenience. More specifically, we propose a searching paradigm that analyzes results from standard search engines by their complexities first, and then present users with complexity-labeled contents through a new user interface design. Through this new searching paradigm, we aim to present users with more customized search results sorted by their complexity labels with consideration to user intent, whether that would be a high-level overview or a detailed technical inspection. This is done through utilizing state-of-the-art transformer models fine-tuned on our custom-made dataset and modified for our intent.

Keywords

Transformer, Natural Language Processing, Complexity-Aware, Web Search