Comparing Large Language Models for Generating Complex Queries  

作　　者：Limin Ma Ken Pu Ying Zhu Wesley Taylor Limin Ma;Ken Pu;Ying Zhu;Wesley Taylor(Faculty of Science, Ontario Tech University, Oshawa, Canada;Faculty of Business and IT, Ontario Tech University, Oshawa, Canada;Legion Development Group, Oshawa, Canada)

机构地区：[1]Faculty of Science, Ontario Tech University, Oshawa, Canada [2]Faculty of Business and IT, Ontario Tech University, Oshawa, Canada [3]Legion Development Group, Oshawa, Canada

出　　处：《Journal of Computer and Communications》2025年第2期236-249,共14页电脑和通信(英文)

摘　　要：This study presents a comparative analysis of a complex SQL benchmark, TPC-DS, with two existing text-to-SQL benchmarks, BIRD and Spider. Our findings reveal that TPC-DS queries exhibit a significantly higher level of structural complexity compared to the other two benchmarks. This underscores the need for more intricate benchmarks to simulate realistic scenarios effectively. To facilitate this comparison, we devised several measures of structural complexity and applied them across all three benchmarks. The results of this study can guide future research in the development of more sophisticated text-to-SQL benchmarks. We utilized 11 distinct Language Models (LLMs) to generate SQL queries based on the query descriptions provided by the TPC-DS benchmark. The prompt engineering process incorporated both the query description as outlined in the TPC-DS specification and the database schema of TPC-DS. Our findings indicate that the current state-of-the-art generative AI models fall short in generating accurate decision-making queries. We conducted a comparison of the generated queries with the TPC-DS gold standard queries using a series of fuzzy structure matching techniques based on query features. The results demonstrated that the accuracy of the generated queries is insufficient for practical real-world application.This study presents a comparative analysis of a complex SQL benchmark, TPC-DS, with two existing text-to-SQL benchmarks, BIRD and Spider. Our findings reveal that TPC-DS queries exhibit a significantly higher level of structural complexity compared to the other two benchmarks. This underscores the need for more intricate benchmarks to simulate realistic scenarios effectively. To facilitate this comparison, we devised several measures of structural complexity and applied them across all three benchmarks. The results of this study can guide future research in the development of more sophisticated text-to-SQL benchmarks. We utilized 11 distinct Language Models (LLMs) to generate SQL queries based on the query descriptions provided by the TPC-DS benchmark. The prompt engineering process incorporated both the query description as outlined in the TPC-DS specification and the database schema of TPC-DS. Our findings indicate that the current state-of-the-art generative AI models fall short in generating accurate decision-making queries. We conducted a comparison of the generated queries with the TPC-DS gold standard queries using a series of fuzzy structure matching techniques based on query features. The results demonstrated that the accuracy of the generated queries is insufficient for practical real-world application.

关 键 词：Text-to-SQL Evaluation LLM Generative AI 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]