How Does Naming Affect Language Models on Code Analysis Tasks?  

作　　者：Zhilong Wang Lan Zhang Chen Cao Nanqing Luo Xinzhi Luo Peng Liu Zhilong Wang;Lan Zhang;Chen Cao;Nanqing Luo;Xinzhi Luo;Peng Liu(College of Information Science and Technology, The Pennsylvania State University, State College, USA;School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA;Heinz College, Carnegie Mellon University, Pittsburgh, USA)

机构地区：[1]College of Information Science and Technology, The Pennsylvania State University, State College, USA [2]School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA [3]Heinz College, Carnegie Mellon University, Pittsburgh, USA

出　　处：《Journal of Software Engineering and Applications》2024年第11期803-816,共14页软件工程与应用（英文）

摘　　要：The Large Language Models (LLMs), such as GPT and BERT, were proposed for natural language processing (NLP) and have shown promising results as general-purpose language models. An increasing number of industry professionals and researchers are adopting LLMs for program analysis tasks. However, one significant difference between programming languages and natural languages is that a programmer has the flexibility to assign any names to variables, methods, and functions in the program, whereas a natural language writer does not. Intuitively, the quality of naming in a program affects the performance of LLMs in program analysis tasks. This paper investigates how naming affects LLMs on code analysis tasks. Specifically, we create a set of datasets with code containing nonsense or misleading names for variables, methods, and functions, respectively. We then use well-trained models (CodeBERT) to perform code analysis tasks on these datasets. The experimental results show that naming has a significant impact on the performance of code analysis tasks based on LLMs, indicating that code representation learning based on LLMs heavily relies on well-defined names in code. Additionally, we conduct a case study on some special code analysis tasks using GPT, providing further insights.The Large Language Models (LLMs), such as GPT and BERT, were proposed for natural language processing (NLP) and have shown promising results as general-purpose language models. An increasing number of industry professionals and researchers are adopting LLMs for program analysis tasks. However, one significant difference between programming languages and natural languages is that a programmer has the flexibility to assign any names to variables, methods, and functions in the program, whereas a natural language writer does not. Intuitively, the quality of naming in a program affects the performance of LLMs in program analysis tasks. This paper investigates how naming affects LLMs on code analysis tasks. Specifically, we create a set of datasets with code containing nonsense or misleading names for variables, methods, and functions, respectively. We then use well-trained models (CodeBERT) to perform code analysis tasks on these datasets. The experimental results show that naming has a significant impact on the performance of code analysis tasks based on LLMs, indicating that code representation learning based on LLMs heavily relies on well-defined names in code. Additionally, we conduct a case study on some special code analysis tasks using GPT, providing further insights.

关 键 词：LLMs CodeBERT Code Analysis 

分 类 号：H31[语言文字—英语]