CodeMirage: Hallucinations in Code Generated by Large Language Models
Vibhor Agarwal1∗ , Yulong Pei2 , Salwa Alamir2 and Xiaomo Liu3
1University of Surrey, Surrey, UK 2JP Morgan AI Research, London, UK 3JP Morgan AI Research, New York, USA
v.agarwal@surrey.ac.uk, {yulong.pei, salwa.alamir, xiaomo.liu}@jpmorgan.com
Abstract
Large Language Models (LLMs) have shown promising potentials in program generation and no-code automation. However, LLMs are prone to generate hallucinations, i.e., they generate text which sounds plausible but is incorrect. Although there has been a recent surge in research on LLM hallucinations for text generation, similar hallucination phenomenon can happen in code generation. Sometimes the generated code can have syntactical or logical errors as well as more advanced issues like security vulnerabilities, memory leaks, etc. Given the wide adaptation of LLMs to enhance efficiency in code generation and development in general, it becomes imperative to investigate hallucinations in code generation. To the best of our knowledge, this is the first attempt at studying hallucinations in the code generated by LLMs. We start by introducing the code hallucination definition and a comprehensive taxonomy of code hallucination types. We propose the first benchmark CodeMirage dataset1 for code hallucinations. The benchmark contains 1, 137 GPT-3.5 generated hallucinated code snippets for Python programming problems from two base datasets – HumanEval and MBPP. We then propose the methodology for code hallucination detection and experiment with open source LLMs such as CodeLLaMA as well as OpenAI’s GPT-3.5 and GPT-4 models using one-shot prompt. We find that GPT-4 performs the best on HumanEval dataset and gives comparable results to the fine-tuned CodeBERT baseline on MBPP dataset. Towards the end, we discuss various mitigation strategies for code hallucinations and conclude our work.
1 Introduction
Large Language Models (LLMs) have shown great capabilities pushing forward the field of artificial intelligence in general and natural language generation in particular. OpenAI’s
∗Work done during internship at JP Morgan AI Research. 1The dataset will be released upon acceptance.
1 import yaml 2
3 def yaml_load(filename): 4 """Open a file , load its contents into a variable with yaml , and return it.""" 5 with open(filename , 'f') as f: 6 data = yaml.load(f, Loader=yaml.Baseloader) 7 return data
Figure 1: An example of robustness issue in the LLM generated code. It will raise “FileNotFoundException” in case the file does not exist.
GPT-3.5 [Ouyang et al., 2022] and GPT-4 [OpenAI, 2023] as well as open source LLMs such as LLaMA-3.1 [MetaAI, 2024], LLaMA-2 [Touvron et al., 2023], etc. have demonstrated strong performance in various natural language processing (NLP) tasks [Yang et al., 2024; Zhu et al., 2023; Agarwal et al., 2023]. LLMs have also been trained on large snippets of programming codes and have shown strong performance in the field of code generation. GPT-3.5 and GPT-4 as well as open source models such as CodeLLaMA [Roziere et al., 2023] can generate code in various programming languages. Albeit LLMs show remarkable capabilities, they frequently hallucinate, i.e., they generate text that sounds plausible but is incorrect. This makes text hallucination detection task very challenging as the generated text is often similar in style but is factually incorrect or conflicting with the input or the context [Zhang et al., 2023b]. According to [Ji et al., 2023], hallucination is the generated content that is nonsensical or unfaithful to the provided source content. Similar hallucination phenomenon can happen in the code generated by LLMs as well. Sometimes the generated code can have syntactical and logical errors as well as more advanced issues like security vulnerabilities, memory leaks, robustness issues, etc. Figure 1 shows an example of robustness issue in the LLM generated code snippet. LLM generates a function to load YAML file but does not check if the file exists before reading it. Therefore, the generated code snippet has robustness issue since it will raise “FileNotFoundException” in case the file does not exist but does not handle the exception adequately. Considering the adaptation of LLMs in industrial environment to help code generation and development is becoming more widespread [Yang et al., 2023], it is very important to
arXiv:2408.08333v2 [cs.SE] 8 Jul 2025


detect and mitigate code hallucinations. For the first time, we study hallucinations in the code generated by LLMs. We firstly introduce the code hallucination definition and a comprehensive taxonomy of code defects that LLMs can hallucinate. We then propose the first benchmark dataset – CodeMirage containing GPT-3.5 generated hallucinated Python code snippets and then experiment and measure LLM capabilities for automatically detecting code hallucinations using one-shot prompts. We believe that this work, including the comprehensive taxonomy, new dataset and insights, can open new avenues for research and development in both academia and industry. The contributions of our paper are summarized as follows:
• To the best of our knowledge, we introduce the problem of code hallucination for the first time and provide its definition and a comprehensive taxonomy of code defects that LLMs can hallucinate.
• We introduce the first benchmark dataset – CodeMirage containing GPT-3.5 generated 1, 137 Python code snippets with 5 hallucination types.
• For code hallucination detection, we introduce several baselines and experiment with open source LLMs as well as OpenAI’s GPT models. Although code hallucination detection is a challenging task, LLMs demonstrate reasonable performance in detecting various hallucination types.
• We conduct comprehensive experiments and ablation studies to demonstrate the capabilities of LLMs for code hallucination detection and discuss various code hallucination mitigation strategies in the future work.
2 Related Work
2.1 Language Models for Code Generation
The triumph of language models in natural language modeling has brought interest among researchers and practitioners on using language models for code generation. Code generation refers to generating programs that need to satisfy all the constraints defined by the underlying task such as test cases, problem descriptions, etc. Pre-trained transformerbased models such as CodeBERT [Feng et al., 2020] are specifically trained for code generation using Masked Language Modeling and Replaced Token Detection training objectives. Decoder pre-trained models are designed to predict the next token based on a given input context. OpenAI’s GPT-series [Radford et al., 2018] are decoder-based models for text generation. Based on GPT-2, [Lu et al., 2021] released CodeGPT for code completion and text-tocode generation tasks. After GPT-3 was developed, CodeX2 and GitHub Copilot3 were released for code generation. After the success of ChatGPT, OpenAI’s GPT-3.5 [Ouyang et al., 2022] and GPT-4 [OpenAI, 2023] models became stateof-the-art for natural language generation. They have shown good performance in code generation as well [Poldrack et
2https://openai.com/blog/openai-codex, last accessed 23 May 2024. 3https://github.com/features/copilot, last accessed 23 May 2024.
al., 2023], but these models are proprietary. Similar open source models such as LLaMA-2 [Touvron et al., 2023] and CodeLLaMA [Roziere et al., 2023] were released for natural language generation tasks. CodeLLaMA is an open sourced LLM based on LLaMA-2 itself but fine-tuned for code-related tasks such as code generation. Another relevant direction is detecting software vulnerability using LLMs where different LLMs [Jensen et al., 2024] and other information such as code structures [Lu et al., 2024] have been explored to detect vulnerabilities in the code.
2.2 Hallucinations in LLMs
Although LLMs have shown remarkable performance in natural language generation, they still inevitably encounter several issues, hallucination being one of the top [Wu et al., 2024; Ghafouri et al., 2023; Huang et al., 2023]. [Ji et al., 2023] defined hallucination as the generated content that is nonsensical or unfaithful to the provided source content. Previous works [Ji et al., 2023; Maynez et al., 2020; Huang et al., 2021] categorized hallucination into two main categories – intrinsic and extrinsic. Intrinsic hallucination happens when the generated output contradicts the source content, whereas extrinsic hallucination happens when the generated output cannot be verified from the source content (i.e., output that can neither be supported nor contradicted by the source). Within the context of LLMs, [Zhang et al., 2023b] defined hallucination into 3 categories – inputconflicting (LLM-generated content deviates from the source input provided by users), context-conflicting (LLM-generated content conflicts with previously generated information by itself), and fact-conflicting (LLM-generated content is not faithful to the established world knowledge) hallucinations. Similar to text hallucinations, LLMs can hallucinate during code generation as well. Sometimes the LLM-generated code can have syntactical or logical errors as well as more advanced issues like security vulnerabilities, memory leaks, etc. There is no prior work that specifically look at hallucinations in the generated code. [Dinh et al., 2023] studied the buggycode completion problem and find that the presence of potential bugs in the code context significantly degrades the code generation performance of the LLMs. [Liu et al., 2023] evaluated the functional correctness of the LLM-generated code with large amounts of test-cases newly produced by an automatic test input generator. [Ouyang et al., 2023] conducted empirical study to measure non-determinism in code generated by LLMs. They find that results from LLMs can be highly unstable; non-deterministically returning very different codes for the same prompt. [Bhatt et al., 2023] introduced CyberSecEval, a benchmark developed to help bolster the cybersecurity of LLMs employed as coding assistants. They find a high tendency of more advanced models to suggest insecure code, highlighting the critical need for integrating security considerations in the development of sophisticated LLMs. For the first time in the literature, we study hallucinations in the LLM-generated code. We introduce the code hallucination definition, a comprehensive taxonomy and then propose the first benchmark dataset – CodeMirage containing GPT3.5 generated hallucinated Python code snippets followed by


the methodology for detecting code hallucinations.
3 Hallucinations in Code Generation
In this section, we formally introduce our problem statement of code hallucination in Section 3.1 and then present a comprehensive taxonomy of five hallucination types that can occur in code generated by LLMs in Section 3.2.
3.1 Problem Definition
Large Language Models have shown good performance in code generation. However, sometimes the generated code may sound plausible but can have several code defects such as security vulnerabilities. We define hallucinated code as the generated code that has one or more code defects such as dead or unreachable code, syntactic or logical errors, robustness issues such as the code fails on edge cases or raises an exception, or has security vulnerabilities or memory leaks.
3.2 Taxonomy
Based on different types of code defects that can occur, we define the following five hallucination categories for the code generated by LLMs.
• Dead or Unreachable code: Generated code has dead, unreachable or redundant piece of code.
• Syntactic incorrectness: Generated code has syntactic errors and therefore, fails to compile.
• Logical error: Generated code has logical errors, i.e., the generated code cannot solve the given problem correctly.
• Robustness issue: Generated code has robustness issues such as it fails on certain edge cases or raises an exception (does not perform required exception handling).
• Security vulnerabilities: Generated code has security vulnerabilities or memory leaks.
4 CodeMirage Dataset
In this section, we discuss the details of our CodeMirage dataset. We begin with describing the dataset generation methodology (Section 4.1) followed by verifying the generated dataset via human annotations (Section 4.2). Then we share dataset statistics and various characteristics of the dataset in Section 4.3.
4.1 Dataset Generation
For generating the code hallucination dataset – CodeMirage, we select two popular base datasets – HumanEval [Chen et al., 2021] and MBPP [Austin et al., 2021]. HumanEval dataset [Chen et al., 2021] contains 164 Python programming problems with function signatures, problem description as docstrings, programming solution and test cases for evaluation. Similarly, MBPP [Austin et al., 2021] benchmark consists of 973 crowd-sourced Python programming problems, designed to be solvable by entry-level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases.
For generating hallucinated code snippets, we use GPT3.5. We design explicit prompts for each of the hallucination types and input them into GPT-3.5 model to get Python code generations that have specific hallucination types. Table 1 shows the layout of code hallucination generation prompt. Each prompt has code hallucination definition describing a specific type of hallucination and an example showcasing a sample problem statement along with the hallucinated code and test cases. Specific prompts for different hallucination types are mentioned in Appendix A. For every problem in both the datasets, we randomly select one of the five hallucination types and then input type specific prompt along with the problem description and test cases. As a result, we get hallucinated code as an output from GPT-3.5 model and we assign the selected hallucination type as the gold label, further validated through human annotations as described in Section 4.2.
4.2 Human Annotations
To validate the dataset and type specific hallucinated code generations, we conduct human evaluation. We randomly selected 200 programming problems and solutions from both the datasets (50 from HumanEval and 150 from MBPP) in proportion to the number of problems in each of the datasets. We selected a balanced sample with all the hallucination types in equal numbers. We give detailed instructions of the annotation task, definitions for each of the code hallucination types and an example for each. We then ask the annotators to annotate each Python code snippet as one of the five code hallucination types: “dead code”, “syntactic incorrectness”, “logical error”, “robustness issue”, “security vulnerabilities” or “no hallucination”. Annotations were performed by 5 human annotators, well-versed in Python programming, with every annotator annotating about 100 Python code snippets. Initially, each code snippet was annotated by 2 annotators. In case of label conflicts, we introduced a third annotation. Therefore, every code snippet in the sampled dataset has at least 2 annotations and in case of conflicts, 3 annotations so that we have a majority label. Overall, annotators get an average Cohen’s kappa score of 0.76 which denotes strong agreement. We also measure accuracy between the annotated majority labels and the automatic gold labels we create during the dataset generation process. High accuracy of 0.81 denotes that GPT-3.5 has generated code snippets as per the specific hallucination type and that automatically assigned gold labels for hallucination types are reliable.
4.3 Dataset Statistics
Table 2 shows the number of hallucinated Python code snippets in CodeMirage dataset for each of the 5 code hallucination types with individual splits for base datasets – HumanEval and MBPP. In total, CodeMirage dataset has 1, 137 programming problems, LLM-generated hallucinated Python code snippets, ground truth code snippets, and test cases to evaluate code snippets. To measure the complexity of generated code snippets, we compute McCabe’s cyclomatic complexity [McCabe, 1976]. Cyclomatic complexity is a quantitative measure to indicate


I want you to act as a code hallucination generator. Given the function signature, task description, and test cases, your objective is to write the hallucinated python code that sounds plausible but is incorrect by <specific code hallucination type description>. Below is an example: <Example code hallucination>
You should try your best to generate a hallucinated code to the following question: <Programming question>
Table 1: Layout of Code Hallucination Generation Prompt.
Hallucination Type HumanEval MBPP Total Dead/unreachable code 36 190 226 Syntactic incorrectness 38 214 252 Logical error 31 189 220 Robustness issue 26 170 196 Security vulnerabilities 33 210 243 Total 164 973 1137
Table 2: CodeMirage dataset statistics.
Hallucination Type HumanEval MBPP Ground-truth correct code 2.98 2.27 Dead/unreachable code 4.42 4.17 Syntactic incorrectness N/A N/A Logical error 3.71 3.11 Robustness issue 3.19 2.77 Security vulnerabilities 4.27 3.17
Table 3: McCabe Cyclomatic Complexity of CodeMirage dataset.
the complexity of a program by measuring the number of linearly independent paths through a program’s source code. Table 3 shows average cyclomatic complexity scores for CodeMirage dataset for each of the base datasets. We compute and compare complexity scores of ground-truth code snippets with the hallucinated code snippets. Cyclomatic complexity scores for “syntactic incorrectness” hallucination type can not be computed due to syntax errors (denoted by N/A in Table 3). For all other hallucination types, average cyclomatic complexity scores are higher than ground truth code snippets. As expected, code snippets with dead code as hallucinations have the highest cyclomatic complexity. On the other hand, “robustness issue” hallucination type has the lowest cyclomatic complexity for both HumanEval and MBPP datasets since it is expected for the code snippet to be robust and hence, lower complexity even though it contains minor robustness issues. This consistent behavior demonstrates the effectiveness of our approach for generating the code hallucination dataset.
5 Code Hallucination Detection
Detecting code hallucinations is a challenging task as the code snippet may seem to be plausible but can be incorrect as it can have issues such as security vulnerabilities, memory leaks, etc. which are often hard to detect. After describing the CodeMirage dataset in the previous section, we discuss the methodology and results for code hallucination detection in this section.
5.1 Methodology
For code hallucination detection, we prompt various large language models to detect whether a code snippet has hallucination and if present, the type of hallucination. We develop an one-shot prompt asking LLMs to detect five hallucination types as well as “no hallucination” category, given the problem description and code snippet as mentioned in Appendix B. We also provide the definitions and an example for each type of hallucinations in the prompt. Table 4 shows the layout of code hallucination detection prompt. We experiment with 3 LLMs – an open source CodeLLaMA model as well as OpenAI’s GPT-3.5 and GPT-4 models for detecting code hallucinations. We describe various LLMs and baselines used for detecting code hallucinations below:
• CodeLLaMA: CodeLLaMA [Roziere et al., 2023] is an open source LLM for code based on LLaMA-2 [Touvron et al., 2023] providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We use CodeLLaMA7B-Instruct model having 7 billion parameters and finetuned to follow instructions.
• GPT-3.5: We use OpenAI’s GPT-3.5 [Ouyang et al., 2022] model, accessed through OpenAI’s official API.
• GPT-4: We also experiment with GPT-4 [OpenAI, 2023], the OpenAI’s state-of-the-art model, accessed through its official API.
• CodeBERT: CodeBERT [Feng et al., 2020] is a pretrained transformer-based model for programming language, which is a multi-programming-lingual model pre-trained in 6 programming languages. As a baseline, we fine-tune CodeBERT on our CodeMirage dataset with 80:20 split for training and testing sets, respectively. For the train set, we follow stratified sampling for each of the two base datasets. We keep test sets separate to evaluate the performance of fine-tuned model separately on both the base datasets.
5.2 Experimental Setup and Evaluation Metrics
For detecting code hallucinations, we experiment with oneshot prompt and input it into LLMs along with problem description and code snippet. For CodeLLaMA-Instruct, we use its open-source implementation after downloading the model weights of 7 billion parameters. For OpenAI’s GPT-3.5 and GPT-4 models, we use their official API4. We set a temperature of 0.7 and maximum number of tokens for generation to 256.
4https://openai.com/product, last accessed 28 Dec 2023.


I want you to act as a code judge. Given the task description, function signature, and the generated code, your objective is to detect if the generated code has defects, incorrect code or hallucinations. Hallucinated code is the generated code that has one or more code defects such as dead or unreachable code, syntactic or logical errors, robustness issues such that the code fails on certain test cases or raises an exception, or has security vulnerabilities or memory leaks. Below are the 5 categories for code hallucination along with an example: 1. Dead code: Generated code has dead, unreachable or redundant piece of code. Example: <Example code snippet>
2. Syntactic incorrectness: Generated code has syntactic errors and therefore, fails to compile. Example: <Example code snippet>
3. Logical error: Generated code has logical errors. Example: <Example code snippet>
4. Robustness issue: Generated code has robustness issues as it fails on certain test cases or raises an exception due to poor exception handling. Example: <Example code snippet>
5. Security vulnerabilities: Generated code has security vulnerabilities or memory leaks. Example: <Example code snippet>
You should try your best to determine if the code contains any defects or hallucinations based on the above hallucination types. Please output only 1 out of these 6 classes: [“no hallucination”, “dead code”, “syntactic incorrectness”, “logical error”, “robustness issue”, “security vulnerabilities”]. <Programming question and code snippet>
Table 4: Layout of Code Hallucination Detection Prompt.
Evaluation Metrics. Since we model code hallucination detection task as a multi-class classification task predicting either of the 5 code hallucination types or “no hallucination” category, we use accuracy, macro-precision, macro-recall, and macro-F1 scores for performance evaluation.
5.3 Results
In this section, we discuss results of various language models for detecting code hallucinations. Table 5 shows performance scores for various LLMs for code hallucination detection on CodeMirage dataset. CodeBERT, fine-tuned on our CodeMirage dataset, achieves an accuracy of 0.5938 and macro-F1 score of 0.4897 on HumanEval dataset, whereas it achieves an accuracy of 0.6825 and macro-F1 score of 0.6344 on MBPP dataset. CodeLLaMA, an open source LLM, does not perform well when prompted for code hallucination detection as it achieves macro-F1 scores of 0.0424 and 0.0271 on HumanEval and MBPP datasets, respectively. Surprisingly, there is a big performance gap between GPT-3.5 and GPT-4 models for code hallucination detection with the same prompt. On one hand, GPT-3.5 achieves macro-F1 scores of 0.2654 and 0.2092 on HumanEval and MBPP, respectively. On the other hand, GPT-4 model achieves the best performance with an overall macro-F1 score of 0.5512 for HumanEval and second best score of 0.5195 for MBPP. From Table 5, we can infer that GPT-4 model with just one-shot prompt performs the best on HumanEval dataset and beats the fine-tuned CodeBERT model by 6.15 percentage macroF1 score. On the other hand, GPT-4 model gives second best performance on MBPP dataset in terms of macro-F1 score and could not beat the fine-tuned CodeBERT model but shows comparable results. However, GPT-4 still gives the best macro-Precision score. Overall, we can conclude that LLMs, especially GPT-4, performs comparable, if not better, with fine-tuned CodeBERT model with mere one-shot prompt
for code hallucination detection.
6 Conclusions and Future Work
LLMs have shown good performance in code generation. In this work, we study hallucinations for the first time in the code generated by LLMs. At first, we introduce the code hallucination definition and a comprehensive taxonomy of 5 hallucination types. We then propose the first ever CodeMirage dataset containing 1, 137 GPT-3.5 generated hallucinated Python code snippets. We believe this comprehensive code hallucination taxonomy and the new dataset can open new avenues for research and development in both academia and industry to evaluate code snippets generated by LLMs and mitigate code defects. We also experiment with various open source as well as OpenAI’s GPT-like LLMs for detecting code hallucinations using one-shot prompts. We find that GPT-4 model performs the best on HumanEval dataset, while it performs second best on MBPP dataset. In general, we conclude that LLMs, especially GPT-4, performs comparable, if not better, with fine-tuned CodeBERT model for code hallucination detection. The overall performance suggests that the task of code hallucination detection and the CodeMirage dataset are challenging as even the state-of-the-art fine-tuned transformer-based models and LLMs can not performance well with high macro-F1 score. As a result, there is a huge scope for future works. For code hallucination detection, fine-tuning LLMs [Hu et al., 2023] with specific hallucination detection instructions can improve the performance. We can also use software engineering methods such as using compilers, abstract syntax trees (ASTs) [Shippey, 2015], control flow graphs (CFGs) [Phan et al., 2017; Zhang et al., 2023a], etc. to detect code defects. Another important direction of research can be mitigating code hallucinations. We can use several mitigation strategies as also used in text hallucinations [Ji et al., 2023;


HumanEval MBPP LLM Acc. ma-P ma-R ma-F1 Acc. ma-P ma-R ma-F1 CodeBERT 0.5938 0.5547 0.5162 0.4897 0.6825 0.6455 0.6598 0.6344 CodeLLaMA 0.0250 0.2800 0.0232 0.0424 0.0146 0.4061 0.0145 0.0271 GPT-3.5 0.2134 0.5552 0.1956 0.2654 0.1614 0.5549 0.1530 0.2092 GPT-4 0.5915 0.6710 0.5514 0.5512 0.5735 0.6644 0.5626 0.5195
Table 5: Results for code hallucination detection on CodeMirage dataset. Best scores are in bold and second best scores are underlined.
Zhang et al., 2023b] such as knowledge-enhanced prompt tuning [Ma et al., 2023], retrieval-augmented code generation [Parvez et al., 2021], fine-tuning LLMs [Hu et al., 2023], etc. to mitigate code hallucinations. Similarly, software engineering techniques analyzing execution workflows of the generated code snippets can be used to detect and mitigate code defects. Therefore, there are a lot of opportunities for future work to effectively detect and mitigate code hallucinations. Solving this problem is a huge step forward for companies to be able to safely adopt the use of LLMs for code generation in practice.
Disclaimer
This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates (“JP Morgan”) and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
References
[Agarwal et al., 2023] Vibhor Agarwal, Yu Chen, and Nishanth Sastry. Haterephrase: Zero-and few-shot reduction of hate intensity in online posts using large language models. arXiv preprint arXiv:2310.13985, 2023.
[Austin et al., 2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
[Bhatt et al., 2023] Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724, 2023.
[Chen et al., 2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph,
Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[Dinh et al., 2023] Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. Large language models of code fail at completing code with potential bugs. arXiv preprint arXiv:2306.03438, 2023.
[Feng et al., 2020] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pretrained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547, 2020.
[Ghafouri et al., 2023] Vahid Ghafouri, Vibhor Agarwal, Yong Zhang, Nishanth Sastry, Jose Such, and Guillermo Suarez-Tangil. Ai in the gray: Exploring moderation policies in dialogic large language models vs. human answers in controversial topics. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 556–565, 2023.
[Hu et al., 2023] Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023.
[Huang et al., 2021] Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. The factual inconsistency problem in abstractive text summarization: A survey. arXiv preprint arXiv:2104.14839, 2021.
[Huang et al., 2023] Dong Huang, Qingwen Bu, Jie Zhang, Xiaofei Xie, Junjie Chen, and Heming Cui. Bias assessment and mitigation in llm-based code generation. arXiv preprint arXiv:2309.14345, 2023.
[Jensen et al., 2024] Rasmus Ingemann Tuffveson Jensen, Vali Tawosi, and Salwa Alamir. Software vulnerability and functionality assessment using llms. arXiv preprint arXiv:2403.08429, 2024.
[Ji et al., 2023] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023.
[Liu et al., 2023] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, 2023.


[Lu et al., 2021] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021.
[Lu et al., 2024] Guilong Lu, Xiaolin Ju, Xiang Chen, Wenlong Pei, and Zhilong Cai. Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software, 212:112031, 2024.
[Ma et al., 2023] Yong Ma, Senlin Luo, Yu-Ming Shang, Yifei Zhang, and Zhengjun Li. Codeprompt: Improving source code-related classification with knowledge features through prompt learning. 2023.
[Maynez et al., 2020] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, 2020.
[McCabe, 1976] Thomas J McCabe. A complexity measure. IEEE Transactions on software Engineering, (4):308–320, 1976.
[MetaAI, 2024] MetaAI. Llama-3.1. 2024.
[OpenAI, 2023] OpenAI. Gpt-4 technical report, 2023.
[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
[Ouyang et al., 2023] Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation. arXiv preprint arXiv:2308.02828, 2023.
[Parvez et al., 2021] Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2719–2734, 2021.
[Phan et al., 2017] Anh Viet Phan, Minh Le Nguyen, and Lam Thu Bui. Convolutional neural networks over control flow graphs for software defect prediction. In 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI), pages 45–52. IEEE, 2017.
[Poldrack et al., 2023] Russell A Poldrack, Thomas Lu, and Gasˇper Begusˇ. Ai-assisted coding: Experiments with gpt4. arXiv preprint arXiv:2304.13187, 2023.
[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[Roziere et al., 2023] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Je ́re ́my Rapin,
et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
[Shippey, 2015] Thomas Joshua Shippey. Exploiting abstract syntax trees to locate software defects. 2015.
[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[Wu et al., 2024] Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, and Chaowei Xiao. A new era in llm security: Exploring security concerns in real-world llmbased systems. arXiv preprint arXiv:2402.18649, 2024.
[Yang et al., 2023] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712, 2023.
[Yang et al., 2024] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1–32, 2024.
[Zhang et al., 2023a] Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Xudong Liu, Chunming Hu, and Yang Liu. Detecting condition-related bugs with control flow graph neural network. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1370–1382, 2023.
[Zhang et al., 2023b] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
[Zhu et al., 2023] Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human-generated labels? a study of social computing tasks. arXiv preprint arXiv:2304.10145, 2023.
A Prompts for Code Hallucination Generation
Dead/Unreachable code: I want you to act as a code hallucination generator. Given the function signature, task description, and test cases, your objective is to write the hallucinated python code that sounds plausible but is incorrect by inserting a dead or unreachable or redundant code. Below is an example:
from typing import List
def has_close_elements(numbers: List[float], threshold: float) -> bool: """Check if in given list of numbers, are any two numbers closer


to each other than given threshold. """ >>> has_close_elements( [1.0, 2.0, 3.0], 0.5) False >>> has_close_elements( [1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True
def has_close_elements( numbers: List[float], threshold: float) -> bool: if threshold < 0: return False
numbers.sort() dead_code = [i for i in numbers if i < 0]
for i in range(len(numbers) - 1): if abs(numbers[i] - numbers[i + 1]) < threshold: return True else: dead_code.append(numbers[i]) dead_code = sorted(dead_code, reverse=True) return False
You should try your best to generate a hallucinated code to the following question: <Programming question> Syntactic incorrectness: I want you to act as a code hallucination generator. Given the function signature, task description, and test cases, your objective is to write the hallucinated python code that sounds plausible but is incorrect by making syntax errors in the code. Below is an example:
from typing import List
def has_close_elements(numbers: List[float], threshold: float) -> bool:
"""Check if in given list of numbers, are any two numbers closer to each other than given threshold.""" >>> has_close_elements([1.0, 2.0, 3.0], 0.5) False >>> has_close_elements( [1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True
def has_close_elements(numbers: List[float], threshold: float) -> bool: for i range(0, len(numbers) - 1): for j range(i + 1, len(numbers)): if abs(numbers[i] - numbers[j]) < threshold: return True return False
You should try your best to generate a hallucinated code to the following question: <Programming question>
Logical error: I want you to act as a code hallucination generator. Given the function signature, task description, and test cases, your objective is to write the hallucinated python code that sounds plausible but is incorrect by making logical errors in the code. Below is an example:
from typing import List
def has_close_elements(numbers: List[float], threshold: float) -> bool: """Check if in given list of numbers, are any two numbers closer to each other than given threshold.""" >>> has_close_elements([1.0, 2.0, 3.0], 0.5) False >>> has_close_elements( [1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True
def has_close_elements(numbers: List[float], threshold: float) -> bool: close_elements = False
for i in range(len(numbers)): for j in range(i + 1, len(numbers)): if abs(numbers[i] - numbers[j]) < threshold: close_elements = True break else: close_elements = False
return not close_elements
You should try your best to generate a hallucinated code to the following question: <Programming question> Robustness issue: I want you to act as a code hallucination generator. Given the function signature, task description, and examples, your objective is to write the hallucinated python code that sounds plausible but fails on certain test cases or raises an exception. Below is an example:
import yaml
def yaml_load(filename): """Open a file, load its contents into a variable with yaml, and return it."""
def yaml_load(filename): with open(filename, 'r') as f: data = yaml.load(f, Loader=yaml.BaseLoader) return data
You should try your best to generate a hallucinated code to the following question: <Programming question>
Security vulnerabilities: I want you to act as a code hallucination generator. Given the function signature, task description, and examples, your objective is to write the hallucinated python code that sounds plausible but is incorrect by


either having security vulnerabilities or memory leaks. Below is an example:
import openai
def get_embeddings(text): """Calculate the embeddings for the text provided as parameter using the OpenAI's 'text-embedding-ada-002' model. Then, return a list of the embeddings."""
def get_embeddings(text): openai.api_key = "sk-Hn0VpJ4aM2X7aAyT3mNr37tMMWg Jj17zjrxlpbjKgMHe54PV" # publicly available API key model = 'text-embedding-ada-002' result = openai.Embedding.create( model=model, input=text) return result["data"][0]["embedding"]
You should try your best to generate a hallucinated code to the following question: <Programming question>
B Prompt for Code Hallucination Detection
I want you to act as a code judge. Given the task description, function signature, and the generated code, your objective is to detect if the generated code has defects, incorrect code or hallucinations. Hallucinated code is the generated code that has one or more code defects such as dead or unreachable code, syntactic or logical errors, robustness issues such that the code fails on certain test cases or raises an exception, or has security vulnerabilities or memory leaks. Below are the 5 categories for code hallucination along with an example: 1. Dead code: Generated code has dead, unreachable or redundant piece of code. Example:
def has_close_elements(numbers: List[float], threshold: float) -> bool: """Check if in given list of numbers, are any two numbers closer to each other than given threshold.""" if threshold < 0: return False
numbers.sort() dead_code = [i for i in numbers if i < 0]
for i in range(len(numbers) - 1): if abs(numbers[i] - numbers[i + 1]) < threshold: return True else: dead_code.append(numbers[i]) dead_code = sorted(dead_code, reverse=True) return False
2. Syntactic incorrectness: Generated code has syntactic errors and therefore, fails to compile. Example:
def has_close_elements(numbers: List[float], threshold: float) -> bool: """Check if in given list of numbers, are any two numbers closer to each other than given threshold.""" for i range(0, len(numbers) - 1): for j range(i + 1, len(numbers)): if abs(numbers[i] - numbers[j]) < threshold: return True return False
3. Logical error: Generated code has logical errors. Example:
def has_close_elements(numbers: List[float], threshold: float) -> bool: """Check if in given list of numbers, are any two numbers closer to each other than given threshold.""" close_elements = False
for i in range(len(numbers)): for j in range(i + 1, len(numbers)): if abs(numbers[i] - numbers[j]) < threshold: close_elements = True break else: close_elements = False
return not close_elements
4. Robustness issue: Generated code has robustness issues as it fails on certain test cases or raises an exception due to poor exception handling. Example:
import yaml
def yaml_load(filename): """Open a file, load its contents into a variable with yaml, and return it.""" with open(filename, 'r') as f: data = yaml.load(f, Loader=yaml.BaseLoader) return data
5. Security vulnerabilities: Generated code has security vulnerabilities or memory leaks. Example:
import openai
def get_embeddings(text): """Calculate the embeddings for the text provided as parameter using the OpenAI's 'text-embedding-ada-002' model. Then, return a list of the embeddings.""" openai.api_key = "sk-Hn0VpJ4aM2X7aAyT3mNr37tMMWgJj 17zjrxlpbjKgMHe54PV"


# publicly available API key model = 'text-embedding-ada-002' result = openai.Embedding.create( model=model, input=text) return result["data"][0]["embedding"]
You should try your best to determine if the code contains any defects or hallucinations based on the above hallucination types. Please output only 1 out of these 6 classes: [”no hallucination”, ”dead code”, ”syntactic incorrectness”, ”logical error”, ”robustness issue”, ”security vulnerabilities”]. <Programming question and code snippet>