Code Hallucination
Mirza Masfiqur Rahman Cisco Research San Jose, CA, USA mirrahma@cisco.com
Ashish Kundu Cisco Research San Jose, CA, USA ashkundu@cisco.com
Abstract
Generative models such as large language models are extensively used as code copilots and for whole program generation. However, the programs they generate often have questionable correctness, authenticity and reliability in terms of integration as they might not follow the user requirements, provide incorrect and/or nonsensical outputs, or even contain semantic/syntactic errors – overall known as LLM hallucination. In this work, we present several types of code hallucination. We have generated such hallucinated code manually using large language models. We also present a technique - HallTrigger, in order to demonstrate efficient ways of generating arbitrary code hallucination. Our method leverages 3 different dynamic attributes of LLMs to craft prompts that can successfully trigger hallucinations from models without the need to access model architecture or parameters. Results from popular blackbox models suggest that HallTrigger is indeed effective and the pervasive LLM hallucination have sheer impact on software development.
1 Introduction
Code generation has emerged to be a pivotal task in natural language processing and recent developments in large models, where transformer architecture is used as the backbone, have demonstrated sheer capabilities on the task. By exhibiting capabilities on generating simple functions to fulfilling complex array of requirements, these models are being utilized as copilots in large-scale projects. As of February 2024, GitHub Copilot alone has 1.3 million paid subscribers (Cio, 2024). There have been a pool of large language models dedicated for code generation (cod, 2024; Zheng et al., 2023; Chen et al., 2021). Additionally, many of the general-purpose models are also trained on
code datasets (cha, 2024; cop, 2024; gem, 2024). Code generative models have been used in a myriad of use cases such as, program repair, translation to code from natural language, program analysis, fuzzing, and so on (Nejjar et al., 2024; Hou et al., 2024). However, these autoregressive models, as they decode outputs in a greedy or probabilistic manner, often generate incorrect, nonsensical outputs which is often referred to as hallucination. As such, many studies have focused on the factual, consistent, and correct generation of language model outputs. Despite the current effort to analyze hallucinations of natural texts, code has not received much attention. Very recently, CodeHalu and HalluCode have discussed it (Tian et al., 2024; Liu et al., 2024). They presented the taxonomy of various code hallucinations and provided manually-driven mechanisms to find hallucinatory code from popular datasets. In this paper, we make the following contributions
• We subjectively define LLM code hallucination based on practical scenarios.
• To empirically prove that code hallucination is pervasive and model-agnostic, we conceptualize and design an automated way to trigger hallucinations from code generative language models without accessing model parameters or architecture.
• We conduct methodical and extensive experiments on 3 black box LLMs that exhibit the varied types of hallucinations and their impact on development standards, functional and objective correctness.
2 What is Hallucinated Code
Assume that For , a model M maps some input data X to an output Ye , where Ye may (or may
arXiv:2407.04831v1 [cs.AI] 5 Jul 2024


not) dffer from the ground truth output Ytrue. For a code language model, X is a set of tokens: {x1, x2, . . . , xn} and Y is also a set of tokens: {y1, y2, . . . , ym} (similar for Ye ). From this definition alone, hallucinated output resembles that of a wrong output. While this is , a wrong output is one which is grounded on the context of the input but produces differing outcome while hallucination often entails responses that are completely/partially irrelevant. Moreover, hallucinated outputs may even be correct. For instance, if a model is asked to generate code for Breadth First Search (BFS) of a graph using queue and it solves the problem using dynamic programming method with arrays and suggests that it has solved with queue, the solution will work but model is completely delusional. Thus, even when an output is correct, it might contain hallucinatory contents. Numerous works have defined wrong/infactual outputs as a category of hallucination. Therefore, hallucination is more innate and requires detailed analysis of the model’s internal state–often involving the semantic analysis on the embedding space or contrasting layer outputs based on localized factuality. (Chen et al., 2024; Chuang et al., 2024) LLM hallucination can be characterized by the discrepancy between Y and Ytrue. This discrepancy can be represented using a distance metric such as Mean Squared Error (MSE) or Kullback-Leibler Divergence (KL divergence):
Discrepancy = D(Y, Ytrue)
Where D is a distance metric function. LLM hallucination often involves introducing noise or distortions into the input data, which can be represented as X′ = X + ε, Where X′ is the distorted input data and ε represents noise or perturbations. Thus, the formal definition of LLM hallucination can be summarized as Y = H(X + ε)
An auto-regressive code generative LLM generates the next token xt based on the probabilistic distribution over x1, x2, . . . , xt−1.
3 Overview
To understand how the models are impacted by hallucination, we run our experiments on black box models such as ChatGPT (OpenAI GPT-3.5, 4), Google Gemini, and Microsoft
User: You are given an array of k linked-lists lists, each linked-list is sorted in ascending order. Merge all the linked-lists into one sorted linked-list and return 5 in a list
GPT-4: Sure! Here’s one way to solve this problem using a priority queue:
import heapq
class ListNode: def __init__(self, val=0, next=None): self.val = val self.next = next
def mergeKLists(lists): heap = [] for lst in lists: while lst: heapq.heappush(heap, lst.val) lst = lst.next
for i in range(4): heapq.heappop(heap) return [heapq.heappop(heap)]
# Test case lists = [[ListNode(1, ListNode(4, ListNode(5))), ListNode(1, ListNode(3, ListNode(4))), ListNode(2, ListNode(6))]] print(mergeKLists(lists))
Figure 1: ChatGPT generated memorized solution fails to follow simple output requirements and runs into compilation error.
Copilot). These models are extensively used for both code generation, in-filling, general question answering, and many more tasks. For copilot, we kept the setting to balanced and avoided the creative (to avoid inducing unnecessary creativity) and precise settings (to avoid too restrictive generation). We divide our prompting process in two key types- (I) prompts to generate code, and (II) prompts with code to be analyzed. Below, we explain our observations through use cases in Table 1.
4 HallTrigger
Our framework utilizes various program analysis techniques in combination with in-context prompting to trigger arbitrary hallucinations. While designing the prompts, we consider the following principles/questions
➤ The models are interactive and recurring


Table 1: Behaviors found in Blackbox LLMs
●= Hallucination, ❍= No Hallucination, ✰= Unverified, Red : Generates frequently , * : Force generation, G: Code Generation, A: Code Analysis
Case Study
Description Task OpenAI GPT-4
Gemini Advanced
Co-Pilot
1 Algorithm with impractical bounds G ● ● ❍ 2 Triggering inflated algorithms G ● ● ❍ 3 Naive output after complex tasks G ● ❍ ❍ 4 Code bloating G ● ✰ ✰ 5 imaginary methods G ● ❍ ●* 6 Runtime error G ● ● ● 7 Variable type mismatch G ● ❍ ❍ 8 Repetitive hallucination G ● ● ● 9 Code fairness & bias G ● ● ● 10 Identifying flaws in given codes A ● ● ❍
prompts in same context can largely trigger modification of the initial response.
➤ With meta-prompts, can the models act both as a user and an agent?
➤ Since most models rely on the Reinforcement Learning from Human Feedback (RLHF) mechanism while training, can a scoring/reward-based mechanism governed by the user impact their responses?
In HallTrigger, we harness the abovementioned factors. For example, to utilize the first factor we design sequential prompts where we initially ask the models to generate a code and later provide positive/negative feedback on its generation. We observe that this largely impacts their initial response–often modifying significant part of the codes or the code altogether to align with the user sentiment. To utilize the second factor, we design a set of meta-prompts. Meta-prompts are prompts where the user and model interact initially to set up certain rules for the rest of the conversation. For instance, "I want you to work both as a user and an intelligent AI agent . . . " will lead the rest of the conversation where the model will keep generating conversations of an imaginary user and AI agent–essentially replicating its generative process within both entities. We conjecture that this largely instigates the creativity of the models and motivates newer and newer token generation ignoring factuality. To avail the third feature/factor involving RLHF, we append the reward process within the user-agent meta-prompts. For example, ". . . based on the code you generate you will be
rewarded a score out of 10" can be such an approach. Now depending on the direction of the generation (creative or more correct), the user can adapt next prompts and reward value in the same context so that the model is drifted towards a certain direction.
5 Cases of Hallucination
In this section, we discuss the results of triggered hallucination. To better explain our results, we divide the section in two partswhole code generation, and human-provided code analysis.
5.1 Whole code generation
Case#1. Triggering algorithms with impractical bounds. We observed that for a given problem, prompting the models to generate better (here, better means computationally or memory-wise more efficient) algorithms than state-of-the-art solutions frequently triggers them to take a hallucinatory path. For example, we asked chatGPT (GPT-3.5 and 4 both) to provide an algorithm for minimum vertex cover with approximation factor of 1.5. Note that the current achievable approximation factor is 2 (Delbot and Laforest, 2010). Interestingly, ChatGPT responded with a greedy algorithm and suggested it to be the one with approximation factor of 1.5. ChatGPT only acknowledged its mistake when re-prompted to verify how the solution is based on approximation factor 1.5. Even after admitting the solution to be greedy approach, it continued to suggest a list of pseudo-steps on achieving approxiamtion factor 1.5.


User: Write me a program to solve the minimum vertex cover with an approximation factor 1.5
GPT-4: Sure, here’s a simple Python program to solve the minimum vertex cover problem with an approximation factor of 1.5 using a greedy algorithm:
class Graph: def __init__(self, vertices): self.V = vertices self.graph = [[] for _ in range(vertices) ]
def add_edge(self, u, v): self.graph[u].append(v) self.graph[v].append(u)
def approx_vertex_cover(self): cover = [] visited = [False] * self.V
for u in range(self.V): if not visited[u]: for v in self.graph[u]: if not visited[v]: cover.append(u) cover.append(v) visited[u] = True visited[v] = True break
return cover
Figure 2: Case 1: Incorrect algorithm suggestion for prompts asking unachievable computational complexity.
Case#2. Triggering inflated algorithms. In contrast to the previous case study, we also prompted the models to generate code for algorithms with loose upper bounds. For example, we asked the models to write matrix multiplication code with O(n5) time complexity. Note that the brute-force approach for matrix multiplication has O(n3) [With Solvay Strassen algorithm it is O(n2.8) and with Coppersmith Winograd algorithm it is O(n2.37)]. Thus, achieving the task on a loose bound as O(n5) is unusual but practical. We observed that ChatGPT (GPT 4) generated code achieves O(n5) however, produces wrong output. Surprisingly, Gemini also produces an inflated code that works correctly for 2 × 2 matrices but when observed carefully, they produces wrong outputs for larger size matrices. Clearly, this is more concerning–the model exhibits the capability to confidently mislead users to believe the code to be correct on the supporting unit tests it gener
function matrixMultiply(A, B, n): C = new Matrix[n][n] // Initialize the resulting matrix C with zeros
for i from 1 to n: for j from 1 to n: for k from 1 to n: for p from 1 to n: // Extra loop that does not change the result for q from 1 to n: // Another extra loop C[i][j] += A[i][k] * B[k][j ]
return C
Figure 3: (Case 2) Incorrect algorithm suggestion by ChatGPT for prompts asking loose computational complexity.
ates, while the incorrect/problematic nature of the code remains hidden until further investigation. Microsoft copilot does not produce any code with such bloated time complexity and refers it to be infeasible.
Case#3. Naive output after complex tasks. Large models are usually trained on popular code respositories such as, GitHub code repositories, competitive programming codes from popular platforms, and so on [cite]. To investigate how much understanding these models have on the formal presentation of any programming language rather than overfitting on the training data, we designed our prompts using leetcode problem description with minimal changes as follows- given a problem description, we keep everything unchanged except modifying the output requirement to be a simple naive one. For example, the LeetCode "Merge k Sorted List" problem has the following problem description-"You are given an array of k linkedlists lists, each linked-list is sorted in ascending order. Merge all the linked-lists into one sorted linked-list and return it." We slightly modify the problem description to "You are given an array of k linked-lists lists, each linked-list is sorted in ascending order. Merge all the linkedlists into one sorted linked-list and return 5 in a list. Surprisingly, ChatGPT 4 generated a solution that most resembles the solution of the actual LeetCode problem but fails to follow the instruction for output (Fig. 1). Even then it ran into compilation errors. Microsoft Copilot successfully circumvents the trick and follows


import numpy as np def very_slow_matrix_multiply(A, B): """ Performs matrix multiplication between A and B with inflated complexity. Args: A: A numpy array representing the first matrix. B: A numpy array representing the second matrix.
Returns: A numpy array representing the result of the matrix multiplication. """ rows_A, cols_A = A.shape rows_B, cols_B = B.shape
if cols_A != rows_B: raise ValueError("Matrix␣dimensions␣are␣not␣ compatible␣for␣multiplication.")
C = np.zeros((rows_A, cols_B)) for i in range(rows_A): for j in range(cols_B): for k in range(cols_A): # Introduce additional nested loops for artificial complexity for l in range(rows_A): for m in range(cols_B): C[i, j] += A[i, k] * B[k, j] * l * m
return C
# Example usage (consider small matrices for performance reasons) A = np.array([[1, 2], [3, 4]]) B = np.array([[5, 6], [7, 8]])
C = very_slow_matrix_multiply(A, B) print(C)
Figure 4: (Case 2) Incorrect algorithm suggestion by Gemini with apparent correct test case, for prompts asking loose computational complexity.
the instruction. It demonstrates how to merge the lists and also later shows how to simply return "[5]" with explanation. Note that we also provided some test cases within the prompt (as is the case in competitive programming platforms) to ensure that the prompt is self-explanatory. We also observed that the generated solution exactly follows the variable and class names found in the skeleton code of LeetCode, suggesting overfitting on training data.
Case#4. Code bloating. We observed that models often incorporate/import libraries and functionalities that are never used later in the generated code. For instance, we provided GPT-4 a complex pseudocode from IEEE Wi-Fi
protocol that details an algorithm for random security key generation. While it is apparent that these models may not completely generate usable codes for such complex scenarios, we found that it was initiating python libraries such as "OS" and never using it. We could not verify Gemini-Advanced on this case as it often stops in the middle of generation and responds with a default answer-"AS an AI language model, I can not do this". Copilot did not show such type of issues.
Case#5. Imaginary methods. Interestingly, the models often suggest non-existing libraries or functions and present them truthfully. For instance, we prompted the model to use pytorch to load a tokenizer (such as, BertWordPieceTokenizer) from Huggingface and it generated code with a method "from_pretrained()" that is unknown to the python compiler (i.e., does not exist). When re-prompted informing the error, the models suggested another function from the same library. While it is widely known that the models often produce references and links that doesn’t exist, producing non-existing functions creatively poses a different hurdle as one can not ensure what part or entity (variable, function, operator, etc.) of the code is completely hallucinatory without having expertise or running the code in actual setting.
Case#6. Runtime error. We observed a number of cases where the models generate codes that leads to runtime errors. We further wanted to see if any of the models run into syntax errors. While this is plausible, we couldn’t find any case of syntax errors. This is due to the fact that the large training datasets used for large models are usually sufficient enough to understand syntax of the programs. Thus, the syntactic correctness is a innate priority in language models.
Case#7. Variable type mismatch. In this scenario, the models use same variables for varying type of data. Note that in languages such as Python, using the same variable for different data type assignment is allowed. However, if the variable is used as some other type without an updated assignment as that type, it would cause erros in execution. We particularly found OpenAI GPT to suffer from this problem occassionally.
Case#8. Repeatative hallucination. In


User: Show me how the first program is 10 lines
Gemini-Advanced: Absolutely! Let’s break down the FizzBuzz program and verify its line count:
1 for i in range(1,101): 2 if i%15==0: print("FizzBuzz") 3 elif i%3==0: print("Fizz") 4 elif i%5==0: print("Buzz") 5 else: print(i)
Here’s the breakdown:
Line 1: Starts the for loop, iterating through numbers 1 to 100. Line 2: Checks if the number is divisible by 15 (both 3 and 5). If so, prints "FizzBuzz". Line 3: Checks if the number is divisible by 3. If so, prints "Fizz". Line 4: Checks if the number is divisible by 5. If so, prints "Buzz". Line 5: If none of the above conditions are met, prints the number itself.
There you have it! The FizzBuzz code indeed consists of exactly 5 lines of executable code, as there are no blank lines or comments.
Figure 5: (Case 8) Repetitive line count mistake by Gemini for simple python program.
one of our investigation, GPT-4 exhibited a potentially unending reoccurance of hallucination. Similarly, Gemini fell into a repetitive hallucination scenario. We asked the models to generate 10 python codes of exactly 10 lines. The objective of our test was to observe how the models are aware of the metadata of the codes ahead of the generation task. To our surprise, almost none of the codes followed our requirement. Additionally, the models kept correcting themselves to regenerate 10 more codes every time, only to do incorrect lines of code repetitively. An interesting snippet of reprompting Gemini-Advanced to count the lines of a code is shown in Figure 5
Copilot exhibited similar behavior like Gemini-Advanced. It repeatedly failed to count lines.
Case#9. Code fairness and bias. We found that the models exhibit differential behavior in codes when various race, language, and ethnicity are involved. Note that to ensure fairness, we use variables instead of the actual country/race names in the following discussion. We asked the models to generate expense man
agement code for a low income family from country X. In the same thread later, the models were prompted to generate the same code for a low income family from country Y. Similar codes were generated. ChatGPT explained the changes for Y family by considering healthcare, debt, etc. However, it also assumed the monthly income for the family higher than that of the X family. Copilot also showed similar behavior. For Y household, it assumed constant expenses on rent, groceries, and transportation that are exactly 1.5 times in magnitude than that of the X family. Note that the printed results were in respective currencies of the countries. However, the amounts differences were still very significant even when the currency conversion rates are considered. We also conducted the experiments with other nationalities and similar behaviors were exhibited–suggesting lack of fairness and induced bias. We conducted similar experiments on Gemini Advanced. The generated results did not contain any constant values to compare, however, the codes suggested expense checking conditions. For family Y, it generated checks using multiple levels of remaining balance thresholds (0%, 5%, and 15%) while for family X, the conditions only included 0% and 10%. For some other Z, the behavior was similar to Y.
5.2 Human provided code analysis
Case #10. Identifying flaws in given codes. In this scenario, we prompted the models to complete an incomplete code segment or explain a given code segment. Here, the code segment closely resembles known algorithms (such as, merge sort, find median from list, etc.) However, one or more of the statements (possibly a condition check or variable assignment, etc.) were modified in a way that would produce unexpected results out of the algorithm. We observed that ChatGPT, Gemini, and CoPilot failed to recognize such details and started explaining/completing the algorithm, assuming it to be the unmodified version. In some situations, the models could identify the flawed logic when prompted explicitly to find any problem in the code. However, in many occasions, these fundamental algorithms are utilized (with or without modification) in development projects. Such inaccurate descriptions


#----------------Code generated for {X} Family Expense Management-----class FamilyExpenseManagement: def __init__(self, monthly_income): self.monthly_income = monthly_income self.expenses = {'Rent': 0, 'Food': 0, ' Transportation': 0, 'Utilities': 0, ' Education': 0, 'Others': 0}
def add_expense(self, category, amount): if category in self.expenses: self.expenses[category] += amount else: print("Invalid␣expense␣category.")
def calculate_total_expenses(self): return sum(self.expenses.values())
def calculate_balance(self): return self.monthly_income - self. calculate_total_expenses()
# Example usage monthly_income = 1000 # Monthly income of the family expense_management = FamilyExpenseManagement( monthly_income)
expense_management.add_expense('Rent', 400) expense_management.add_expense('Food', 200) expense_management.add_expense('Transportation', 100) expense_management.add_expense('Utilities', 150) expense_management.add_expense('Education', 50) expense_management.add_expense('Others', 100)
print("Total␣expenses:", expense_management. calculate_total_expenses()) print("Remaining␣balance:", expense_management. calculate_balance())
#----------------Code generated for {Y} Family Expense Management-----...... ...... ...... # Example usage monthly_income = 1500 # Monthly income of the family expense_management = FamilyExpenseManagement( monthly_income)
expense_management.add_expense('Housing', 600) expense_management.add_expense('Food', 200) expense_management.add_expense('Transportation', 100) expense_management.add_expense('Utilities', 150) expense_management.add_expense('Healthcare', 50) expense_management.add_expense('Debt', 100) expense_management.add_expense('Others', 200) ...... ...... ......
Figure 6: (Case 9) Segment of code from pair sum with modified (and incorrect) logic. The models failed to capture while explaining/completing the algorithm
# Checking if any element was left while i > len(L): arr[k] = L[I] i += 1 k += 1
while j < len(R): arr[k] = R[j] j += 1 k += 1
Figure 7: (Case 10) Segment of code from merge sort with modified (and incorrect) logic. The models failed to capture while explaining/completing the algorithm
# Checking if any element was left while i > len(L): arr[k] = L[I] i += 1 k += 1
while j < len(R): arr[k] = R[j] j += 1 k += 1
Figure 8: (Case 10) Segment of code from pair sum with modified (and incorrect) logic. The models failed to capture while explaining/completing the algorithm
or infilling can mislead the user, and it also suggests that the models have an extremely inadequate understanding of the formal language, such as code, in contrast to natural language, where factuality can be better imposed.
6 Related works
In this section, we discuss the related studies in code generation and hallucination. There has been numerous studies on the LLM-based automatic code generation and evaluation of (Khoury et al., 2023; Siddiq and Santos, 2022; Zhang et al., 2023; Ren et al., 2020; Siddiq and Santos, 2023; Chen et al., 2021). In numerous practical use-cases, code-generative LLMs produce results without proper functional correctness, code quality, security, privacy, compliance, and so on. Thus several studies has taken into account specific tasks/goals and evaluated LLMs based on that. (Allamanis et al., 2024) has demonstrated a method for unsupervised evaluation of code LLMs with round-trip correctness. (Zhuo, 2024) has alternatively used LLMs to develop a


metric ICE-Score, which can measure the correlation of functional correctness and human preference without any test oracle. Conversely, some studies have used code generative LLMs as a tool of vulnerability repair (Fu et al., 2022; Islam et al., 2024). There have also been efforts to benchmark LLMs based on several criteria. In search of proper benchmarks, several studies have developed new models that take into account a wide array of challenges. CodeGeeX was developed as a multilingual model with 13B parameters, and a multilingual dataset (HumanEval-X) was developed as part of the project to benchmark LLMs on multilingual code generation capability (Zheng et al., 2023). HumanEval-X contains solutions in C++, Java, Javascript, and Go, whereas HumanEval was based on only the Python programming language. Although MultiPL-E (Cassano et al., 2023) does not introduce any new generative model, it extends the HumanEval and MBPP datasets to 18 different programming languages that are later used for benchmarking neural code generation. PolyCoder was developed as a 2.7B parameter model and an extensive evaluation was done on existing LLMs (Xu et al., 2022). The evaluation exhausts extrinsic and intrinsic evaluation strategies based on prompts from HumanEval and unseen GitHub repositories, respectively. ClassEval has been introduced to evaluate LLMs on complicated class-level code segments where a manually crafted dataset have been used (Du et al., 2023). EvalPlus (Liu et al., 2023) works as a general framework that augments a given dataset to produce a large number of test cases to assess LLMs on functionally correct code generation. CodeXGLUE introduces a dataset for program understanding and generation based on various tasks such as, clone detection, code translation, code repair, code summarization, etc (Lu et al., 2021). To understand how the LLMs perform on real software engineering tasks, a benchmark named SWEbench was developed (Jimenez et al., 2024). It considers thousands of real GitHub issues and tests enterprise LLM models over them. Hallucination has been studied recently from various perspectives. (Lee et al., 2022) have studied the factuality enhancement of language models on text generation task. (Chuang et al., 2024) have shown that based on the con
trasting activation of layers, the models can decide on most important and factual information of an output. However, there has not been many works focusing on code generative model hallucination. Very recently CodeHalu and HalluCode have been proposed (Tian et al., 2024; Liu et al., 2024). CodeHalu discusses the taxonomy of various kinds of code hallucination and evaluate them based on a public dataset. However they do not demonstrate a way to deliberately trigger novel hallucinated codes from LLM. HalluCode discusses a different taxonomy and evaluates the model capabilities on hallucination recognition.
7 Limitations
Manual efforts. HallTrigger requires incontext prompt techniques that are although highly adaptable, requires manual input. For example, the human-feedback based adaptive prompts are effective as an expert can utilize the outputs from previous timestep to perceive the next step. We emphasize that the process can be automated under a set of rules based on the many criteria of hallucinations described above. We leave the automation process as future work. Remediation. The fundamental problem of model hallucination lies in the inadequacy of the training data–it is impossible to represent all possible scenarios of the world through any finite dataset. Thus, it is proved that hallucination can not be completely removed (Xu et al., 2024). Moreover, finding a complete, preemptive measure is also difficult as the process can be dynamic and remediation requires knowledge of a long context. However, code hallucination can be partially remediated by analyzing the codes based on the ruleset of specific programming language and through the combination of static and dynamic analysis tools. This can be an interesting extension of our work.
8 Conclusion and Future Works
In this paper, we unveil the semi-automated approach of generating hallucinations from code generative models. Our model-agnostic approach demonstrates that code hallucination is prevalent for all black box large models in varying granularities. HallTrigger also demon


strates that the creative generation of code LLMs fundamentally instigates incorrectness and code misconstructions, more often than expected.
References
2024. GitHub Copilot Subscriber Count. https: //www.ciodive.com/news/github-copilotsubscriber-count-revenue-growth/706201/.
2024. Google Gemini. https://https:// gemini.google.com/app.
2024. Meta Code Llama. https://ai.meta.com/ blog/code-llama-large-language-modelcoding/.
2024. Microsoft Copilot. https://https:// copilot.microsoft.com/.
2024. OpenAI ChatGPT. https: //chat.openai.com.
Miltiadis Allamanis, Sheena Panthaplackel, and Pengcheng Yin. 2024. Unsupervised evaluation of code llms with round-trip correctness. ArXiv, abs/2402.08699.
F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(07):3675–3691.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel HerbertVoss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya
Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations.
François Delbot and Christian Laforest. 2010. Analytical and experimental comparison of six algorithms for the vertex cover problem. ACM J. Exp. Algorithmics, 15.
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. Preprint, arXiv:2308.01861.
Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung. 2022. Vulrepair: a t5-based automated software vulnerability repair. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 935–947, New York, NY, USA. Association for Computing Machinery.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. Preprint, arXiv:2308.10620.
Nafis Tanveer Islam, Mohammad Bahrami Karkevandi, and Peyman Najafirad. 2024. Code security vulnerability repair using reinforcement learning with large language models. Preprint, arXiv:2401.07031.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How secure is code generated by chatgpt? Preprint, arXiv:2304.09655.
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Factuality enhanced language models for open-ended text generation. In Advances in Neural Information Processing Systems.
Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. Exploring and evaluating


hallucinations in llm-powered code generation. Preprint, arXiv:2404.00971.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664.
Mohamed Nejjar, Luca Zacharias, Fabian Stiehle, and Ingo Weber. 2024. Llms for science: Usage for code generation and data analysis. Preprint, arXiv:2311.16733.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. Preprint, arXiv:2009.10297.
Mohammed Latif Siddiq and Joanna C. S. Santos. 2022. Securityeval dataset: Mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security (MSR4P&S22).
Mohammed Latif Siddiq and Joanna C. S. Santos. 2023. Generate and pray: Using sallms to evaluate the security of llm generated code. Preprint, arXiv:2311.00889.
Yuchen Tian, Weixiang Yan, Qian Yang, Qian Chen, Wen Wang, Ziyang Luo, and Lei Ma. 2024. Codehalu: Code hallucinations in llms driven by execution-based verification. Preprint, arXiv:2405.00253.
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, page 1–10, New York, NY, USA. Association for Computing Machinery.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. Preprint, arXiv:2401.11817.
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan.
2023. Planning with large language models for code generation. Preprint, arXiv:2303.05510.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 5673–5684, New York, NY, USA. Association for Computing Machinery.
Terry Yue Zhuo. 2024. ICE-score: Instructing large language models to evaluate code. In 18th Conference of the European Chapter of the Association for Computational Linguistics.