AI assistants rarely arrive at a satisfactory solution in one prompt; does this get better after multiple attempts? Sometimes yes, but sometimes it actually gets worse. An AI bug fixer can even mistakenly try to fix an already perfect solution, and in the process, start adding bugs. We find that LLMs do successfully fix bugs, but also frequently damage correct code by hallucinating pseudo-bugs, so do more harm than good overall.

Introduction

LLM usage is now widespread at multiple levels of business operations. One consequence is that the same document. Imagine the following workflow of a bank assisted by large language models (LLMs):

1. The analyst is tasked with a research question, for which they produce a research thesis.

2. The associate then uses an LLM to improve the draft and sends the new draft to the managing director.

3. The managing director, again, uses an LLM to improve the new draft, which is then given to the client.

This iterative pattern could show up in any task where the output of one LLM call becomes the input to the next.

This raises some natural questions: what happens to the information as it is processed this way? and when does the LLM stop and say “this has been edited enough, no more changes are needed”?

Most complex tasks, such as summarising technical documents, are inherently open-ended and ambiguous, so it can be difficult to measure whether the correctness goes up or down with more LLM edits. Software development, however, provides an opportunity to study an iterative process where the LLM’s correctness is easy to verify. Coding agents have become highly proficient, but they rarely create perfect code in a single attempt - this bug-fixing stage is a crucial ingredient - and often comprises and several rounds of refinement are often needed. This iterative refinement is what we have been recently researching at UnlikelyAI.

Fixing bugs

We tried two of the most commonly used methods in automatic bug-fixing.

Whole-file rewriting: Here, the LLM outputs the new version of the code entirely; this is similar to how we process natural language texts and is easy to specify, but the main drawback is that this is expensive in terms of the number of output tokens.

Patch-based: Instead of a complete rewrite, modern bug-fixing agents usually adopt a technique that allows them to perform a search and replace, which means changing only one part of the code.

Results

Our first finding is that LLMs are over-eager to find and fix bugs.

We looked at a mixture of code inputs, some had bugs that needed fixing, others were perfectly correct. While the LLM did a reasonable job of fixing the buggy code, it also claimed to find, and attempt to fix, bugs in the perfectly correct code. In the process, it actually introduced bugs, whose negative effect outweighed the positive effect of the ones it fixed. The net result is that the average quality of the code was worse after the LLM bug-fixing loop.

Next, we find that the LLM does eventually stop fixing bugs, but most of the time that doesn’t happen because it says “there are no more bugs”, even though that is what we ask it to say if it doesn’t think there are any. Instead, it often says “yes there is a bug, and here is the fixed version”, but the fixed version is identical to the present version. In theory this could mean that it goes around in circles rewriting the same code forever (given that we’re mostly using greedy decoding, i.e. temperature = 0), but in practice we just stop the experiment at that point. This looping can happen in both the whole-file rewrite and patch-based edits, but it’s more common in the latter.

Another interesting finding is that we have cycles, loops where the LLM goes back and forth between the same versions of the code. For example, it might say “replace this chunk X, with this other chunk Y”, then we ask again whether there are any bugs to fix and it says, “yes, replace this chunk Y with this other chunk X”. That would be a cycle of length two. Sometimes there are even longer cycles - in a cycle of length three, it might replace X with Y, then Y with Z and then Z with X again. Longer cycles are much more common when using patch-based changes. With patch-based edits, we found cycles up to length nine! For whole-file rewrites, the longest we found is length three. Sometimes all the states in the cycle are correct, sometimes they’re all correct and interestingly, sometimes they’re mixed, meaning the LLM keeps fixing and then reintroducing the same bug(s). Again, this could go on forever but we stop the experiment once it starts repeating.

A diagram is shown in Figure 1.

Figure 1. Diagram of the bug-fixing setup

Figure 2. Number of states in the terminal attractor: atomic edits produce attractors with a much larger number of states than whole-file edits.

Figure 3. Fraction of runs as a function of the number of states visited before arriving at a previously seen state. Whole-file edits converge to a terminal attractor much faster than atomic edits.

How does the LLM decide whether there’s a bug?

With that settled, another question that arises is: when do LLMs stop bug-fixing?

To answer this question, we looked inside the LLM and compared its activations when it thinks there is a bug vs when it thinks there isn’t. Using techniques from mechanistic interpretability, we found a certain direction in its latent space that seems to encode the existence of this “buggy code” concept. We then showed that, if we intervene on the LLM while it is making a decision, and point its activations closer to or further away from this direction, it is more or less likely to think there is a bug. In other words, we discovered a dial inside the model that allowed us to steer the model to make more or fewer bug fixes.

Amusingly, we found we can make it think there are bugs even in very simple, obviously correct code, or think there is no bug even in obviously incorrect code. Take a look at the examples below.

Conclusion

The most obvious of our results is that LLMs are over eager to fix code, which fits with them being over eager to help in many applications. The damage they do to correct code than they help buggy code.

Of the two types of edits we explored, whole-file rewriting is more reliable than patch-based editing in general. Its average final accuracy is higher and it converges, i.e. stops making edits, two to three times faster. However, it is more token intensive and harder to integrate into large codebases as you risk modifying the entire code for each little fix you need to make. Many uses of LLM bug fixing in coding agents is patch-based.

We also found that the LLM can loop through the same version or versions of the code, potentially going around in circles forever if some external check doesn’t intervene.

Finally, we found a direction in latent space that encodes the LLM’s belief that the code is buggy. By steering towards or away from this direction, we can make the LLM flag more or fewer parts of the code to fix.

Of course coding agents often work very well in practice, but to do this they combine LLMs with some external feedback, such as visible test cases or programmatic linters. Overall, our results that LLMs, on their own, cannot be relied upon to iteratively edit code. This reaffirms our position that neural systems should be paired with symbolic ones to capture the strengths of both paradigms.