An interesting observed behaviour of LLMs is that they sometimes place nearly all their attention onto a single meaningless token. Why this happens is not fully understood. New research uncovers a smaller, subtler variant that appears mid-generation, and it's another sign that LLMs decide what matters in ways that look nothing like human reasoning.

Inside the LLM black box

It is a strange world inside an LLM. These models that constitute most of the current excitement in AI take text as input and produce text as output, but what they do in between is not what you might expect.

The words that we feed into LLMs are converted to large arrays of numbers which are multiplied, added and divided with each other in lots of different ways to produce a single output, which is the next word to use. As this process is iterated, word after word is chosen to produce the text that AI is now famous for generating. Everything an LLM does comes down to how it chooses the next word, and this decision process is still partially mysterious. Most ways of bashing big arrays of numbers together would not be at all able to properly choose the next word, and so the generated text would be gibberish, but somehow during training the LLM finds a particular, finely-balanced way to do this that is able to generate not only fluent text, but very informative and useful text, and we don’t fully understand why it works.

Attention sinks explained

A paper just published from UnlikelyAI labs, which appeared in ICLR’s 2026 Workshop on Unifying Concept Representation Learning, has contributed to furthering our understanding. The paper concerns the distribution of what is called attention inside the LLM. When it chooses the next word, it, of course, bases this decision on the previous words, but some words are more important than others, and attention is the way this importance is modeled. The LLM is composed of a sequence of layers, and each layer first computes the amount of attention it should pay to each of the previous words, and then it draws information from each of them in that layer in proportion to their attention. Words with more attention are weighted more heavily, literally by just multiplying them by a bigger number before adding them to the others.

When LLMs attend to nothing

As well as improving the quality of the generated text, attention is one of the windows we have into the confusing inside of the model. If we want to know why it chose a given word, we can inspect the attention pattern to see which previous words it mostly drew on to make that decision. This can give surprising results. Sometimes, we find that the LLM is attending to strange words that shouldn’t be relevant to the given word. One particularly weird thing it does is attend to a dummy token that is prepended to every input just to mark where the sentence begins. This token has no real meaning at all, but sometimes the LLM places almost all its attention in all layers on it. This behaviour is called attention sinking. It’s not exactly clear why it happens, but the hypothesis is that it allows the LLM to effectively attend to nothing, whenever it may want to do that.

Meet the secondary sink

What our paper uncovers is that attention sinking occurs in more diverse ways than we previously thought. Sometimes during long generations it throws in a smaller sink, smaller in the sense that it doesn’t pull quite as much attention, and it only lasts for a few layers, unlike normal sinks which draw lots of attention in every layer. We call these secondary sinks. By tracing the flow of numbers through the model, we identify where secondary sinks are likely to happen, and also show that if you directly remove the sink, the model will put it right back on the next word.

A comma is never just a comma

Although the research community is starting to document attention sinks, they are still enigmatic, both the ones that everyone already knew about and the new ones we discovered. They reveal how different LLMs are from humans in how they use language. Both may produce similar answers to the same question, but in highly dissimilar ways. When we write a comma, for example, we just use it for its grammatical/stylistic function, but an LLM also sometimes uses it as an opportunity to control the attention pattern for the next few dozen words.

The anthropomorphism trap

This is a reminder of the more general caution we should have when comparing AI and human behaviour. Just because AI does something that seems human-like externally, does not mean its internal mechanism is anything like ours. A common failure mode in interpreting AI is the following: “most humans who can do A can also do B, this LLM can do A, therefore it can do B”. A person who performs well on Leetcode problems will be a good software engineer, a person who solves classic logical syllogisms flawlessly will be good at logical reasoning in general, a person who can solve university level maths problems will be able to count the number of words in a paragraph - not so with LLMs.

Different, not less

So while they are amazing things and are changing the world, LLMs are likely to continue to surprise us with what they can and cannot do. Whatever intelligence they might be said to have, it is fundamentally different to humans’. This means we should resist simplistic interpretations that they are or will be as smart as us, and we should be hesitant to trust them replacing the work we do, but it also means they are fascinating objects to study.