The Context Window Is Not a Memory: What Long-Context Models Actually Hold Onto
Models can now read a novel in one go. That doesn't mean they remember it the way you do. Here's what a context window really is, and how to work around its blind spots.
Research Editor
Imogen Achterberg, a paralegal at a small employment-law firm, did what many of us now do without thinking. She pasted roughly 600 pages of deposition transcripts into a chat model and asked when the company first learned about a safety complaint.
The answer came back in seconds, with a date and a page reference, and it was right. Encouraged, she asked a second question: which manager had approved the overtime schedule? She got another confident answer with another page reference. This one was wrong. The name came from a different, similar-sounding exchange, and the cited page said something else entirely.
"It read the whole thing," she said. "So why did it act like it had skimmed the middle?"
It's a good question, and the answer says a lot about what long-context models are and aren't.
A desk, not a diary
A model's context window is the amount of text it can consider at once. That includes your instructions, the documents you paste, the conversation so far and its own replies. Windows have grown enormously, from a few pages of text in early chat models to whole books or codebases today.
The best metaphor I know is a desk. Everything the model works with has to be on the desk, and it can glance at any paper there. But the desk is cleared at the end of the session, and nothing on it was ever filed into long-term memory. Training gave the model its general knowledge. The context window gives it the specific material for this job. They're different systems.
Context is measured in tokens, chunks of text that are often a word or part of one. As a loose rule of thumb, an English word averages a little more than one token, so a 100,000-token window holds something in the neighbourhood of 75,000 words.
The murky middle
A bigger desk doesn't mean the model reads every paper with equal care. Researchers have repeatedly documented a pattern sometimes called being "lost in the middle": models tend to make better use of information near the beginning and end of a long input than of information buried in the middle. Newer models have improved, but the tendency hasn't vanished, and its strength varies from model to model and task to task.
That fits Imogen's experience. Her first answer sat near the start of the transcripts. The second lived around page 340.
There's another wrinkle. A popular way to test long context is the "needle in a haystack": hide one odd sentence in a mountain of text and ask the model to find it. Modern models are very good at this. But finding one needle is not the same as connecting a dozen threads scattered across a document, such as noticing that a date on page 40 contradicts a remark on page 510. That second skill, synthesis across a long input, is harder and less reliably measured.
A long context window is a bigger desk, not a better memory. The model can see more papers, but nothing promises it will read them all with the same attention.
Retrieval versus reasoning
It helps to split long-document tasks into two kinds:
- Retrieval: "Find the clause about termination notice." One fact in one place. Long-context models are generally strong here.
- Synthesis: "Is the company's account of events consistent across all of these depositions?" Many facts in many places, compared against each other. Here quality drops off faster and errors are harder to spot.
Most real work is a mix of the two. The trouble is that a model's tone doesn't change between them. It sounds just as sure when synthesising across 600 pages as when quoting one line from page two.
What "memory" features really do
Many chat products now advertise memory. It's worth knowing what that usually means under the hood, because it isn't a bigger desk.
Typically, the product saves short notes about you or your past conversations, such as "prefers metric units" or "is writing a novel set in Lagos," and quietly puts the relevant ones back on the desk when a new chat starts. Another common technique, retrieval-augmented generation, searches a library of your documents and places only the most relevant excerpts into the context.
Both are useful, and both amount to an assistant who fetches papers for the desk. If the fetcher grabs the wrong excerpt, the model will reason beautifully about the wrong thing.
How to work with a long-context model
You don't need to understand attention mechanisms to get better results. You need a few habits.
| If you want… | Try this |
|---|---|
| A single fact from a long document | Ask for the answer plus a direct quote and its location |
| Consistency across many sources | Ask it to extract claims into a table first, then compare the table |
| Reliable instructions | Put instructions at the top and repeat the key question at the end |
| Fewer confident gaps | Ask: "What did you look for and not find?" |
| A clean slate | Start a new session rather than piling onto a long one |
Here's why each one helps:
- Quotes keep it honest. When a model has to reproduce the exact sentence, mismatches become visible. Imogen now asks for verbatim quotes and spot-checks two of them every time.
- Tables turn synthesis into retrieval. Pulling each deposition's key claims into rows breaks one hard task into many easy ones, and the comparison happens in a format you can read.
- Long conversations get crowded. Every reply takes up desk space. After an hour of back-and-forth, early instructions can lose their grip. A fresh session with a tidy summary often works better than a sprawling one.
- Absence is information. Asking what it couldn't find gives the model permission to say "I don't know," which it will otherwise rarely volunteer.
The honest summary
Long context is one of the genuinely useful advances of the past few years. Being able to hand a model an entire contract, codebase or book removes a huge amount of fiddly chopping and pasting. But a capability that sounds like "it read everything" invites us to trust it the way we'd trust a colleague who did.
Imogen still uses it daily. She just treats it like a fast, brilliant temp on their first day: excellent at finding things, and worth double-checking on anything that requires knowing where everything was.
"I stopped asking it whether it had read everything," she said. "Now I ask it to show me where it looked."
How this was made Human-written
Human-written. Reported and written by a person. AI not used for the text.
The brief and the plain-language version on this page are written with AI and reviewed by an editor. The story itself is the source of truth.
Ask about this story
Conversation2
Nandini Iyer
'A bigger desk, not a better memory' is going on a sticky note. I lost a full afternoon to an assistant inventing a table from the middle of a thesis before I read this.
Arjun Pillai
The advice to repeat key instructions at the end of a long prompt sounds too simple to work, but it fixed a summarising problem I'd been fighting for weeks.