People talk about a language model's "memory" like it's some mysterious, ever-growing pool of understanding. It isn't. A context window is a fixed, countable, spendable resource — closer to a budget than to a brain — and understanding it that way changes how you should actually work with one.
What it actually is
A context window is the total amount of text — measured in "tokens," roughly word-fragments — that a model can look at in a single request. Every message you send, every file it reads, every tool result it gets back, every word it writes in response, all of it counts against that same fixed number. There's no separate long-term storage being filled up behind the scenes. When the conversation gets too long for the window, older content has to be dropped or summarized to make room, whether or not that's the part you actually needed it to remember.
That's the whole mechanism. It's not learning as you talk to it, in the way a person would. It's re-reading the entire relevant conversation, every single turn, and the window is simply how much of that re-reading it can afford to do at once.
Why "budget" is the right mental model
A budget forces tradeoffs, and so does a context window:
- Every token spent on irrelevant content is a token not available for something that matters. Pasting an entire log file when three relevant lines would do isn't free — it's spending window space that could've gone toward the actual problem.
- Old information doesn't get "forgotten" gently — it gets compressed or dropped outright once the budget is exceeded, and that process doesn't necessarily preserve the details you'd have picked yourself.
- Bigger isn't automatically better. A larger context window means a bigger budget, not smarter spending. Dumping in everything you have because there's room for it still crowds out the model's ability to weigh what's actually relevant — the same way having a bigger bank account doesn't make you a better budgeter.
What this means practically
If you're working with an AI assistant on anything nontrivial, a few habits follow directly from taking the budget framing seriously:
- Front-load what actually matters. Don't assume it'll "remember" something you mentioned in passing forty exchanges ago as clearly as something you said just now — restate load-bearing facts if a conversation is running long.
- Offload durable knowledge outside the conversation. This is exactly the problem a persistent memory system (see the earlier post on beads and mnemoria) solves — decisions and hard-won discoveries that need to survive past any single conversation shouldn't be relying on staying inside one finite window in the first place.
- Prefer specific over exhaustive. A well-chosen excerpt beats a full document dump, almost every time — not because the model can't technically handle the volume, but because volume itself has a cost, paid in how much attention is left for the parts that actually matter.
None of this requires understanding the underlying architecture. It just requires treating "how much can I usefully put in front of this thing at once" as a real, finite constraint you're actively managing — because you are, whether you're thinking about it or not.