Large language models power the chat assistants that now draft emails, summarize documents, and answer questions. To use them well, it helps to understand two technical ideas that shape everything they do: tokens and context windows. Neither is complicated, and both explain behavior that otherwise looks mysterious, such as why a model suddenly forgets the start of a long conversation.
What a token is
A language model does not see letters or whole words the way you do. Before any processing happens, your text is broken into tokens, which are chunks that are often a word, part of a word, or a piece of punctuation. A common word like the might be a single token, while an unusual word might be split into several. As a rough guide, in English one token is about four characters, and one hundred tokens is roughly seventy five words.
Tokens matter for a practical reason: they are the unit that model providers measure and bill. When a service quotes a price per million tokens, it is charging for the chunks of text going in and coming out. Longer prompts and longer answers cost more simply because they contain more tokens.
What a context window is
The context window is the maximum number of tokens a model can consider at once. Think of it as the model's short-term working memory for a single request. Everything the model can use to produce its next answer must fit inside this window: your latest message, the earlier conversation, any documents you pasted, and the system instructions the developer added.
Context windows have grown dramatically, from a few thousand tokens in early models to hundreds of thousands or more in recent ones. A larger window means the model can take in a long report or an extended conversation without losing track. But the window is still finite, and that limit produces some of the most common frustrations people report.
Why models seem to forget
When a conversation grows longer than the context window, something has to give. Older messages get pushed out to make room for new ones, and once text falls outside the window the model can no longer see it. From your side it looks like the assistant forgot what you said earlier, but really that text is simply no longer in front of it.
This is also why a model has no memory between separate chats unless the product deliberately stores and re-feeds information. Each request is judged only on what fits in the window at that moment. A few habits help you work within this limit:
- Put the most important instructions and facts near the start or the end of a long prompt.
- Summarize a long conversation yourself and paste the summary if you need to continue later.
- Trim irrelevant material so the useful content has more room.
- For document analysis, give the model only the sections that matter rather than an entire archive.
Context is not the same as understanding
A big context window lets a model look at a lot of text, but looking is not the same as reliably using every detail. Research has shown that models can struggle to retrieve facts buried in the middle of a very long input, a pattern sometimes called the lost in the middle effect. Placement still matters even when everything technically fits, which is why concise, well-organized prompts often beat enormous ones.
Putting it to work
Once you internalize these two ideas, a lot of behavior stops being surprising. Costs rise with length because billing is per token. Responses get cut off when the answer would exceed the model's output limit. Long chats drift because early context has scrolled out of the window. And carefully structured prompts outperform sprawling ones because the model can only weigh what it can currently see.
You do not need to count tokens by hand to benefit from this. Just remember that the model has a fixed window of attention, that everything relevant must fit inside it, and that clear, focused input tends to produce clearer, more focused output. That single mental model will make you a noticeably more effective user of any AI chatbot.