← AI for Professionals: Using LLMs at Work
Lesson
Lesson 3: Tokens, the context window, and what models can and cannot do
Explain tokens and the limited context window, and classify everyday tasks into things LLMs are reliably good at versus things they are weak at.
Tokens, context windows, and the model's real limits
What the model can hold — and what it cannot
LLMs do not process text letter by letter or word by word. They break text into chunks called tokens — typically a few characters each. 'unhelpful' might become three tokens: 'un', 'help', 'ful'. A rough rule of thumb is that 1,000 tokens ≈ 750 words, though this varies by language. Every token in your conversation — your messages, the model's replies, any documents you paste in — occupies space in the context window, which is the model's finite working memory for one session.
When the context window fills up, earlier content is pushed out and can no longer influence the answer. Research also shows a 'context rot' effect: even before content drops out entirely, tokens in the early and middle sections of a very long context receive less attention from the model. More text is not always better — extremely long prompts can cause the model to miss instructions you placed at the top. There is also no cross-session memory: when you start a new chat, the window is empty again unless the product explicitly stores and re-injects previous conversations.
Understanding this shapes where to rely on an LLM. It excels at drafting, summarizing, rephrasing, explaining concepts, brainstorming ideas, and writing or reviewing routine text and code — tasks where a fluent, pattern-based output is useful even if it needs a human review. It is weak at providing exact verified facts without a source, reporting recent events after its training cutoff, doing arithmetic reliably without a calculator tool, and making legally or medically binding decisions. Matching the right task to the tool is the foundation of effective AI use at work.
Lesson notes
What the model can hold — and what it cannot
LLMs do not process text letter by letter or word by word. They break text into chunks called tokens — typically a few characters each. 'unhelpful' might become three tokens: 'un', 'help', 'ful'. A rough rule of thumb is that 1,000 tokens ≈ 750 words, though this varies by language. Every token in your conversation — your messages, the model's replies, any documents you paste in — occupies space in the context window, which is the model's finite working memory for one session.
When the context window fills up, earlier content is pushed out and can no longer influence the answer. Research also shows a 'context rot' effect: even before content drops out entirely, tokens in the early and middle sections of a very long context receive less attention from the model. More text is not always better — extremely long prompts can cause the model to miss instructions you placed at the top. There is also no cross-session memory: when you start a new chat, the window is empty again unless the product explicitly stores and re-injects previous conversations.
Understanding this shapes where to rely on an LLM. It excels at drafting, summarizing, rephrasing, explaining concepts, brainstorming ideas, and writing or reviewing routine text and code — tasks where a fluent, pattern-based output is useful even if it needs a human review. It is weak at providing exact verified facts without a source, reporting recent events after its training cutoff, doing arithmetic reliably without a calculator tool, and making legally or medically binding decisions. Matching the right task to the tool is the foundation of effective AI use at work.