What Is a Token in an LLM? Context Windows Explained (2026)

What a Token Actually Is — and Why It Isn't Just a Language Detail

A token is the basic unit of text an LLM's tokeniser produces and the model then processes — not a character, not necessarily a whole word, but a chunk of text that the system counts, prices and stores in memory. Everything downstream of that first step — cost, GPU memory, latency, throughput — is measured in this unit, which is why infrastructure teams need to think about tokens the way they think about IOPS or bandwidth: a countable resource, not a linguistic curiosity.

IBM's own framing of the related concept of context makes this explicit: a context window (or context length) is 'the amount of text, in tokens, that an LLM can consider or remember at any one time.' Tokens are simply the accounting unit that definition depends on — get the unit wrong and every capacity calculation built on top of it is wrong too.

The Context Window: An LLM's Working Memory, Defined

Anthropic's Claude platform documentation defines a context window as 'all the text a language model can reference when generating a response, including the response itself.' That last clause matters more than most explainers admit: the tokens your model is actively generating count against the same budget as the tokens you fed it. A long answer to a long prompt can hit the ceiling from both directions at once.

As of August 2026, Claude Opus 5, Opus 4.8, Opus 4.7, Opus 4.6, Sonnet 5 and Sonnet 4.6 all carry a 1M-token context window on the Claude API, Amazon Bedrock, Google Cloud and Microsoft Foundry. In practical terms, that's enough room to hold a very large codebase, a lengthy contract set, or hours of transcript in a single request — but it does not mean the model treats every one of those million tokens for free at inference time, which is where the infrastructure story starts.

Tokens Per Second: Reading Vendor Benchmarks Without Getting Misled

"Tokens per second" sounds like a single, comparable number. It isn't. NVIDIA's own NIM benchmarking documentation defines at least three different TPS metrics, and mixing them up is one of the most common sizing mistakes UK buyers make when comparing vendor claims.

• •NIM's Total TPS per system: total output tokens per second across all simultaneous requests — a system-wide capacity figure.
• •GenAI-perf's TPS: total output tokens divided by the end-to-end latency between the first request and the last response of the last request — a latency-weighted figure.
• •LLM-perf's TPS: total output tokens divided by the entire benchmark duration — a duration-weighted average.

Why Longer Context Inflates GPU Memory: The KV-Cache Problem

Compute Market's 2026 local-AI benchmark on an RTX 5070 Ti shows exactly how context length bites. Running Llama 3 8B at Q4_K_M quantisation, throughput fell from 105 tok/s at a 4K-token context to 62 tok/s at 16K tokens, while VRAM use climbed from roughly 5.2GB to roughly 10.2GB — nearly doubling for a fourfold increase in context. The same benchmark's Mistral 7B result at Q4_K_M reached 112 tok/s at 4K context using about 4.8GB VRAM, underlining that these figures are tied to a specific model, quantisation and context length, not a general rule for 'any 7-8B model.'

The mechanism behind this is the key-value (KV) cache: every token held in context requires the model to store its key/value representations in GPU memory for as long as that context stays live, so memory pressure grows with context length even when the model's parameter count doesn't change. Research groups are actively attacking this. MiniKV, from the University of Illinois, compresses the KV-cache through token eviction and 2-bit quantisation and supports prompts of up to 44K tokens. ShadowKV reports up to 6x larger batch sizes and 3.04x higher throughput on an A100 GPU without sacrificing accuracy. Microsoft Research's Mnemosyne uses a 3D parallelism strategy to scale interactive inference to context lengths of at least 10 million tokens. For teams sizing their own deployment, see how much VRAM an LLM needs and how model quantisation trades memory for precision.

Context Length as a Line Item in UK Hardware Sizing

For UK IT buyers, a long-context LLM deployment is a capacity and TCO decision, not just a model-selection one. Moving to a 1M-token or multi-million-token context target can materially change GPU count, the memory class you need, and how many concurrent users a system can serve — which flows straight into cloud spend or on-prem power and cooling budgets.

The practical test before signing off any hardware plan is simple: does the vendor's tokens-per-second figure match the exact context length, quantisation, batch size and serving stack you will actually run in the UK, or is it a headline number from a different configuration? For regulated environments, add data residency, vendor tenancy and benchmark transparency to that checklist — because, as the numbers above show, TPS can vary sharply with context length and with which benchmark definition was used. If you're weighing self-hosted GPUs against API spend, this is where self-hosting LLM vs cloud GPU cost comparisons need to be run at your intended context length, not a generic default.

Managing Context Without Losing Performance

Shortening prompts is the crudest lever available and often the wrong one for applications that genuinely need long context — long contracts, full codebases, extended transcripts. The more durable strategies work at different layers of the stack. Serving-side KV-cache compression, as demonstrated by MiniKV's token eviction and 2-bit quantisation and by ShadowKV's sparse KV-cache approach, lets a given GPU serve more effective context or more concurrent users from the same memory budget, without touching the application.

Application-side, retrieval-based approaches reduce how much raw text needs to sit inside the context window at all, by fetching only the passages relevant to a given query rather than pasting an entire document store into the prompt. These two approaches are complementary, not substitutes: one shrinks the memory cost per token held in context, the other shrinks the number of tokens that need to be held in context in the first place. Keeping that distinction clear avoids the common mistake of assuming a bigger context window makes retrieval unnecessary.

Data Handling and the UK Regulatory Angle

Every token an LLM holds in context is, for the duration of the request, sitting in an active memory buffer — and, for self-hosted or dedicated-tenancy deployments, in identifiable GPU VRAM. For UK organisations processing personal or commercially sensitive material, the substantive questions are less about prompt wording and more about infrastructure: where is that GPU physically hosted, who else shares the tenancy, and how transparent is the benchmark the vendor used to size the system in the first place. Long-context deployments raise the stakes on all three, simply because more of an organisation's data can sit inside a single active context at once.

Sources

Every figure in this article traces to the sources below.

• •Claude Platform Docs — context window definition and 1M-token model list
• •IBM — definition of context window/context length
• •NVIDIA — NIM benchmarking definitions of tokens-per-second (Total TPS, GenAI-perf, LLM-perf)
• •IDEALS / University of Illinois — MiniKV KV-cache compression and 44K-token prompt support
• •Proceedings of Machine Learning Research — ShadowKV throughput and batch-size gains on A100
• •Microsoft Research — Mnemosyne 3D parallelism for multi-million-token context inference
• •The Register — RTX 3090 benchmark serving 100 concurrent users
• •Compute Market — 2026 RTX 5070 Ti local AI benchmark (Llama 3 8B, Mistral 7B)
• •GIGAGPU — tokens-per-second benchmark chart (RTX 6000 PRO 96GB reference)