Anthropic's new tokenizer, shipped with Sonnet 5 and Opus 4.7/4.8, quietly rewrites the maths behind Anthropic Claude tokenizer pricing 2026. For UK buyers comparing Claude against GPT-5.x on price lists alone, the real bill can land 27–73 percent higher once token counts are accounted for.
View the data behind this chart
| TypeScript | Rust | JavaScript | Python | |
|---|---|---|---|---|
| Claude vs GPT-5.x… | x1.73 | x1.58 | x1.52 | x1.5 |
Why a tokenizer change moves the whole TCO needle
Every LLM bill is built on tokens, not words or characters, and the exchange rate between text and tokens is set entirely by the vendor's tokenizer. Anthropic has confirmed that the tokenizer introduced with Sonnet 5, and shared with Opus 4.7, changes how text is sliced up to improve performance — but the trade-off, in the company's own words, is that "the same input can map to more tokens: roughly 1.0–1.35x depending on the content type." For procurement teams running cloud vs on-premise TCO calculator exercises, that multiplier sits outside the published per-token price and can silently inflate a forecast built on last quarter's usage data.
How much more token-hungry is Claude, really
Independent analysis from AI app platform Playcode put numbers on the gap: the same TypeScript file processed by Claude consumed up to 73 percent more tokens than OpenAI's GPT-5.x family — a 1.73x multiplier — and 1.32x more than Claude's own previous tokenizer. The overhead varies by language: Rust came in at 1.58x, JavaScript at 1.52x, and Python at 1.50x versus GPT-5.x. For engineering teams generating or reviewing code at scale, that's not a rounding error — it's a structural difference in cost-per-task that needs to sit alongside raw per-token pricing in any vendor comparison.
Sonnet 5 pricing: introductory rate versus the real cost
Anthropic has priced Sonnet 5 at $2 per million input tokens and $10 per million output tokens through 31 August 2026 — a rate the company appears to have set deliberately low to offset the extra tokens its new tokenizer generates, making the change roughly cost-neutral for now. After that date, list prices rise to $3 per million input tokens and $15 per million output tokens. Playcode's analysis goes further on Opus 4.8: if Anthropic's list prices were rebased to be genuinely comparable with GPT-5.x's baseline, Opus 4.8 would cost around $7.50 per million input tokens and $37.50 per million output tokens — well above its published $5/$25 rate. UK buyers modelling budgets past the summer should treat the current Sonnet rate as a promotional floor, not a steady-state number.
A real migration puts the theory to the test
The token-count gap isn't just theoretical. A team at marketing platform Ploy this week published results from a production migration comparing OpenAI's GPT-5.6 Sol against Anthropic's Opus 4.8, and the outcome favoured OpenAI on every measured axis: GPT-5.6 finished pages 2.2 times faster, cost 27 percent less, and used roughly half the output tokens of Opus 4.8. That's one workload, not a universal verdict, but it illustrates why buyers can't rely on headline per-token pricing to predict a real invoice — task completion speed and output-token volume both compound the cost gap.
What this means for UK procurement decisions
None of this makes Claude a bad choice — model quality, harness compatibility (Claude Code, Codex, Pi, OpenCode and similar) and existing tooling investment all matter — but it does mean vendor comparisons built purely on advertised $/million-token rates are no longer reliable for budgeting. Teams evaluating inference spend should re-run cost models using actual token counts for their own code and content types rather than vendor list prices, and factor in the tokenizer-driven multiplier when it applies. For organisations weighing whether to keep renting inference at all, this is also a good moment to revisit self-hosting LLM vs cloud GPU cost comparisons, size requirements with an AI GPU calculator, and check how much VRAM an LLM needs before committing capital. Finance teams exploring capex alternatives to unpredictable API bills can also review IT finance options alongside our research on AI servers when building the business case.