Tokens got two orders of magnitude cheaper. What still meters you?

"Tokens too cheap to meter" topped Hacker News this week. 295 points, 31 comment threads, and the usual split between "this changes everything" and "I've seen this before." I read the whole thing and checked the sources it leans on. The numbers hold up. Cost per task at frontier quality fell about two orders of magnitude in a year, the sources are named and public, and the author's forecast is at least falsifiable: frontier-quality models on commodity hardware within 3 to 6 years.

But the headline buries the more useful question. When inference gets this cheap, a billion tokens stops being a budget line and starts being a design input. What does an agent do with that budget? And which parts of the stack still charge you per use?

The numbers behind the headline

The core claim comes from a 2025-vs-2026 comparison of cost-per-task Pareto frontiers on Artificial Analysis. Same quality targets, X-axis two orders of magnitude cheaper. Broken down: models got roughly 100x more cost-efficient per task over 2025, GPU energy efficiency doubles about every two years per Epoch AI's data, and inference engines added another 10 to 50% year over year. Combine the curves and the author lands at roughly 2.5 orders of magnitude of total cost decline in twelve months.

The number I keep quoting to people, though, is Jev. TypeSafe's classifier reads input at $0.042 per million tokens and generates for free, because output is literally too cheap to meter. That is $42 per billion tokens, public on a pricing page. Their own footnote admits they can't prove the pricing isn't subsidized, which is the right kind of honesty. It doesn't matter much either way: a public number at that magnitude resets what "cheap" means for every classification-shaped workload.

Developers are already treating it that way. jgrep runs a classifier over text in a Unix pipe at about 200 ms and a thousandth of a cent per query. Its README reports 4.6 seconds and $0.012 to score 994 Hacker News titles. A model call that costs less than the grep it wraps is a genuinely new kind of component.

Cheaper than a tool call

The article's most underrated section is a Fermi estimate. GPT-5.6 Luna runs about 30 cents per million tokens, so a 10k-token decision turn costs roughly a third of a cent. Running grep on the author's MacBook costs 0.000007 cents in electricity. Four and a half orders of magnitude apart. Two years ago the model call was the expensive part of every tool-using loop. It is now the cheap part, and the gap is still widening.

That flips the economics of agent design. The standard pattern treats tool calls as free and model calls as scarce, so agents spend their budget deciding when to call. Flip the ratio and you get different systems: model calls inside the tool, classifiers in the pipe, a model grading every diff before a human sees it. None of these were rational designs last year. All of them are rational at current prices.

The memory side of the curve

Tokens are only half of the cost picture. The other half is memory, and it is moving too, just on a different curve. RAM is the scarce input right now, partly because AI companies bought so much of it that component prices spiked. Architectures are responding: Mamba-Transformer hybrids fit about 5x more context in the same memory. NVIDIA's Nemotron-H 47B holds over a million tokens in 32GB of VRAM at 4-bit weights; a comparable dense transformer needs closer to 120GB. Two weeks ago DeepSeek shipped a global KV cache at 890 bytes per token, so a million tokens of context fits in under a gigabyte.

Put those curves together and something clicks. When compute got expensive, we designed databases, caches, and indexes to avoid recomputing. Now that inference is the cheap part, the expensive resources are the ones you can't buy more of: RAM on a local machine, and attention quality over very long contexts. Memory stops being a DB problem and becomes the layout problem of the whole session.

What this means for agent builders

I build agents for a living, so here is how this changed my own planning. First, stop optimizing token counts in hot paths; the meter moved to context assembly and retrieval. Second, assume local hardware catches up: the 3-to-6-year forecast in the article matches what the RAM-density numbers imply, and architecture work like the memory-efficient designs above is exactly how it happens. Third, the layers that still charge per use are the ones to watch. Hosted memory and retrieval services bill per query the way APIs did. Local alternatives exist: our own Uteke recall runs at 0 LLM tokens and about 45 ms warm per query, CPU-only, with 98.4% recall@5 on LongMemEval-S, and it is one Rust binary. (I wrote up the Hermes setup that uses it here.)

That last point generalizes. Every layer that meters you today was cheap to build against when tokens were expensive, because nobody noticed one more line item. The cheap-token era makes those line items visible. Retrieval per query, embeddings per document, egress per GB: each one is now orders of magnitude more expensive than the inference it triggers.

The honest caveats

All the efficiency curves are self-reported or vendor-benchmarked: MLPerf submissions come from NVIDIA and Intel themselves, and the Artificial Analysis frontier is a benchmark suite, which is directional, not a guarantee of task-level cost. The Jev pricing might be subsidized; TypeSafe says so on their own page. And the 3-to-6-year local forecast is a forecast. Reasonable given the curves, but the same genre of prediction has been wrong in both directions before. I treat the two-orders-of-magnitude claim as solid, the architecture trends as directionally solid, and the timeline as a bet worth planning around without betting the company on it.

Where this lands

The article closes by predicting that quality and access become the limiting factors, not token count. My version is narrower: within a couple of years, the expensive parts of an AI system will be the parts that hold state. Memory hierarchies, local retrieval, and whatever keeps a million-token session coherent on hardware you own. Tokens had their price collapse. State didn't, and per-query meters look worse every month that inference gets cheaper. Build for the part of the curve that isn't falling.