DeepSeek open-sourced DeepInfer, the engine behind its inference empire

DeepSeek put a number on its own infrastructure this week. A single cluster serving V4.1 Flash pushes 155 trillion tokens a day and grosses $38 per GPU-hour at off-peak rates, according to a Zhihu announcement from DeepSeek's team that surfaced on Hacker News on October 20. The more interesting sentence came right after: the framework running that workload is now public. DeepInfer, the inference engine behind DeepSeek's production stack, went live on GitHub the same day under the Apache 2.0 license. Within a day it had passed 16,000 stars.

The scope is narrower than the star count suggests, and the README is upfront about it. The release covers offline inference: optimized kernels, the model and sampler runtime, parallelism schemes, speculative decoding with confidence-scheduled verification, asynchronous scheduling, batch-invariant inference, and structured outputs. Three components of the production system are excluded. The KV cache storage layer, the prefill and decode load balancers, and the code that ties them to internal infrastructure all stayed closed. The announcement says the coupling to DeepSeek's internal infrastructure runs too deep to open cleanly.

Where the margin lives

Those closed pieces are where the economics sit. The announcement credits a KV cache retention window of 72 hours or more for cutting repeated prefill work, describes roughly 100 PiB of SSD storage behind it, and reports a measured hit rate above 98 percent on the sliding-window cache that never leaves the machine it was born on. Anyone hoping to replicate the $38-per-card figure will have to rebuild that storage tier from the paper trail. The engine is open, and the storage layer is where the margin lives.

One codebase, two chip ecosystems

The hardware support list deserves a second read. DeepInfer ships kernels for NVIDIA SM100 GPUs and for Huawei Ascend 950 NPUs, with CUDA 13 and CANN 9.2 as the respective build requirements. One codebase, two acceleration ecosystems, and a China-market deployment path that does not depend on a fork. Western inference stacks almost never publish an Ascend path at all, let alone a first-party one.

The acknowledgments section matters just as much. DeepInfer was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. The most-watched inference framework of the last two years now has a production-grade descendant maintained by the largest single tenant of inference compute anywhere. Whatever DeepSeek lands upstream or keeps local, the vLLM lineage is becoming the foundation that frontier-scale serving runs on.

The self-hosting stack, itemized

For a team self-hosting V4.1 Flash, the release converts some assembly from research project to configuration work. The weights are MIT on Hugging Face, the companion toolkit deepseek-recipe converts Messages, Chat Completions, and Responses API requests into DeepSeek's prompt format, and DeepInfer handles the serving loop. You still bring the KV cache tier, the disaggregation layer, and load balancing. That is real engineering, but it is engineering with a reference implementation one directory away.

Numbers you can reproduce

The benchmark story is refreshingly unfussy. The README shows prefill throughput-versus-latency curves measured on 4 SM100 GPUs and decode curves on 64, then ships the sweep script to reproduce both. No headline table, no leaderboard claims. The strongest number in circulation, the 155 trillion daily tokens, comes from the lab's own post rather than the repo, so treat it as a company claim until someone independent replicates it. Hacker News commenters spent the first day poking at exactly that, which is the right instinct.

The same thesis, one layer down

I covered V4.1-Flash itself two weeks ago, when the model's 890-byte-per-token KV cache and its 8-billion-parameter reader paired with a 16-billion-parameter writer made input-heavy agent sessions structurally cheaper to serve. DeepInfer is that thesis pushed one layer down the stack. The architecture made reading cheap. The serving system is what turns cheap reading into a $38-per-card revenue line.

What to watch from here: whether future DeepSeek models ship their inference code in DeepInfer from day one, as the announcement promises, and how quickly the Ascend kernels get stress-tested by people who do not work at DeepSeek. Frontier labs usually treat serving code as competitive advantage and keep it locked. DeepSeek handed over the engine and kept the fuel depot. Now everyone can measure the gap for themselves.