DeepSeek just gave away the engine that prints its money. On October 20, 2026, the Chinese AI lab open-sourced DeepInfer — the high-performance inference framework that powers its large-scale production services — under the Apache 2.0 license.
This is not a demo or a paper. It is the actual production stack behind DeepSeek-V4.1-Flash: 155 trillion tokens per day per cluster, a 96% cache hit rate, 230 tokens per second. If you sell AI services, your cost structure just moved.
What DeepInfer Actually Is
From the official README: “DeepInfer is the high-performance inference framework that powers DeepSeek’s large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.”
It ships with the optimizations that make frontier-class inference cheap:
- Custom kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels, all tuned for the hardware.
- MoE expert parallelism — Attention DP / balanced CP plus MoE expert parallelism with load balancing, so mixture-of-experts models route work efficiently.
- Speculative decoding — DSpark with confidence-scheduled verification: draft tokens accelerate decoding, with a default maximum draft length of 5.
- Structured outputs — constrain generation to a regex, JSON schema or EBNF grammar, the thing every serious agent pipeline needs.
- Async scheduling enabled by default, plus batch-invariant inference and deterministic sampling on NVIDIA.
The supported model is DeepSeek-V4.1-Flash, with the default checkpoint downloading from the Hugging Face cache on first run. Supported hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. DeepSeek notes it was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM.
The Production Numbers
These are the figures DeepSeek put on the table for its V4.1-Flash production deployment:
| Metric | Reported figure |
|---|---|
| Throughput per cluster | 155 trillion tokens/day |
| Input cache hit rate | 96% — most context served from memory, not recomputed |
| Time to first token | ~400 ms average |
| Generation speed | 230 tokens/sec, up to 300 tokens/sec for coding |
| Benchmark hardware | 4 SM100 GPUs (prefill), 64 SM100 GPUs (decode), each within one scale-up domain |
Sources: DeepSeek-DeepInfer on GitHub; production figures reported with the official October 20, 2026 release
What You Get — and What Is Honestly Missing
DeepInfer’s README is refreshingly candid about what a complete inference service still needs — and what this release leaves out:
- A KV cache system is not included.
- Prefill/decode disaggregation is not included (though the README notes most required engine functionality is already implemented).
- Load balancing across data-parallel replicas for prefill and decode is not included.
This is offline inference: a batch-processing powerhouse, not a turnkey API server. For freelancers, that is actually good news — the missing pieces are exactly the integration work you can charge for.
How It Works: From Checkpoint to Tokens
DEEPINFER DEPLOYMENT FLOW
1 · Checkpoint
DeepSeek-V4.1-Flash-1017 pulls from the Hugging Face cache on first run.
2 · Build
Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2. One ./build.sh.
3 · Run offline
Custom kernels, MoE parallelism, draft-token speculative decoding kick in.
4 · Your product
Wrap it in your API. Structured JSON output included. Bill per token.
Benchmarks are reproducible: the repo ships run_sweep.sh and an eval README. Verify before you promise clients a number.
How to Profit From DeepInfer
API bills are someone else’s margin. Self-hosting with DeepInfer moves that margin to you.
1. Kill your own API bill first
If your product burns thousands of tokens per user on a frontier API, price out a self-hosted V4.1-Flash deployment. At a reported 96% cache hit rate, repetitive workloads (agents, support bots, RAG) are exactly where the savings compound.
2. Sell inference-as-a-service to niche clients
Law firms, clinics, and local businesses want private AI without per-token API anxiety. Stand up a DeepInfer-backed endpoint for one client, charge a flat monthly fee, and pocket the spread between your hardware cost and the API price they would have paid.
3. Package the missing pieces
DeepInfer ships offline inference — the KV cache system, disaggregation and load balancing are on you. Build and sell that production wrapper as a productized service.
4. Offer Ascend 950 deployments
DeepInfer is one of the few serious frameworks optimized for Huawei Ascend 950. For clients outside the Nvidia supply chain, that is a rare differentiator.
Frequently Asked Questions
Is DeepInfer really free for commercial use?
Yes. It is released under the Apache License 2.0 — use it commercially, modify it, sell services built on it.
Which models does it support?
Currently DeepSeek-V4.1-Flash (default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017). DeepSeek says inference code for future models will land in the same repository.
What hardware do I need?
NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs, on Linux with GCC 13+, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). This is data-center hardware, not a laptop project.
Can I use it as a drop-in API server?
Not yet. This release targets offline inference; the serving layer is on you.
How does it compare to vLLM?
DeepInfer was originally built primarily on vLLM, then specialized with DeepSeek’s own kernels and serving techniques — vLLM’s DNA with DeepSeek’s production tuning.
Is this related to the US federal DeepSeek ban?
The release landed a day after US federal agencies were ordered to remove DeepSeek models from their systems. That affects US government use — the Apache 2.0 code itself is available to everyone else.
Where do I download it?
github.com/deepseek-ai/DeepInfer — already past 5,800 stars at the time of writing.
Sources
- DeepInfer on GitHub (deepseek-ai/DeepInfer) — official README
- DeepSeek-V4.1-Flash-1017 on Hugging Face
- Crypto Briefing — DeepSeek open-sources DeepInfer (production figures)
KEEP READING