NVIDIA’s new NVIDIA Nemotron 3.5 retrieval models could cut your AI agent’s token bill by roughly 14% per query — and the weights are free to download. Announced October 20, 2026, the five-model Nemotron 3.5 Retriever collection targets the most expensive bottleneck in AI agents: bad search results that force repeated lookups.
Here’s the money logic. When an agent retrieves weak evidence, it burns extra searches and re-reads documents — each retry spending input and output tokens you pay for. NVIDIA tested retrieval quality against total agent inference tokens across ViDoRe v3, BRIGHT, and BrowseComp-Plus, and found that adding its reranker cut agent inference tokens per query by ~14% with Embed 8B and ~13% with Embed 1B, while improving retrieval accuracy.
“An agent’s answers depend on the evidence it can retrieve. When that evidence is incomplete or irrelevant, the agent will repeat searches and inspect additional documents, consuming more inference tokens before completing the task.” — NVIDIA, in its official announcement on Hugging Face.
KEY NUMBERS — NVIDIA Nemotron 3.5 Retriever
- 📉 −14% agent inference tokens/query — Embed 8B + Rerank (NVIDIA’s agent evals)
- 📉 −13% agent inference tokens/query — Embed 1B + Rerank
- 🏆 61.17 nDCG@10 on ViDoRe v3.1 — #1 among dense retrieval models
- ⚡ 128.8 docs/sec — Embed 1B NVFP4 on GB200 (+27.1% vs BF16, 99.85% accuracy kept)
- 📄 75.20 ParseBench — Parse (631M) ranks #1 among models under 1B params
Token figures describe NVIDIA’s tested setups (fixed model, tools, tasks, corpus, execution policy) — not guaranteed savings on every workload.
The 5 New Models, Explained
NVIDIA describes all five as open, commercially available weights for embedding, reranking, and document extraction. Each covers a different stage of the search pipeline — and they combine into one system.
| Model | Parameters | Role | Best for |
|---|---|---|---|
| Nemotron 3.5 Embed 8B BF16 | 8.38B | Retrieves evidence across text and images (tables, charts, infographics, datasheets, schematics) | Quality-focused search over rich documents |
| Nemotron 3.5 Embed 1B BF16 | 1B | Distills the 8B’s multimodal retrieval into a compact model | Multimodal indexing + query serving, small footprint |
| Nemotron 3.5 Embed 1B NVFP4 | 1B (quantized) | NVFP4 quantization on text and image encoders | Large-scale indexing, high-throughput retrieval |
| Nemotron 3.5 Rerank 8B BF16 | 8B | Cross-encoder: scores each query–document pair jointly to refine ranking | Picking the strongest evidence for RAG and agents |
| Nemotron 3.5 Parse | 631M | Converts document images to structured text (Markdown, HTML tables) with reading order and element locations | Document pipelines at scale |
The Embed 8B produces 4,096-dimensional embeddings with a 32,768-token context, is built on Mistral’s Ministral 3 8B Instruct, and ships under the OpenMDW 1.1 license. Embeddings from the earlier Nemotron 3 text models remain compatible — existing vector indexes can be reused.
Why Retrieval Decides Your AI Bill
Most AI cost optimization chases cheaper models. NVIDIA’s argument runs deeper: the retrieval layer determines how many expensive model calls an agent needs at all.
An embedding model converts queries and documents into numerical vectors so a system can compare them. The Nemotron 3.5 Embed family does this for text and document images — a chart or table becomes searchable, not just the surrounding prose. It was evaluated in 34 languages for text retrieval, plus MIRACL Vision (18 languages) and KoViDoRe for visual retrieval.
A reranker is the second stage. Instead of scoring query and document separately, the 8B cross-encoder examines each pair together and separates “broadly related” from “directly answers the question.” Better ordering means the agent’s context fills with strong evidence first — fewer retries, fewer wasted tokens.
In NVIDIA’s pipeline tests (ViDoRe v3 for text+image, BRIGHT and BrowseComp-Plus for text), the retrieval configuration was the only thing varied — the LLM, tools, tasks, corpus, and execution policy stayed fixed. Results:
- Embed 8B + Rerank: ~14% fewer agent inference tokens per query, higher retrieval accuracy
- Embed 1B + Rerank: ~13% fewer agent inference tokens per query, higher retrieval accuracy
- Same-budget bonus: Embed 8B + Rerank beat Qwen3-VL-Reranker-8B by ~0.9 nDCG@10 points at similar token cost
The Pipeline: Where Token Waste Dies
1 · PARSE
PDFs & images → structured text (tables kept as tables)
2 · EMBED
Query + docs → vectors; top candidates retrieved
3 · RERANK ⭐
Cross-encoder re-orders candidates; weak evidence cut
4 · AGENT
Reasons over strong evidence — ~14% fewer tokens
Each document flows once through this chain. Reranking at stage 3 prevents the agent from re-searching — the compounding cost loop is what the 14% figure captures.
The Benchmark Evidence
The numbers NVIDIA published (all from its official announcement):
| Benchmark | Nemotron 3.5 result | Baseline |
|---|---|---|
| ViDoRe v3.1 (dense retrieval) | 61.17 nDCG@10 — #1 | Llama Nemotron Embed VL 1B: 52.23 (+8.94 gap) |
| Embed 1B vs its 8B teacher | 58.38 nDCG@10 — keeps 95.4% of teacher’s score | Tiny model, near-flagship quality |
| MIRACL Vision (visual) | 72.6 nDCG@10 | Qwen3-VL-Embedding-8B: 4.3 points behind |
| ViDoRe v3.1 reranking | 60.49 nDCG@10 | Qwen3-VL-Reranker-8B: 59.17 |
Parse is its own story. The 631M-parameter model ranks #1 on ParseBench among models under 1 billion parameters with 75.20, processing 10.5 pages per second on a single H100. Its accuracy scales with passes:
- Single pass: 72.74 ParseBench · 16.1 pages/sec — fastest ingestion
- Double pass: 74.92 · 11.2 pages/sec — revisits tables/charts
- Triple pass: 75.20 · 10.5 pages/sec — adds a detection pass
And the efficiency angle for scale: Embed 1B NVFP4 hits 128.8 documents per second on GB200 — 27.1% faster than the BF16 version while retaining 99.85% of retrieval accuracy. Via NVIDIA NIM on an RTX PRO 6000 Blackwell Server Edition, Embed reaches 6,031 query requests/second — about 2.9× a comparable vLLM setup.
How to Profit: 5 Concrete Steps for Builders
These models are open weights with commercial use — here’s how to turn the release into a real cost advantage this week:
- Grab the weights on Hugging Face. The collection includes all five models plus fine-tuning recipes. Embed 8B ships under OpenMDW 1.1; all five are described as commercially available.
- Upgrade your document pipeline. Run Parse on your PDFs, charts, and scans first (Markdown + HTML tables with reading order), then index with Embed 1B NVFP4 for high-throughput serving. Old text-only pipelines silently drop chart data — that’s the gap competitors still have.
- Add Rerank 8B as your second stage. NVIDIA’s agent tests show ~14% fewer inference tokens per query (with Embed 8B) when reranking is added. On a 10,000-query/day agent workload, that’s a direct, recurring cut to your LLM API spend. Validate on your own documents first — the figures describe NVIDIA’s fixed test setups.
- Serve it the fast way. Use NVIDIA NIM (6,031 qps measured on RTX PRO 6000, ~2.9× the tested vLLM config) or inference partners already listing the models: Baseten, DeepInfra, FriendliAI, and OpenRouter.
- Fine-tune on your own documents. NVIDIA’s domain-adaptation recipe reported an 18% relative nDCG@10 gain for the 1B model on chart/infographic content (and 3% for the 8B) after one epoch on synthetic queries. Better fit to your terminology = fewer wasted tokens.
If you sell RAG-as-a-service, search products, or agent apps, this release is a pricing weapon: swap your retrieval stack, benchmark token spend before and after, and pocket the margin or undercut the market.
Enterprise Adoption Is Already Rolling
Real companies reported early results in the announcement:
- Elastic — embedding and reranking models are available through Elasticsearch’s Inference API via NVIDIA inference endpoints
- Vectorize — Embed ranked #1 on two long-conversation memory benchmarks: 93.8% recall@10 on LoCoMo and 65.9% on BEAM 100K, against seven other embedding models
- Perplexity — Rerank 8B scored the highest average on Perplexity’s private 72-task reranking benchmark; Embed posted strong results on its Q2D-Web agentic search benchmark
- Turbopuffer — Embed 8B returned ~5% more relevant results at Recall@20 than the next best model on its enterprise PDF search benchmark; now available natively
FAQ
What is NVIDIA Nemotron 3.5 Retriever?
A five-model open-weight collection for AI agent search: three Embed variants (8B BF16, 1B BF16, 1B NVFP4), an 8B cross-encoder Rerank model, and a 631M-parameter Parse model for document extraction. They handle embedding, reranking, and parsing — the three stages before an agent reasons.
How does reranking save me money?
Better evidence ordering means the agent finds what it needs without repeat searches or reading irrelevant documents. NVIDIA measured ~14% fewer agent inference tokens per query (Embed 8B + Rerank) and ~13% (Embed 1B + Rerank) in fixed test setups — tokens you would otherwise pay a model provider for.
Are the models free to use commercially?
Yes. NVIDIA describes all five as open and commercially available weights on Hugging Face. The Embed 8B ships under the OpenMDW 1.1 license. No inference fees flow to NVIDIA unless you choose to deploy on paid infrastructure or managed endpoints.
What can the Embed models actually search?
Document text plus visual content: tables, charts, infographics, datasheets, and schematics. Text retrieval was evaluated in 34 languages; visual retrieval on MIRACL Vision (18 languages) and KoViDoRe.
What’s the difference between embedding and reranking?
Embedding turns queries and documents into vectors to pull a broad candidate set fast. Reranking then scores each candidate against the query jointly (a cross-encoder) to surface the strongest evidence first. One finds the stack of files; the other decides which to read.
Will I really see 14% token savings?
Maybe, maybe not. That figure comes from NVIDIA’s tests with a fixed model, tools, task set, corpus, and execution policy — your mileage depends on your documents, queries, and agent setup. The right move: benchmark token spend on your own workload before and after adding Rerank.
How does Nemotron 3.5 Parse compare to regular PDF extraction?
Standard PDF text extraction loses reading order and table structure; scans may have no text layer at all. Parse converts document images into structured Markdown/HTML with element locations and reading order — and ranks #1 on ParseBench among models under 1B parameters.
KEEP READING
Source: NVIDIA — “Nemotron 3.5 Embed, Rerank, and Parse: Advancing Agents with World-Class Retrieval Models” (Hugging Face blog, October 20, 2026). All figures above come from that announcement; benchmark results describe NVIDIA’s reported tests, not independent verification.