On October 20, 2026, NVIDIA released Cosmos3-H-Dreams, a real-time, action-conditioned generative world model for robotic surgery. The model generates photorealistic endoscopic video at up to 43.2 frames per second on a single RTX PRO 6000 GPU from live robot commands — and the NVIDIA Open Model License says it is ready for commercial use.
Cholecystectomy — the procedure it was trained on — is the surgical removal of the gallbladder, one of the most common robot-assisted operations.
What NVIDIA just shipped
Cosmos3-H-Dreams is a world model: it learns how a scene evolves instead of computing physics. From one camera frame plus a stream of robot movement commands, it generates the video of what the camera would see next — autoregressively, in real time.
The training data is the standout: the model was fine-tuned on video and robot kinematics recorded during real clinical procedures on the CMR Surgical Versius platform. NVIDIA: “To our knowledge, this is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.”
Key stats, straight from NVIDIA
- 43.2 fps at 832×480 with the 2-step model; 30.0 fps with the 4-step model — both on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU
- 15.17 billion parameters, built on Cosmos3-H-Embodied-Nano (pretrained on Open-H-Embodiment)
- 28 action dimensions per step: pose and gripper data for both arms, energy controls, thumbstick inputs
- 64 clinical recordings, ~526 unique minutes of surgical video at 10 Hz (drawn from 4,792 episodes / ~17M raw frames)
- Two controlled instruments: a left Cadiere grasper and a right monopolar hook
- Accelerated by FlashDreams, NVIDIA’s streaming-inference library
The numbers: how fast is it, really?
The model card publishes the inference-throughput table measured in real serving conditions:
| GPU | Resolution | Diffusion steps | Inference FPS |
|---|---|---|---|
| RTX PRO 6000 Blackwell Workstation Edition | 832×480 | 4 | 30.0 |
| RTX PRO 6000 Blackwell Workstation Edition | 832×480 | 2 | 43.2 |
| RTX PRO 6000 Blackwell Max-Q | 832×480 | 4 | 20.2 |
| RTX PRO 6000 Blackwell Max-Q | 832×480 | 2 | 31.2 |
| DGX Spark | 512×288 | 2 | 16.9 |
Source: NVIDIA Cosmos3-H-Dreams model card, Hugging Face (Oct 20, 2026).
Two separately trained checkpoints — 4-step and 2-step — let developers pick the quality-speed tradeoff. The model also runs on Hopper H100 and DGX Spark.
How the world model works
Traditional simulators hand-author tissue geometry and physics. Cosmos3-H-Dreams learns visual dynamics directly from synchronized video and robot kinematics: small instrument movements reshape the whole scene — tissue stretches, reflections shift, smoke reduces visibility — while the model preserves anatomy and instruments as it generates each block.
Per NVIDIA’s pipeline: a bidirectional teacher is fine-tuned on curated cholecystectomy video and kinematics, Self-Forcing distillation converts it into a causal student that generates video in short blocks, and FlashDreams streams it in real time with an adapted FadeMem module consolidating the attention cache.
Background: from tabletop to the operating room
| Aspect | Cosmos-H-Dreams (predecessor) | Cosmos3-H-Dreams (new) |
|---|---|---|
| Base model | Cosmos-Predict2.5 | Cosmos3-H-Embodied-Nano (16B) |
| Training data | dVRK tabletop suturing | Clinical cholecystectomy on CMR Versius |
| Instruments | Suturing setup | Cadiere grasper + monopolar hook, 28D actions |
| Runtime | Interactive | Interactive on one RTX PRO 6000 |
The fine print: research only, not patient care
Cosmos3-H-Dreams is a research simulator, not a validated system for patient care. The company warns that looking realistic “only opens the door to validation and does not show tissue behaves accurately.”
Per NVIDIA: simulate the full operation, evaluation with surgeons, more instruments beyond the two controlled tools, and autonomous surgical policies trained in the loop.
How to profit: the money angle
Why this matters beyond research: the model card says it is “ready for commercial or non-commercial use” under the NVIDIA Open Model License, and the checkpoint and code shipped the same day (Oct 20, 2026) on GitHub and Hugging Face. Open weights plus a commercial-friendly license creates real openings:
- Synthetic data as a product. Generate and sell synthetic cholecystectomy datasets for training downstream surgical AI — NVIDIA lists synthetic data generation as an intended use.
- Policy evaluation as a service. Run robotics startups’ control policies inside Cosmos3-H-Dreams rollouts and sell the validation results.
- Fine-tuning consulting. NVIDIA ships a recipe for teacher fine-tuning and Self-Forcing distillation. Clinics, device makers, and med-tech startups will pay teams that run it on their own data.
- Teleoperation demos. The repo supports live control from a WebXR headset. Build training experiences and demos for medical conferences and surgical education companies.
- Ecosystem tooling. Tutorials, benchmarks, and managed inference endpoints — sold as subscriptions.
One legal guardrail: stay in the research and simulation lane. NVIDIA says the model is not validated for patient care, so do not market anything built on it as clinically ready.
FAQ
What is NVIDIA Cosmos3-H-Dreams?
A real-time, action-conditioned generative world model for robotic surgery, released Oct 20, 2026. It generates live endoscopic video of cholecystectomy from robot movement commands.
How fast does it run?
Up to 43.2 fps at 832×480 (2-step checkpoint; 30.0 fps for 4-step) on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition, per the model card.
What was it trained on?
Clinical recordings from real cholecystectomy procedures on the CMR Surgical Versius platform — 64 recordings, ~526 unique minutes of video — plus pretraining on Open-H-Embodiment.
Is commercial use allowed?
According to NVIDIA, yes: the model card says it is ready for commercial or non-commercial use under the NVIDIA Open Model License. Read the license yourself first.
Can it be used for patient care?
No. NVIDIA calls it a research simulator, not a validated system for patient care. Realistic-looking output does not prove the simulated tissue behaves like real tissue.
Where can I download it?
Checkpoint on Hugging Face, code on GitHub (both Oct 20, 2026). Announcement: the NVIDIA-authored post on Hugging Face.
What comes next?
Per NVIDIA: simulating the full operation, evaluation with surgeons, more instruments, and autonomous surgical policies trained in the loop.
Keep Reading
Sources: NVIDIA announcement · model card.