NVIDIA Research · ICRA 2026 · a field guide to crossing the reality gap

From simulation to the real world

A robot that's flawless in a simulator can fall on its face the instant it meets a real floor. The gap between the two — friction, light, the give of a real object, the noise in a real sensor — is the oldest problem in robot learning. This is the story of how that gap got crossable: a breakthrough idea, a GPU full of imaginary worlds, and eight new NVIDIA papers showing robots that trained entirely in dreams now working in the open.

The shift underway: from scripted demos in a cage toward robots that adapt, generalize, and hold up outside the lab.

Why you can't just teach a robot in the real world

Modern robot skills are learned, not hand-coded — and learning, especially reinforcement learning, is ravenous. A policy might need millions of attempts to get good at something as simple as nudging a block into place. You cannot run millions of attempts on a physical arm: it's slow, it wears out, it breaks things, and a clumsy early policy is genuinely dangerous. So you do the obvious thing — you train in a simulator, where data is effectively free and infinite.

And then you hit the wall that has haunted robotics for decades. Every simulator is built from models of physics, and models are always a little wrong. Friction isn't quite right. Mass and damping are off by a hair. Contact between soft surfaces is faked. Lighting is too clean; the camera too perfect. So a policy that scores 100% in the simulator meets reality and discovers reality didn't read the same rulebook. That mismatch has a name: the reality gap.

The breakthrough: make the simulation lie in every direction at once

For years the instinct was to close the gap by making simulators more realistic — better friction models, better rendering. It helped, but chasing perfect realism is a treadmill. Around 2017–2018 a different idea took hold, most famously when OpenAI trained a robotic hand called Dactyl to manipulate objects with startling dexterity — entirely in simulation, then transferred to a real Shadow Hand with no real-world training at all. Their trick wasn't realism. It was the opposite.

It's called domain randomization: instead of one carefully-tuned simulation, you train across thousands of deliberately varied ones — randomizing textures, colors, lighting, and the physics itself (friction, mass, damping). The policy never sees the same world twice, so it can't overfit to any single one. By the time you show it the real world, reality just looks like one more random variation it has already learned to handle. Toggle it.

The deep, slightly counterintuitive lesson: you bridge the gap not by perfecting the model but by making the model so promiscuously varied that the truth is contained somewhere inside the range you trained on. OpenAI later automated the difficulty curve of that randomization; the recipe — reinforcement learning plus domain randomization — became the standard blueprint for teaching robots hard, contact-rich skills.

The engine underneath: a single GPU full of worlds

Domain randomization needs scale — thousands of varied worlds, millions of trials. Where does that come from? The answer reshaped the field. In 2021, NVIDIA's Isaac Gym moved the entire physics simulation onto the GPU and kept the whole learning loop — observations, rewards, actions — resident there, never shuttling data back to the CPU. The payoff: tens of thousands of environments running in parallel on a single GPU. Experiments that used to demand a datacenter now ran on a researcher's desktop, collecting experience thousands of times faster. Watch the scale.

That GPU-native paradigm grew up. Isaac Gym became Orbit (built with ETH Zurich and the University of Toronto), and then Isaac Lab in 2024 — adding photorealistic rendering, built-in domain randomization, ready-made digital twins of quadrupeds, drones, and humanoids, and multi-GPU scale. Nearly every paper that follows was trained in it.

And notice the thread running through all of it — the same one that runs through your CUDA rabbit-holes: parallelism is the whole game. Domain randomization runs many worlds in parallel. Isaac Lab runs many environments in parallel. And cuRobo — NVIDIA's CUDA motion-planning library — plans by launching hundreds of candidate trajectories in parallel and optimizing them all at once, generating a collision-free motion in ~30 ms where classical CPU planners crawl. It's one idea — throw a GPU's worth of parallel compute at the problem — wearing different hats at every layer of the stack. Click through that stack.

Coordinating arms, navigating bodies, grasping objects

With that foundation — randomized worlds, parallel simulation, GPU motion planning — the eight papers span the real challenges robot builders hit. The first cluster is about the basics done at a new level: moving many arms at once, carrying one skill across different robot bodies, and the deceptively hard act of closing your hand around a thing.

Picture a pharmaceutical lab run by robotic arms — pipetting, transferring, mixing — each step taking a different amount of time. Traditional scheduling software walks those steps one arm at a time. ScheduleStream runs the planning on the GPU so multiple arms plan and move in parallel, landing a 3× speedup on multi-arm scenarios, fast enough to run on a Jetson edge module.

Next, a quieter but deeper problem. Teach a robot to navigate — dodge obstacles, reach a goal — and it learns to do it in one body. Move that same policy into a differently-shaped robot and it falls apart, because every part now moves differently. COMPASS fixes this: it learns a baseline by imitation, then uses residual reinforcement learning in Isaac Lab to grow specialists for each body shape — wheeled robots, humanoids — with no real-world data at any stage. The result is a 4.5× jump over the imitation baseline and roughly 80% success across 20 real-world trials.

Then there's grasping — which sounds trivial and isn't. Most systems identify the object, predict a grasp, plan a path, and execute it as a fixed plan. But the last few centimeters are exactly where tiny errors become missed grabs. Grasp-MPC grabs the way you do — by feel, continuously correcting its motion as it closes in, rather than committing to a plan up front. It was trained on 2 million simulated trajectories across 8,000 objects (grasp annotations from GraspGen, motion from cuRobo), learning from failures as well as successes — and it grasps novel objects in clutter at about 75% success on real robots, versus 41% for the baseline.

Finally, the tangle. Every system above assumes there's a clean object to grab. Deformable Cluster Manipulation takes on the opposite: grasping a whole bundle of flexible, tangled material at once — the motivating task was clearing tree branches grown over a power line, where there's nothing single to grip. The robot uses its entire arm, wrapping around the cluster and sweeping it aside the way you'd gather an armful of cables. To train it, the researchers grew synthetic trees from biological growth equations — thousands of different shapes in NVIDIA Isaac — and the policy deploys to real branches zero-shot.

Assembling with precision

Grasping is hard; assembly is harder. Threading a nut onto a bolt, seating a gear on a shaft, pressing a peg into a hole — these are exactly the tasks where simulation alone tends to fail. Real surfaces aren't perfectly smooth, real sensors don't behave as specified, and a discrepancy a simulator happily ignores can stop a real robot dead. Two of the papers attack this from different angles.

SPARR splits the job in two. A policy trained in Isaac Lab learns the general strategy for an assembly task in simulation. Then, on the actual hardware, a second layer learns to correct for whatever the simulator got wrong — using only the robot's own camera, with no human demonstrations. That correction layer is the part that quietly absorbs the reality gap, lifting success by 38%, cutting cycle time by ~30%, and improving by nearly 75% on NIST assembly tasks never seen in training — approaching methods that need a human in the loop.

Refinery takes on the next layer of difficulty: assemblies with multiple sequential steps, where how you finish step one decides whether step two is even possible. It's the furniture-assembly trap — leave a panel at the wrong angle and the next fastener simply won't go. By learning how success varies across starting conditions, Refinery completes each step and leaves every component positioned to set up the next one, hitting 91% in simulation with policies that chain together into long, multi-part sequences.

Action models that keep their word

The last two papers reach into the frontier of robot intelligence — vision-language-action models. The lineage is young and fast: Google's RT-1 (2022) first mapped camera images and language straight to robot actions with a transformer; RT-2 (2023) made the leap of treating robot actions as just another language, fine-tuning an internet-pretrained vision-language model so the robot inherited web knowledge and could reason about novel objects; OpenVLA (2024) made that open-source. Powerful — but these models have two very human failure modes, and each paper fixes one.

The first is distraction. A robot's camera takes in everything in a scene, most of it irrelevant. PEEK fixes this by putting a vision-language model in front of the policy: it reads the instruction, highlights only the objects that matter, sketches the movement path, and fades out the rest — then the policy acts on that annotated view instead of the raw clutter. The team's demo says it all: "give the banana to NVIDIA's CEO Jensen Huang," with his photo sitting beside one of Michael Jordan and a pile of distractors. Toggle PEEK on and off.

The second failure mode is subtler and, as tasks get longer, more dangerous: a model can reason correctly about what to do — "store everything on the table in the cabinet," broken into the right steps — and then go and execute something different. SEAL ("Do What You Say," with CMU, Utah, and Sydney) closes that gap at runtime, with no retraining: the robot generates several candidate action sequences, mentally simulates where each one would actually end up, and keeps the one whose outcome matches the plan it stated. A small idea with real teeth — up to 15% better, and robust when you rephrase the instruction, swap the objects, or move the camera.

The fuel: open data at world scale

None of this runs on cleverness alone — it runs on data, and NVIDIA has been pouring it into the commons. Alongside the papers, the company is scaling open datasets for robotics: the kind of shared fuel that, in language models, turned isolated results into a field.

The ChatGPT moment for robots

Step back and the whole arc lines up. For decades the reality gap made robot learning a lab curiosity. Domain randomization (2017–2018) made simulation transferable. GPU-parallel simulation (Isaac Gym, 2021 → Isaac Lab, 2024) made it cheap and massive. Vision-language-action models (RT-2, 2023 → OpenVLA, 2024 → NVIDIA's own GR00T) gave robots a brain that reasons in language and acts in the world. Each layer built on the one below.

Jensen Huang calls this the "ChatGPT moment for robotics," and frames the new wave as needing three computers — one to train the model, one to simulate the world it learns in, and one that rides on the robot itself. That is, almost exactly, the stack you walked through up top: datacenter GPUs, Isaac Lab and Omniverse, and Jetson at the edge. These eight papers are what that stack produces at the research frontier — robots that coordinate, generalize across bodies, grasp tangles, assemble in sequence, and do what they say.

For your track, the payoff is the same thread we started with, now closed into a loop: GPU parallelism is the engine of physical AI. It runs the randomized worlds, the parallel environments, the parallel trajectory seeds, and the foundation models that tie them together. The AWS half of your map is how you operate systems at scale; this half — Isaac, cuRobo, GR00T, the whole Physical AI stack — is what all that parallel compute is ultimately for when it leaves the datacenter and picks something up. Same activity, different abstraction layer.

Now — go build.