Robot foundation models

A robot brain in 2026 is a vision-language-action model: a network that reads camera images and an instruction such as "put the mug in the sink" and writes out the next second of joint targets. It works well on narrow tasks the company chose and trained for, reaching 99% on some, and it still fails about half the time in rooms it has never seen. The model architectures are published and several are free. The scarce input is data, which is the next chapter, the data engine.

From pipelines to one network

For fifty years a robot was programmed in stages. One program found the mug in the camera image. A second planned a path for the hand. A third made the motors follow that path. Engineers wrote each stage and the interface between them by hand. That design still runs most robots that earn money today: factory arms repeat programmed motions and most delivery robots run classic mapping and route planning estimate.

The new design is end-to-end. One network takes pixels and words in and puts motor targets out, and it learns everything in between from examples. Nobody writes the "where is the mug" step. The network works out what it needs to know about the mug because that helps it predict the right motion.

The learned part keeps eating downward. The control chapter describes the layers under a robot's brain: balance, joint loops, current loops. In January 2026 Figure's Helix 02 replaced the hand-written balance code for Figure 03 with a learned whole-body controller trained entirely in simulation verified Jan 2026. The current loop in each motor driver stays classic and certified, and so do the safety functions in the plumbing chapter.

What a VLA is, part by part

Start with a foundation model, a big network trained on a broad pile of data and then adapted to many jobs. Chinese engineers call the whole family large models. For robots, the starting point is a vision-language model already trained on internet images and text, so it knows what a mug is and what "sink" means. Add a part that writes robot motion instead of words, train it on robot recordings, and you have a VLA. It is multimodal: images, text and joint angles go in together.

The four parts

1. Vision encoder. Cuts each camera image into patches and turns them into tokens, the same kind of number lists a language model reads. Most VLAs use two or three cameras: one on the head and one on each wrist.
2. Language backbone. A vision-language model with billions of weights: PaliGemma 3B in π0, Gemma 3 4B in π*0.6, a Cosmos-2B variant in NVIDIA's GR00T N1.6 verified Sep 2026. It carries what the internet taught it about objects and words.
3. Action head. The part that writes motion. There are two ways to do it, covered below: write actions as tokens, or run a separate "action expert" that generates a smooth trajectory.
4. Action chunk. The output is a short plan. π0 writes 50 future steps, about one second at 50 Hz verified. The robot executes part of it, looks again, and writes a new chunk.

Two ways to write actions

Actions as tokens. Google DeepMind's RT-2 (July 2023) wrote joint movements as text-like tokens, so the language model could output them the way it outputs words verified. Raw tokens are too coarse for fast, fine motion. Physical Intelligence fixed that in January 2025 with FAST, an action tokenizer. It compresses each joint's trajectory with a discrete cosine transform, rounds the coefficients and packs them with byte-pair encoding, the same trick that shrinks text for language models. A token model trained with FAST matched the diffusion version of π0 with up to 5 times faster training verified Jan 2025.

A separate action expert. Most flagships instead bolt a smaller network onto the backbone that generates continuous motion. Two methods dominate. A diffusion policy starts from random noise and removes it step by step until a motion appears, the way image generators work. NVIDIA's GR00T and Toyota Research Institute's Large Behavior Models use this verified. Flow matching learns a direct route from noise to motion along nearly straight paths, so about 10 steps are enough step count unverified. π0, π0.5, π*0.6 and Hugging Face's SmolVLA use it.

A slow thinker and a fast doer

A 7-billion-weight model cannot run 200 times a second on a robot's onboard computer. So most 2025 to 2026 flagships split the brain in two, borrowing the psychologist Daniel Kahneman's fast and slow thinking. The dual-system architecture runs a large "System 2" model that understands the scene and the instruction a few times a second, and a small "System 1" policy that turns its intent into motor commands many times a second. The newest humanoid stacks add a third layer underneath for balance.

Labs size the two halves very differently. Figure's fast network is about 1% of its weights. In NVIDIA's GR00T N1 (2.2B weights in total, 1.34B in the Eagle-2 vision-language model) the action part is about 40% verified Mar 2025. Google DeepMind's Gemini Robotics 1.5 ships the split as two separate models: ER 1.5 plans and calls tools, and the VLA "thinks" in language before it acts verified Sep 2025. AgiBot's GO-2 (April 2026) calls its version an asynchronous dual system with "action chain-of-thought" verified Apr 2026.

The Chinese split: big brain and little brain

Chinese engineers describe the same idea in anatomy words. The big brain and little brain split puts task understanding and planning in the 大脑 dànǎo, and balance, walking and joint coordination in the 小脑 xiǎonǎo, often called 运控 yùnkòng (motion control). It matches Figure's S2 over S0, with S1 sitting between.

The split is also a business map. Chinese body makers are strongest at the little brain: Unitree's walking and falling recovery come from reinforcement learning in simulation, covered in the locomotion chapter. Big-brain models come from AgiBot (GO-1, GO-2), UBTech (BrainNet 2.0, a "dual-loop" system) verified, and other names that were not verified for this page: Unitree UnifoLM, X Square Robot's WALL-OSS, Galbot's grasping VLAs, BAAI's RoboBrain and GigaAI's GigaBrain unverified. In the West the same map has three camps: in-house brains (Figure, Tesla, 1X, Dyna, Agility), Google DeepMind's Gemini Robotics inside other people's bodies (Apptronik, Boston Dynamics), and merchant brains sold to anyone (Physical Intelligence, Skild AI, NVIDIA GR00T) verified Sep 2026.

How a robot brain is trained

Training runs in stages, the same shape as a chatbot's but with robot data in the middle.

The first two stages are imitation learning: the model sees what a human did and learns to do the same. The plainest form is behaviour cloning, ordinary supervised learning on pairs of "what the robot saw" and "what the operator did". It is stable and needs relatively little data. Its ceiling is the demonstrator. Worse, small errors push the robot into situations the demonstrations never showed, and those errors compound verified.

Learning from mistakes: π*0.6 and Recap

The fix is practice. In November 2025 Physical Intelligence published π*0.6, trained with a method it calls Recap. Three stages: learn from demonstrations, collect expert corrections while the robot runs on its own, then improve with reinforcement learning from the robot's own attempts, graded by a separate 670M-weight value function verified Nov 2025. On its hardest tasks throughput more than doubled and the failure rate roughly halved. It made espresso for 18 hours straight and reached 97% on laundry after a round aimed at removing failures company figures. In April 2026 π0.7 went further and trained on autonomous data including failures, each labelled with how well it went, so at run time you can ask for the good behaviour verified Apr 2026.

One brain, many bodies

Cross-embodiment training pools data from different robots so each helps the others. The Open X-Embodiment project pooled about 1 million trajectories from 22 robot types; a model trained on the pool beat single-robot specialists by 50% verified Oct 2023. Pooling very different bodies naively still fails verified. In 2026 the claims got bigger: π0.7 folded laundry on a robot that had no laundry data, Gemini Robotics On-Device 2 adapts to a new body with "a few hours of data", and Skild AI says its model adjusts to a new body from its recent experience without retraining company claims. The general skill is called generalization: doing the task in a room, with an object or on a body the model never trained on.

World models: a simulator that learned from video

A world model predicts what the camera will see next if the robot does a given action. Think of it as a video generator that takes joint commands as its prompt. Labs use it three ways.

1. Make training data. NVIDIA's GR00T-Dreams generated 780,000 robot trajectories in 11 hours from single images and text, which let NVIDIA build GR00T N1.5 in 36 hours instead of about three months, with 40% better results than real data alone verified Jun 2025.
2. Grade a policy before it touches hardware. 1X's World Model (January 2026) predicts the result of NEO's actions and scores policies inside the prediction verified Jan 2026.
3. Plan. Meta's V-JEPA 2 predicts in a compressed internal space instead of pixels. After pretraining on more than 1 million hours of internet video and under 62 hours of robot video, it planned pick-and-place moves on Franka arms in two labs, scoring 65 to 80% on unseen objects verified Jun 2025.

The other names: NVIDIA's Cosmos family, open and downloaded more than 2 million times by January 2026, used by 1X, Agility and Figure; Cosmos 3 in June 2026 verified. Google DeepMind's Genie 3 (August 2025) generates navigable 3D worlds from a text prompt, at 720p and 24 frames a second for a few minutes unverified. Dyna-2 (August 2026) is a "world-action model" that predicts video and actions together; NVIDIA's GR00T N2, previewed in March 2026, takes the same approach company claims. World-model startups raised about US$6 billion in the first quarter of 2026 verified Jun 2026.

The landscape, September 2026

Sizes are weights (parameters). "Open" means you can download the weights. Company-reported unless marked. verified Sep 2026

Three patterns sit in that table. First, the biggest labs moved to human video in August and September 2026: Dyna-2 on 1 million hours of head-camera footage, Figure's Helix 2.5 on its Index crowd video, and both claim a scaling law for moving skill from human video to robots company claims. Second, the open models lag the closed ones by about a year, and they are good enough to start with. Third, brain labs re-rate about three times per funding round: Physical Intelligence went from US$2.4 billion to US$5.6 billion to about US$11 billion, Skild AI from US$4.5 billion to more than US$14 billion on about US$30 million of 2025 revenue verified.

What works today and what does not

The success rate alone misleads. A robot that succeeds 95% of the time at 20% of human speed with a person watching is a different product from one that succeeds 80% of the time at 60% speed with nobody watching. Four numbers together tell the story: success rate, speed against a person, interventions per hour, and hours of continuous running without a reset.

The evaluation crisis

1. No shared hardware. A language model benchmark runs anywhere. A robot result depends on the arm, the gripper, the cameras, the lighting and the table height.
2. Resets are slow. Someone has to put the towel back. Toyota Research Institute needed 1,800 real rollouts plus 47,000 simulated ones for one careful study, the most statistically careful in the field verified Jul 2025.
3. Small samples lie. Twenty trials give a wide margin of error. Most published results use fewer.
4. Simulation benchmarks saturate. Reported scores on the LIBERO benchmark rose from about 75% to about 98% in 16 months, and a perturbed version, LIBERO-PRO, shows scores nearly collapse when object positions or wording shift slightly verified. AgiBot still headlined LIBERO 98.5% for GO-2 in April 2026.
5. Companies pick the tasks. Every bar in FIG 10.4 is a company test on a company-chosen task.

Real autonomy or a person with a headset

Many robot demos have a human driving. Critics reported that the drink-serving robots at Tesla's October 2024 event were largely teleoperated without saying so. The Wall Street Journal found every household task in a 1X NEO demo session in October 2025 was teleoperated unverified primary. A Figure livestream in May 2026 claimed "zero failures" while viewers saw fumbled packages and resets unverified. Teleoperation is a legitimate way to collect data and fill gaps; the problem is only when it is sold as autonomy. Use this in showrooms.

Dance and kung-fu routines are trained or scripted for that one routine. They show you balance and hardware quality, and very little about the brain.

Open models anyone can fine-tune

Open-source robot models are real and free. What you need is robot data for your task and a GPU. Running a trained model on the robot is inference; a Jetson Thor developer kit, the common onboard computer for Western humanoids, costs US$3,499 and draws up to 130 W from the same battery as the motors verified Aug 2025.

Terms in this chapter