Anyone who has spent time around robotics or Physical AI in the last two years has run into the term VLA model. It shows up in NVIDIA keynotes, Physical Intelligence’s funding announcements, robotics papers on arXiv, and, increasingly, in the roadmaps of companies that have nothing to do with robots yet, because the same architectural ideas are starting to appear in autonomous driving stacks, warehouse automation, and even surgical robotics.
So what is a VLA model, actually? A Vision-Language-Action (VLA) model is an AI model that takes in what a robot sees (vision) and what it’s told to do (language), and outputs what the robot should physically do next (action) – all from one unified neural network. Instead of a robot running through a chain of separate perception, planning, and control systems, a VLA model collapses that chain into a single model that goes straight from pixels and words to motor commands.
This guide walks through what a VLA model is, why it matters for robotics, how it works under the hood, how it compares to LLMs and VLMs, which models are shaping the field right now, and the data that train them. It’s written for the people who actually have to make decisions and evaluate what Physical AI means for their strategies.
What Is a VLA Model?
Definition of Vision-Language-Action (VLA) models
A VLA model is a type of foundation model for robotics that takes images (or video) and a language instruction as input and directly outputs robot actions, including joint positions, end-effector movements, gripper commands, or other low-level control signals. The name literally describes the three things being fused into one system:
- Vision: the model perceives its environment through camera input.
- Language: the model understands natural-language instructions, not just a fixed set of pre-programmed commands.
- Action: the model produces the actual motor commands that move the robot, rather than just describing or reasoning about the scene.
That last part is what separates a VLA model from everything that came before it in robotics and AI. Plenty of models can look at an image and describe it. Plenty of models can follow language instructions. A VLA model is built to do both of those things and then close the loop by acting on the physical world, which is why it’s often described as the architecture that turns a “thinking” model into a “doing” one.
Why VLA models are gaining attention
Recently, many companies like 1X Technologies, Figure, Unitree, Agility Robotics, and Tesla have all been racing to build general-purpose robot bodies amid the humanoid robotics boom. Every one of them needs a “brain” that can generalize across tasks without being hand-coded for each one. A VLA model is the closest thing the field has to a reusable brain that can be fine-tuned onto a new robot body rather than rebuilt from scratch.
Furthermore, the biggest AI infrastructure players have all planted a flag here. NVIDIA has built an entire platform for its compute stack. Google DeepMind effectively created the category with RT-2 and has since moved that research line into its Gemini Robotics work. Physical Intelligence, a startup founded in 2024, raised roughly $1.07 billion from backers including Jeff Bezos, Khosla Ventures, Sequoia Capital, Thrive Capital, Lux Capital, and the OpenAI Fund specifically to build general-purpose VLA models. When that much capital and engineering talent converges on one architecture family, it tends to draw the rest of the industry’s attention.
Why Are VLA Models Important for Robotics?
Traditional robotics software is a fragile pipeline: a perception module identifies objects, a planning module maps steps, and a control module executes motor commands. Usually built by three separate engineering teams, these systems are glued together with rigid, hand-written logic.
VLA models replace this brittle pipeline with a single neural network trained end-to-end, unlocking four key advantages:
- Semantic generalization: By leveraging internet-scale vision-language pretraining, VLAs exhibit “common sense” to handle novel objects and instructions.
- Cross-embodiment transfer: A single foundation model can learn from diverse robot bodies and be fine-tuned onto a new one.
- Natural-language interfaces: Operators can direct robots using plain language rather than rewriting control code, making deployment in real environments feasible.
- Faster iteration cycles: Fine-tuning an existing foundation model on a small set of demonstrations is drastically faster and cheaper than building bespoke systems from the ground up.
How Does a VLA Model Work?
Every VLA model follows the same basic loop: it takes in one or more camera images and a language instruction, processes them together through a shared model, and outputs an action, which is sent to the robot’s low-level controller. That action gets executed, the robot’s cameras capture the new state of the world, and the loop repeats; often dozens of times per second.
The interesting differences between VLA models show up in how the “language and vision go in” and “action comes out” steps are actually implemented. There are two dominant approaches in use today:
Token-based (autoregressive) VLAs: Models like RT-2 and the original OpenVLA generate robot actions as discrete tokens, similar to how LLMs predict the next word. Robot movements (e.g., position, rotation, gripper state) are tokenized, making it easy to extend existing language models. However, converting continuous motion into discrete tokens can reduce precision, especially for fast or dexterous tasks.
Diffusion and flow-matching VLAs. Newer models such as π0 and NVIDIA GR00T generate continuous action trajectories instead of discrete tokens. Using diffusion or flow-matching techniques, they predict short sequences of future actions in one pass, resulting in smoother, more precise movements that are better suited for complex manipulation tasks.
Many modern VLA models, including GR00T N1 and π0, separate reasoning from control. A vision-language module interprets the scene and task (“thinking”), while a faster action module generates real-time motor commands (“doing”). This two-system design enables robots to plan intelligently while responding quickly to dynamic environments.
VLA vs. LLM vs. VLM: What’s the Difference?
It’s easy to get these three terms tangled up, since they’re built on the same underlying transformer architecture and often share components. The clearest way to separate them is by tracing what each one actually does, in order.
An LLM is a pure text-in, text-out system: it reads and generates language, but it has no way to perceive images at all. A VLM adds an image encoder on top of that same language backbone, so it can perceive and reason about visual input; but it still only produces text as output, whether that’s a caption, an answer to a visual question, or a description of what’s happening in a scene. A VLA model takes that same vision-language foundation one step further and swaps in an action output: instead of describing what it sees, it acts on it, sending motor commands to a robot instead of words to a screen.
Let’s explore and compare the differences among LLM, VLM, and VLA models:
| Large Language Model (LLM) | Vision-Language Model (VLM) | Vision-Language-Action Model (VLA) | |
| Input | Text | Image + text | Image (+ video) + text instruction |
| Output | Text | Text (description, answer, reasoning) | Robot action (motor commands) |
| Core Capability | Language understanding and generation | Multimodal perception and reasoning | Perception, reasoning, and action execution |
| Example models | GPT-4, Llama, Claude | CLIP, Qwen-VL, LLaVA, GPT-4V | OpenVLA, RT-2, NVIDIA GR00T, π0 |
Popular Vision-Language-Action Models
The VLA landscape moves fast, but a handful of models have defined the field so far, each representing a different point on the open-vs-closed and token-based-vs-continuous-action spectrum.
OpenVLA
OpenVLA proved that open and small can beat closed and huge: in its original evaluation, it outperformed the 55-billion-parameter RT-2-X by 16.5% in task success across 29 tasks, despite having roughly seven times fewer parameters. That result, combined with a fully open release, made it the default starting point for VLA research.
NVIDIA GR00T
NVIDIA Isaac GR00T is built specifically for humanoid robots and relies on NVIDIA’s full simulation-to-deployment stack, such as Omniverse, Cosmos, and Jetson Thor, to generate the data that trains it. The model line moves fast: N1 launched in March 2025, N1.5 roughly tripled zero-shot success on new task verbs, and the current N1.7 is a general-availability release under Apache 2.0.
RT-2
RT-2 is the Google DeepMind model that is widely credited with founding the VLA category. Rather than training something new, DeepMind fine-tuned existing large vision-language models like PaLI-X (55B) and PaLM-E (12B), representing robot actions as text tokens so the same next-token-prediction objective used for language could also predict motor commands. RT-2 was never released publicly, and DeepMind’s production work has since moved to Gemini Robotics, but it remains the paper most cited as the origin of the VLA concept.
π0 (Physical Intelligence)
π0 is Physical Intelligence’s bet that one VLA model can generalize across robot bodies the way one LLM generalizes across text tasks. Its core contribution is flow matching: instead of discretizing actions into tokens or using slow iterative diffusion, it learns a vector field that maps noise directly to a target action distribution, producing 50 future waypoints at 50Hz in a single pass; precise enough for long, dexterous tasks like folding laundry or bussing tables.
A quick comparison of the four models
| Model | Developer | Backbone / Action generation | Strengths |
| OpenVLA | Stanford / UC Berkeley / Google DeepMind / Toyota Research Institute | Llama 2 + DINOv2/SigLIP vision, discretized action tokens | Best open baseline for fine-tuning on modest hardware |
| NVIDIA GR00T | NVIDIA | Vision-language module + diffusion transformer action head | Humanoid robots, full-stack simulation-to-deployment pipeline |
| RT-2 | Google DeepMind | PaLI-X / PaLM-E fine-tuned with action tokens | Historical reference point; strongest semantic generalization demos |
| π0 | Physical Intelligence | PaLI-based VLM + flow-matching action head | Long-horizon, dexterous, bimanual manipulation |
The pattern across all four is the same, even though the specifics differ: start from a model that already understands images and language broadly, then teach it to act. Where they diverge is how open the weights are, how the action is represented, and what kind of task the team behind the model optimized for.
What Data Powers a VLA Model?
None of these architectures matter without data. In fact, at this stage of VLA development, data quality and volume play a far more critical role than model size or algorithmic tweaks.
Common datasets used for VLA training
- Open X-Embodiment (OXE) is the most widely used pretraining dataset in the field, aggregating over a million robot trajectories across 22 robot platforms and 34 institutions for broad cross-embodiment pretraining.
- Task- and platform-specific datasets: fill in more targeted gaps such as BridgeData V2 (~60,000 trajectories, WidowX-250), DROID (~76,000 trajectories, Franka robots), and RH20T (~110,000 episodes across 20 robots), each adding depth on a narrower slice of hardware and tasks.
- Egocentric human video datasets: Ego4D (3,670 hours), Ego-Exo4D, and more recent efforts like EgoScale (20,854 hours) have become a key data source because filming humans doing everyday tasks is far cheaper than collecting teleoperated robot data, and studies show performance scales with video volume.
- Simulation-generated data, produced by pipelines like NVIDIA’s Isaac GR00T Blueprint, multiplies a handful of real demonstrations into hundreds of thousands of synthetic trajectories, scaling data volume without a proportional increase in real-world collection.
When demand grows for both the volume and quality of datasets that power VLA models across Physical AI projects, LTS GDS has expanded its services to include MOCAP, teleoperation, and egocentric data collection and annotation for clients across the US, China, and Europe, with an acceptance rate of up to 96%. Ready to optimize your training data? Connect with the LTS GDS engineering team to discuss your project requirements.
FAQs About VLA Models
1. What is a VLA model?
A Vision-Language-Action (VLA) model is an AI model that takes camera images and a natural-language instruction as input and outputs robot actions, like joint movements or gripper commands. It’s the architecture that lets a robot go directly from seeing and being told what to do, to actually doing it, without a separate hand-built planning and control system in between.
2. What are the key components of a VLA model?
Every VLA model is built around three core components: a vision component that processes camera input, a language component that processes natural-language instructions, and an action component that converts the combined vision-language representation into robot motor commands. Most models build the vision and language components from pretrained encoders and language models, then add the action component as the piece that’s unique to VLA architectures.
3. What is a VLA model in robotics, specifically?
In a robotics context, a VLA model functions as the control policy: the software that decides, at every timestep, what the robot should do next based on what its cameras see and what instruction it’s been given. It typically sits directly on top of (or replaces parts of) a robot’s lower-level motor controllers, which handle the physical execution of whatever action the VLA model outputs.
4. What is the difference between a world model and a VLA model?
A VLA model is reactive by design; it looks at the current scene and instruction and directly outputs the next action. A world model is predictive; it’s trained to simulate how an environment will change over time, including in response to hypothetical actions, before any action is actually taken. Put simply, VLA models act, and world models imagine what might happen if they did. The two aren’t mutually exclusive: a growing body of research combines them, using a world model’s predictive capability to help a VLA model plan further ahead or generate additional synthetic training data, rather than treating the two as competing approaches. NVIDIA’s Cosmos platform, for example, is explicitly built to combine world-model-style prediction with VLA-style action generation rather than choosing one over the other.
5. How much data does training a VLA model require?
It depends heavily on which stage of training is in question. Pretraining a generalist VLA model from scratch typically draws on hundreds of thousands to millions of trajectories or thousands of hours of video – Open X-Embodiment alone contains over a million trajectories, and recent egocentric pretraining efforts have used upwards of 20,000 hours of human video. Fine-tuning an already-pretrained model onto a specific robot and task, by contrast, often only requires a few dozen to a few thousand high-quality demonstrations, since the model is adapting existing general knowledge rather than learning from zero. At the fine-tuning stage, data quality and consistency tend to matter more than raw volume.
The Future of VLA Models and Physical AI
Vision-Language-Action (VLA) models are transforming robotics by enabling machines to perceive, reason, and act in the physical world. These models continue to grow, their success will increasingly depend on access to large-scale, diverse, and high-quality multimodal data; from robot demonstrations and egocentric videos to sensor fusion and human-robot interaction datasets.
For organizations building the next generation of Physical AI, investing in the right data strategy is just as critical as choosing the right model architecture.
Looking to accelerate your VLA development? LTS GDS provides end-to-end AI data solutions for Physical AI and robotics, including multimodal data collection, annotation, validation, and quality assurance. Whether you’re training warehouse robots, humanoids, or autonomous systems, our team can help you build production-ready datasets that power reliable VLA models.
