Why Foundation Models for Robots Are Different
The lessons from large language models don't transfer cleanly to embodied systems. Action spaces, physical grounding, and the sim-to-real gap make robot foundation models a fundamentally harder problem — and a more interesting one.
The past three years have produced a remarkable series of demonstrations: robots that can follow verbal instructions to sort laundry, assemble furniture from scratch, operate in kitchens they've never been in before, and respond to corrections mid-task. The AI research community, energized by the success of large language models, has poured extraordinary resources into foundation models for robots — the thesis that the same principles producing GPT-4 can be applied to physical action. It is a legitimate and important research direction. It is also much harder than the LLM analogy suggests, and understanding precisely where that analogy breaks down matters for building conviction in the right companies.
The Token-Action Gap
Language models work because language is discrete, compositional, and captured at extraordinary scale in human-generated text. The space of possible next tokens at any position in a sentence is bounded and categorical. A model trained on enough text learns representations of language, reasoning, and world knowledge because those things are encoded in the statistical structure of written human communication.
Robot actions are different in almost every dimension. Actions are continuous: the space of possible joint velocities, gripper forces, and end-effector positions is infinite-dimensional. There is no internet-scale corpus of robot demonstration data — the training data must be collected specifically, in the physical world, at significant cost per hour. The consequence of a wrong prediction is not a grammatically awkward sentence but a physical event with physical consequences, potentially including damage to equipment, parts, or people. And crucially, the same action can be correct in one physical context and catastrophically wrong in another based on physical state differences that are not visible in any single camera image.
This is why naive application of the LLM recipe to robotics fails. You cannot scrape the internet for robot demonstration data. Scaling compute alone does not produce emergent manipulation capabilities the way scaling produces emergent reasoning in language. The data modalities, action representations, and training objectives all need to be rethought from the ground up for the embodied setting.
What Physical Intelligence Got Right
Physical Intelligence (Pi), founded by Sergey Levine, Chelsea Finn, and colleagues from the Berkeley robotics and ML community, has thought more carefully about these distinctions than almost anyone. Their π0 model is notable for several architectural decisions that reflect the specific demands of robot learning rather than borrowed assumptions from language modeling.
π0 uses a flow matching objective for action generation — a continuous, diffusion-like process that's better suited to the continuous action space of manipulation than the discrete token prediction used in language models. It conditions actions jointly on language instructions and visual observations, enabling natural-language task specification grounded in visual scene understanding. And it was pre-trained on a large, diverse dataset of robot demonstrations across dozens of task types and robot morphologies — the breadth of pre-training that the foundation model paradigm requires to produce generalizable representations.
The results show meaningful zero-shot generalization to new objects and instructions, and with small amounts of fine-tuning data, the model adapts to new tasks that would have required extensive programming effort under classical robotics approaches. It is not general-purpose manipulation. It is a meaningful, honest step toward it.
The Data Flywheel Bet
Covariant's approach to the same problem prioritizes real-world scale over architectural novelty. Their RFM-1 (Robotics Foundation Model) is trained on hundreds of millions of real picks across their deployed fleet — data that no academic lab and no simulation environment can replicate. The bet is that real-world experience at scale, not synthetic diversity or architectural cleverness, is what ultimately produces reliable, generalizable robotic behavior. There's empirical support for this view: the models that have generalized best in language have generally been the ones trained on the most diverse, highest-quality real data, not the ones with the most novel architecture.
Skild AI, the Pittsburgh-based startup founded by Deepak Pathak and Abhinav Gupta, is building foundation models with a particular emphasis on physical grounding — training on diverse robotic experience data specifically to produce models that understand physical object properties and interaction dynamics, not just appearance statistics. The research insight is that generalization in the physical world requires representations of mass, friction, compliance, and geometry that are not encodable in pixel or token statistics. Getting these physical priors into the model requires specific training regimes and data collection strategies that are different from what works in the language domain.
Simulation's Specific Role
For robot foundation models, synthetic data from simulation has a specific and important role that is different from its role in perception. The goal is not photorealism but physical diversity: pre-training a model on a wide range of simulated physics and object interactions so that it develops representations of the physical world that transfer to real deployment, even when the simulation looks nothing like reality. This is the sim-to-real insight applied to learning rather than perception.
The companies that figure out how to pre-train on simulated physical diversity and then fine-tune efficiently on small amounts of real demonstration data — achieving the data efficiency that makes real-world collection feasible at scale — will have a durable advantage. Every deployment contributes real data. The flywheel compounds across the fleet. The gap between a company with one hundred deployed robots and one with one thousand is not linear — it's the gap between a training dataset and a flywheel.
The Investment Angle
Foundation models for robots are probably three to five years away from being reliable enough for most unstructured industrial applications. The companies that will benefit most from that transition are being built right now — and the most interesting ones are not necessarily the model companies themselves. They're the companies building the infrastructure that robot foundation models require: data collection platforms, high-fidelity physics simulation environments, evaluation frameworks for robotic generalization, and the deployment tooling that takes a trained model from a research environment into a production system.
The LLM analogy is useful as a framing device but dangerous as an investment map. The companies that understand exactly where the analogy holds and where it breaks down — and build for the specific requirements of embodied intelligence rather than borrowed assumptions — are the ones I'm watching most closely.