💻 Code | 📊 Information | 🤖 Fashions | 📖 Documentation
Robots are studying to fold laundry, kind components, and deal with unfamiliar objects. However these demonstrations are sometimes reasonably slender and do not clarify what a robotic has discovered or why it fails.
To review a posh system, scientists typically start with a small, managed surroundings, equivalent to a petri dish or terrarium, the place they’ll fluctuate situations and observe the consequences. Robotics wants environments that make this sort of investigation potential—researchers ought to have the ability to change the coaching information or mannequin design, repeat a check, and hint the reason for a failure.
At present we’re introducing a framework for creating these environments; we name them FoundationWorlds. Every pairs a managed, simulated world focused at learning robotics in a particular surroundings with instruments for robotically producing duties and coaching demonstrations, in addition to metrics for measuring a robotic’s conduct over time. Collectively, they supply a setting for learning questions equivalent to how fashions enhance as extra coaching information turns into accessible, and find out how to greatest mix high-level reasoning with exact bodily management.
Our first basis world is ToyBlocks: an open simulation framework constructed round a robotic arm and blocks on a tabletop. The setting is intentionally easy, however the robotic should nonetheless cause about how the blocks work together and find out how to organize them whereas pushing, flipping, and stacking.
ToyBlocks robotically generates demonstrations, that are recordings of a robotic finishing duties in simulation, with out requiring time-consuming teleoperation, by which an individual remotely controls the robotic. Researchers can simply fluctuate these coaching experiences. For instance, a group may generate extra pushing or flipping demonstrations, then consider efficiency on duties saved out of coaching.
Beginning with ToyBlocks as our first FoundationWorld, we intention to increase the method to more and more advanced environments.
How ToyBlocks works
Constructing on our earlier work in open robotic studying, together with MolmoAct 2, MolmoBot, and MolmoSpaces, ToyBlocks provides researchers a strategy to research robotics questions in a intentionally constrained area.
As a FoundationWorld, ToyBlocks is designed round 4 necessities: (1) sensible physics and look, (2) computerized technology of duties and demonstrations at scale, (3) challenges that mix reasoning with dexterity, and (4) broad protection of the duties potential inside a website.
Open-ended domains require modeling a variety of objects and interactions, making it infeasible to meet all these necessities. ToyBlocks’ slender scope, however, makes it potential to meet all of them.
Inside ToyBlocks’ smaller setting, researchers can work towards investigating how a robotic can study to carry out any affordable job whereas being assured that the simulated arm and blocks behave very equally to their bodily counterparts.
A bodily robotic gripper above picket blocks within the ToyBlocks setup.
ToyBlocks is not only about choosing and stacking blocks: photos and letters on the blocks’ faces permit for looking, sorting, and arranging duties. A robotic might have to show a block to see a hidden face earlier than deciding the place it belongs. Area randomization – various block measurement, desk peak, and different bodily properties from scene to scene – assessments how nicely fashions generalize because the geometry modifications.
ToyBlocks robotically generates expertise (behaviors for brief duties) and job demonstrations that can be utilized to coach fashions or adapt beforehand educated fashions by extra coaching (fine-tuning). Excessive-level reasoning fashions resolve what to do; low-level motion fashions management the robotic’s actions. The framework will also be used to simply design experiments and validate hypotheses.
ToyBlocks makes use of reinforcement studying (RL) and movement planning, which computes how the robotic ought to transfer, to generate expertise for short-horizon duties, which contain transient sequences of actions. Ability coaching takes place in Isaac Lab, Nvidia’s open-source framework for robotic studying in simulation. By trial and error, these insurance policies – fashions that select the robotic’s actions – study non-prehensile expertise equivalent to pushing a block throughout the desk or flipping it onto one other face with out choosing it up. Ability technology can draw on privileged data – values the simulator can provide instantly however an actual robotic can not, equivalent to a block’s actual place and orientation – as a substitute of estimating it from digital camera photos. This allows the automated technology of expertise that may require teleoperation in the true world.
To generate demonstration information, a task-and-motion planner (TAMP) robotically selects and sequences the abilities wanted to unravel a job and plans the robotic’s actions between them. Operating every plan in simulation produces a dataset of recorded demonstrations together with labels describing which job or ability the robotic performs. ToyBlocks scales information technology by producing and executing tons of of such plans in parallel on a single graphics processing unit (GPU), reaching as much as 24 instances real-time efficiency.
Appearing from sensor observations provides a notion downside: a mannequin has to estimate a block’s pose – its place and orientation – which the simulator may provide instantly. To mirror real-world situations, ToyBlocks trains fashions that map uncooked sensor observations to actions.
Execution of Molmo 2 (a high-level mannequin) and ManiFlow (a low-level mannequin) educated in ToyBlocks. Molmo 2 selects expertise equivalent to rotation and flipping, indicating the related block through a pointing interface, whereas ManiFlow controls the simulated arm to execute the commanded expertise.
ToyBlocks’ controllable information generator lets customers create coaching mixtures that emphasize explicit expertise or fill gaps in current datasets. Researchers can research how a lot information a mannequin wants and which experiences assist it generalize to unseen duties. For instance, they might check whether or not including rotation and pick-and-place demonstrations improves flipping, or fluctuate the steadiness of examples inside and out of doors an outlined workspace area—evaluating their decisions on duties saved out of coaching. Importantly, analysis of educated fashions generates metrics past success charges; it supplies data equivalent to execution speeds, forces exerted on objects, and job progress measured through the identical specs utilized by the planning module.
A webpage interface lets customers join Claude – or one other frontier mannequin – and use pure language to instruct the robotic or attempt duties, providing a strategy to discover how a mannequin plans and acts without having robotics experience.
Analysis constructing on ToyBlocks
The evaluations under use ToyBlocks to check how coaching information and strategies have an effect on efficiency, spatial reasoning, generalization to new letters and block appearances, multi-step planning, switch to bodily robots, and frontier fashions that act by instruments. Customers can simply design their very own duties and measures to analyze different questions—for instance, whether or not a mannequin performs higher when it selects objects by pointing or by issuing textual content instructions equivalent to “choose the yellow block.”
Low-level motion fashions: VLA spatial generalization
Low-level motion fashions, equivalent to vision-language-action fashions (VLAs), map language directions and visible observations to robotic actions.
The premise of VLAs is that, by pretraining on large-scale datasets, they’ll quickly generalize to totally different eventualities or a particular workspace. Nevertheless, proof on how nicely these motion fashions generalize stays conflicting. Some research report that these motion fashions are brittle to modifications in inputs, equivalent to totally different object layouts and lighting, whereas others present that VLAs generalize nicely. We argue that these research typically depend on real-world datasets with restricted flexibility in spatial variations. ToyBlocks permits us to check how nicely motion fashions generalize in a easy, simply reproducible setting with significant task-space range.
We design generalization assessments that change the ability sequences wanted to finish an instruction, the letters or colours on the blocks, and the variety of blocks in a scene. A stacking research can withhold four-block stacks from coaching, then check whether or not a mannequin can mix acquainted actions to construct them. Workspace research distinguish efficiency at new positions inside a coaching area from efficiency past its boundaries, the place the mannequin has to extrapolate. We illustrate the research setup and findings under.
We take into account 4 duties: Stack, Unstack, Flip, and Rotate. For these duties, we research interpolation (new configurations throughout the coaching vary) and extrapolation (configurations past that vary). For instance, we consider generalization by coaching on eventualities the place objects spawn solely within the left area (inexperienced) and testing on the correct area (orange), or by coaching on stack heights of two and 5 blocks and testing on heights 3 and 4.
Spatial generalization research for VLA fashions (outcomes based mostly on ManiFlow, π0.5, and MolmoAct 2). Regardless of large-scale pretraining, these fashionable motion fashions generalize poorly within the preliminary outcomes. We enhance generalization and analyze the contribution of every method. The chart’s LoRA label stands for low-rank adaptation, a technique for fine-tuning a mannequin with a small set of trainable weight updates. We imagine motion mannequin generalization stays removed from solved; ToyBlocks supplies a priceless testbed for systematically exploring generalization methods.
Excessive-level reasoning: vision-language fashions
Lengthy-horizon duties, which require many steps, permit us to measure how nicely fashions can plan a sequence of actions to unravel them. Our framework permits us to create such long-horizon duties, and create oracles – reference parts that present right plans or actions – at each the excessive and low ranges. This permits us to isolate whether or not the high-level planning or low-level execution failed when a mannequin fails at a job. An instance long-horizon job is likely to be to spell “Ai2” with a block stack; as a result of the letters might not be instantly seen, the agent should first select actions that reveal data, after which plan and execute a sequence of actions to construct the stack.
We consider hierarchical fashions with express high-level planning and low-level execution fashions. On this occasion, we select Molmo 2, a vision-language mannequin (VLM), because the high-level planner, and ManiFlow because the low-level coverage. Molmo 2 chooses the ability to execute and the block to execute it on, whereas the low-level mannequin executes the instruction. We prepare low-level fashions equivalent to ManiFlow to pause after finishing the execution and question the high-level mannequin for the following instruction. We consider the mannequin’s planning talents on three eventualities of accelerating issue. The simplest state of affairs initializes with every character upright and going through the digital camera. The medium state of affairs has characters that might not be upright, and the toughest state of affairs incorporates characters that aren’t seen initially of the episode.
Success charges on three-letter block-spelling duties utilizing Molmo 2 because the high-level reasoning mannequin and both ManiFlow or a movement planner for low-level manipulation. An instance job is “Stack Ai2”. Molmo 2 and ManiFlow are educated on ToyBlocks and evaluated throughout three issue settings and two generalization situations.
Excessive-level reasoning: agentic methods
Our experiments with agentic methods check how frontier fashions plan and act by instrument calls. We research three settings utilizing Claude Opus 5.5. Within the search-and-spell setting, we expose ToyBlocks expertise by the Mannequin Context Protocol (MCP), a regular for connecting AI purposes to instruments. The agent sees the scene by digital camera photos, calls expertise equivalent to choose, flip, or rotate by pointing at a pixel, and decides when to go looking, construct, or declare the duty full or unattainable. The identical MCP instruments can execute both the scripted expertise that generate ToyBlocks demonstrations (motion-planned choose and place, plus RL-based rotate and flip) or discovered ManiFlow expertise. ToyBlocks consists of the framework behind these experiments, which additionally powers the natural-language internet interface described earlier.
A frontier mannequin (Claude Fable 5.1) controls a robotic in a ToyBlocks world to seek for and stack blocks right into a tower spelling “Ai2.” Through the episode, the agent muses, “I want to determine whether or not pick-and-place naturally straightens orientation or if I must right it manually… I’ve 63 calls left, lots to run a managed check” (0:06). The agent later hypothesizes: “…possibly the gripper’s yaw is radial—pointing from the robotic base towards the block” (0:15) and ultimately confirms this speculation (0:26) earlier than setting up the three-block tower, however the ‘A’ is sideways (1:11). Surprisingly, present frontier fashions typically make comparable perceptual errors (see the following determine), regardless of their spectacular planning, spatial reasoning, and search capabilities.
For classical planning, the agent (our system utilizing Claude Opus 5.5 to plan and select actions) solves all 30 of a gaggle of block-stacking duties just like ones that frontier fashions struggled with a couple of years in the past, utilizing textual content descriptions. It additionally solves 30 out of 30 whereas repeatedly observing digital camera photos and selecting its subsequent motion in a closed loop.
Within the search-and-spell setting, the agent needed to generate stacks spelling out three-letter phrases. It needed to discover the blocks with the corresponding letters amongst as much as eight blocks within the scene. Importantly, 50% of the scenes didn’t have blocks with all of the required letters, rendering the duty infeasible. On this case, we required the agent to state that the duty is infeasible. We discovered that on this experiment, the agent was in a position to efficiently remedy or appropriately declare infeasible 63% of the duties. Errors had been largely attributable to the agent complicated letters with look-alike letters (13%) or selecting an incorrect orientation for a letter in a stack (15%).
Our outcomes additionally present that the agent was in a position to remedy the duty extra effectively when the blocks had semantic regularities, or patterns, benefiting from such patterns to cause about occluded faces. For example, in a lot of the infeasible scenes, when the blocks had no patterns, the agent appropriately checked out all of the blocks’ faces earlier than declaring infeasibility, whereas it was in a position to declare infeasibility considerably quicker when the blocks had semantic patterns.
Our third experimental setting asks whether or not an agent can find out how its expertise behave, utilizing experiments as a scientist would. When a ability behaves unexpectedly, the agent kinds a speculation, designs a small check, and revises its understanding from the consequence. We increase on this concept by having an agent run its personal experiments and write its personal ability descriptions based mostly on what it finds. These discovered descriptions increase success charges from 24% with the default descriptions to 63%, outperforming the 36% achieved with descriptions written by a human knowledgeable.
Share of letter-block faces the agent had uncovered by the tip of every “search and spell” episode (bars averaged over 20 episodes; one marker per episode, styled by end result). In a possible job each letter of the three-letter goal phrase is current; in an infeasible job, one letter is lacking and the agent ought to say so. Every patterned block carries uppercase and lowercase cases of a letter, plus photos that start with the letter (e.g., A-block: apple, ant, A, 1, airplane, a); an unpatterned block’s six faces are randomized and unrelated. Appropriate: the tower spelled the phrase, or for infeasible duties, the agent appropriately declared the duty unattainable. Letter rotated or look-alike: the agent constructed the tower, however a letter was turned or a look-alike substituted, together with 5 infeasible duties declared solved with a look-alike within the lacking letter’s place. Block fell out of attain; reported: a wanted block was pushed out of the arm’s attain, so the agent reported that its job had grow to be unattainable. No report: the agent used up its motor-skill price range and stopped with out claiming both end result. With patterned blocks, the agent finishes having seen fewer faces, as a result of it might infer what hidden faces should depict reasonably than checking each one.
Motion-conditioned world fashions as simulators
Motion-conditioned world fashions are educated to foretell how a scene modifications in response to a robotic’s actions. Utilizing them as an alternative choice to model-based simulators for coaching and evaluating robotic insurance policies is an energetic space of analysis. How nicely they serve this function depends upon how precisely they predict bodily interactions, which is difficult to measure with out realizing the true state of the scene. ToyBlocks supplies this floor reality – the precise simulated scene state – enabling extremely managed assessments of how lately developed world fashions carry out in low-level object manipulation. Right here, the arm’s finish effector is its gripper.
Particularly, we examined whether or not action-conditioned world fashions are able to predicting the motions of blocks or the outcomes of ability executions after being educated on instance executions. To take action, we selected 4 totally different expertise (pick-and-place, flip, rotate, and a combined job) and generated 900 trajectories – sequences of robotic states and actions over time – for every ability. 50% of those trajectories had been profitable demonstrations of the ability, and 50% had been unsuccessful due to small modifications to spatial configurations. With equal numbers of profitable and unsuccessful executions, a mannequin that at all times predicts success can be right solely half the time. The fashions had been fine-tuned on these demonstrations after which examined on their potential to foretell the result of motion sequences of the identical expertise.
We discover that Wan 2.2 predicts block movement and job outcomes most precisely. Astra tracks the gripper nicely however struggles to simulate contact. LeWM is quick and extra correct over quick interplay segments, although it drifts over full trajectories. In extra experiments, we additionally confirmed that fashions that rating greater on pose and success prediction additionally produce higher coaching information: including 100 model-generated trajectories (rollouts) from the very best video-generation mannequin raises a coverage’s success in an unseen area from 19% to 78%.
We examined three varieties of fashions: a video prediction mannequin with 5 billion parameters (Wan 2.2), an agentic mannequin that created its personal physics simulator tuned to match the demonstration information (Astra), and a latent world mannequin (LeWM). Throughout testing, every educated mannequin has to foretell future photos, simulator states, or latent states (discovered inner representations), respectively, based mostly on an preliminary picture and a sequence of actions. For full trajectories, fashions use the preliminary picture as the one noticed picture, together with the motion sequence. For brief interplay segments – a single choose, place, flip, or rotate – fashions begin from the true picture instantly earlier than that interplay. Full trajectories present how errors accumulate over lengthy horizons, whereas interplay segments isolate a single interplay, such because the gripper making contact with a block, and present how nicely the mannequin predicts it. To check fashions pretty, we probe every mannequin’s predictions, studying out the place it expects the blocks and gripper to be at totally different factors within the trajectory and whether or not it expects the duty to succeed on the finish.
Actual-world experiments with high- and low-level fashions
Sim-to-real evaluations check insurance policies educated in simulation on bodily robots performing block manipulation duties. By evaluating simulation and real-world execution, we assess whether or not behaviors and failure patterns noticed in simulation present significant indications of real-world efficiency. For instance, researchers can consider particular person expertise and their composition in longer-horizon duties to determine the place failures come up and whether or not these patterns carry over to bodily deployments.
We check whether or not the fashions educated completely on ToyBlocks simulation datasets might be deployed in the true world zero-shot, with out coaching on real-world examples. We consider ManiFlow and every low-level ability (choose, place, flip, and rotate) individually. We check whether or not the hierarchical system (Molmo 2 + ManiFlow) can full long-horizon three-block spelling duties requiring sequences of those actions. Actual-world experiments use a Franka FR3 arm with a Robotiq gripper throughout thirty scene configurations. Every scene incorporates 4–10 blocks seen throughout coaching. An episode is profitable provided that the related blocks are moved to focus on poses with out collisions. If the operator intervenes to cease the robotic as a consequence of an unsafe movement throughout an episode, it’s marked as a failure.
ManiFlow’s atomic-skill success charges had been comparable in simulation and the true world (80.0% and 81.1%). The hierarchical Molmo 2 + ManiFlow system succeeded in 55.5% of simulated episodes (111/200) and 50.0% of real-world episodes (15/30); additional trials are wanted to ascertain statistical confidence throughout duties.
Success charges of the ManiFlow low-level mannequin and the hierarchical Molmo 2–ManiFlow system in simulation and the true world. For ManiFlow, the overall rely represents particular person ability executions (choose, place, flip, and rotate), whereas for Molmo 2–ManiFlow, it represents block-spelling job episodes. The Molmo 2–ManiFlow evaluations use the simplest scene configuration outlined within the simulation analysis, with block characters seen and upright.
Facet-by-side real-world and simulated block-stacking executions utilizing the identical high-level planner. Pink rings mark the planner’s goal level for every ability. The scenes use totally different block labels, and simulation playback is retimed to align grasps and releases with the true robotic.
To switch Molmo 2 and ManiFlow – each educated completely on ToyBlocks information – to a bodily robotic, we tackle the variations between simulation and actuality in three levels.
Earlier than information technology, we carry out system identification by tuning simulated robotic parameters, together with joint friction and armature (the motor and gearbox inertia mirrored on the joint), to match real-robot trajectories. Throughout information technology, we randomize bodily properties (object measurement, friction, and desk peak) and visible properties (lighting, backgrounds, supplies, and digital camera parameters). At inference time, observations are captured utilizing a front-facing ZED digital camera and a wrist-mounted ZED Mini. For insurance policies that use depth photos, we generate clear depth maps, which estimate distance at every pixel, from paired red-green-blue (RGB) digital camera photos utilizing FastFoundationStereo, reasonably than counting on noisy depth sensor measurements.
The bodily ToyBlocks setup: a robotic arm, picket blocks, and a digital camera.
Opening the instruments for additional analysis
ToyBlocks connects information technology, coaching, and analysis in a single managed setting, so researchers can examine how robots study over time and check how modifications to a mannequin or its coaching information have an effect on robotic conduct. We’re releasing simulation belongings, procedural job and demonstration mills, robotically generated demonstrations, two educated fashions showcasing what the framework can do (one based mostly on Molmo 2 and one based mostly on ManiFlow), coaching code, and analysis code and validation units. This permits different groups to construct their very own coaching mixtures and examine failure modes past these lined in our preliminary experiments.
Our preliminary information launch is a proof level with 85,336 long-horizon demonstrations, every comprising about 6–12 ability demonstrations, relying on the set. The preliminary datasets cowl practically 1,600 hours of robotic execution information and 144 million frames. In a couple of weeks, we’re concentrating on a second launch with a larger-scale dataset, extra expertise, and extra complete benchmarking. We’ll additionally add totally built-in RL coaching functionality that’s not contained within the present package deal.
Different instructions for ToyBlocks embody bimanual (two-arm) manipulation and dexterous robotic palms, geared up with power and contact sensors for learning movement and duties that contain frequent bodily contact. We’re additionally exploring play with unstructured piles of blocks as a strategy to accumulate information and research how new behaviors and duties would possibly emerge by interplay.
Future route: RL-based pile manipulation.
Future route: bimanual block manipulation.
With ToyBlocks, we need to advance the science of robotic studying towards a future the place robots can reliably apply what they’ve discovered to unfamiliar duties and environments. We encourage the analysis neighborhood to discover ToyBlocks for their very own investigations and at the same time as a foundation to construct their very own FoundationWorlds.
