Two agent modes
Multimodal models can already control a robot without special training. The question is how they compare with policies trained specifically on manipulation, and how best to combine the two. The authors tested this on the EBench simulation benchmark: 26 tasks and 510 test episodes of varying length and precision.
- Direct control. The agent itself outputs the waypoints of the robot's motion.
- Agent plus policy. The agent can temporarily hand control to a trained policy — a VLA or a world action model — and then take it back.
Direct control
On its own the agent solved 46.7% of tasks and came third among nine participants. Its profile is uneven: on short tasks it is the best — 73.2% against 65.9% for the strongest policy, OpenWAM-α. On tasks that need high precision and long contact with an object it fails: 11.3% against 32.5%.
Agent plus policy
Authors' data. Each pair beats both the agent and its own policy taken separately.
The pairing had one more effect: the agent spends fewer tokens per successful task — 3.5–4.0 million instead of 8.1 million, i.e. 51–57% less. That does not mean less compute overall: the policy adds its own 45–135 runs per episode.
Where the result is lost
The gain is not free. The pair with InternW0-Δ won 96 episodes that the agent alone failed, but lost 49 of those it used to solve. The authors see three reasons.
- Blind delegation. How often the agent hands over control barely depends on how strong the policy is at the task.
- No subgoals. The policy gets the general task instruction, not a specific next step from the agent.
- Failure on return. After taking control back, the agent sometimes drops an object the policy was already holding.
What it means in practice
The same principle works far beyond robotics: a corporate LLM orchestrator agent calls specialized models — for image, document and speech recognition. The gain comes from decisions about when and with which subtask to hand work over, and from a clean return of the result. Limitation of the study: one model and simulation only; the authors left real robots for future work.