Changlin Song · Australian National University · with GE and the University of Pennsylvania · ongoing work, November 2026
We want a legged robot that picks up new whole-body tasks quickly and works outside the lab. An LLM agent is too slow to be the control loop, so it writes the control program instead: a reward from a sentence and a few rounds of feedback. A physics-based sampling MPC then runs that reward 20 times a second on a Boston Dynamics Spot, with no demonstrations, no per-task training and no motion capture.
Real robot, outdoorsRolling a jug to a clicked goal on an agent-written reward · 1× speed
The agent writes what "good" means
The robot sees the object itself
Physics tries 24 futures, 20 times a second
How It Works
A slow loop where the agent writes, and a fast loop that never waits for it.
my componentsfrom Sumo, deployed by me
End-to-end VLA
A new task needs new demonstrations
Teleoperate, then fine-tune.
One RL policy per skill
A new task needs hours of training
Write a reward, then train in simulation.
This project
A new task needs a sentence and a few rounds of feedback
The agent writes the reward and the MPC runs it at once.
A Pipeline That Leaves the Lab
Spot's own cameras replace motion capture, so the planner's simulator stays in sync with the world outdoors. I built the deployment end to end.
Illustration (AI-generated): capture coverage stays with the room; onboard perception moves with the robot.
Six onboard cameras fused into one RGB-D scene.
Multi-view 6-DoF tracking with SAM 3 and FoundationPose.
my componentsexisting models and hardware, integrated by me
Object position bias fed to the planner
World frame, SLAM in loop6.3 cm
Robot frame (used)0.64 cm
Dashed band: the 2–3 cm bias the MPC tolerates. One HOT3D test clip.The full run with the operator console (left) and the robot (right). Bystanders are blurred.
A Skill Is a Reward the Agent Writes
The agent writes and debugs the reward. A person reviews it in plain words. Each revision is tested by rerunning the MPC, with no retraining.
9rounds of feedback in words
116 kgbarrel with half-fill water load
0.25 mfinal distance to goal
1 seedsimulation, segmented run
What Should the Agent Own?
In simulation I handed each stage of the pipeline to an LLM agent. It is good at writing programs that run in the loop, and inefficient at being the loop.
Agent does well
Writes and fixes the reward
Real jug roll; 116 kg barrel in simulation.
Agent does well
Writes its own control code for the object, with or without MPC tools
64 of 72 runs: pushes 32/32, uprights 32/40.
Agent does well
Hands hard contact to MPC when MPC is offered as a tool
Better left to a tool
Being the fast loop
LLM commands at 1 Hz: 8 of 30. MPC at 20 Hz: 28 of 30. The same MPC at 1 Hz: 16 of 30.
Better left to a tool
Walking from raw joint commands
Only the newest models manage it, on some tasks. The 50 Hz policy already does it.
LLM issues commands, 1 HzSumo MPC, 1 Hz
Same rate, same 71 start states: 19 vs 34 successes.
Three clocks (log scale): an LLM call takes seconds, MPC replans every 50 ms.
Outlook: a Data Engine
Not a result yet. Skills the agent authors and MPC executes produce robot data with no human demonstrations, which can later train RL or VLA policies that run fully onboard.
"No training" means per task: the whole-body policy is a pretrained RL policy from Sumo, and the object needs a known mesh.
Perception and MPC run on an off-board workstation; only the whole-body policy runs on the robot.
Barrel push: one seed, segmented simulation, uncalibrated water dynamics. Agent study: simulation with privileged state; agent runs had no time limit.
Built on Sumo (Zhang et al., 2026) and its Judo MPC framework, FoundationPose, and SAM 3. Thanks to collaborators at GE and the University of Pennsylvania.