The Agent Authors the Skill. Physics Runs It.

Changlin Song · Australian National University · with GE and the University of Pennsylvania · ongoing work, November 2026

We want a legged robot that picks up new whole-body tasks quickly and works outside the lab. An LLM agent is too slow to be the control loop, so it writes the control program instead: a reward from a sentence and a few rounds of feedback. A physics-based sampling MPC then runs that reward 20 times a second on a Boston Dynamics Spot, with no demonstrations, no per-task training and no motion capture.

Real robot, outdoorsRolling a jug to a clicked goal on an agent-written reward · 1× speed
The agent writes what "good" means
Spot body camera view with the tracked jug outlined.
The robot sees the object itself
Physics tries 24 futures, 20 times a second

How It Works

A slow loop where the agent writes, and a fast loop that never waits for it.

my componentsfrom Sumo, deployed by me

End-to-end VLA

A new task needs new demonstrations

Teleoperate, then fine-tune.

One RL policy per skill

A new task needs hours of training

Write a reward, then train in simulation.

This project

A new task needs a sentence and a few rounds of feedback

The agent writes the reward and the MPC runs it at once.

A Pipeline That Leaves the Lab

Spot's own cameras replace motion capture, so the planner's simulator stays in sync with the world outdoors. I built the deployment end to end.

Illustration: external optical motion capture covers only a calibrated volume; robot-mounted egocentric perception moves with the robot.
Illustration (AI-generated): capture coverage stays with the room; onboard perception moves with the robot.
Six onboard cameras fused into one RGB-D scene.
Multi-view FoundationPose: shared pose candidates scored in every camera and aggregated.
Multi-view 6-DoF tracking with SAM 3 and FoundationPose.
my componentsexisting models and hardware, integrated by me

Object position bias fed to the planner

World frame, SLAM in loop6.3 cm
Robot frame (used)0.64 cm
Dashed band: the 2–3 cm bias the MPC tolerates. One HOT3D test clip.
The full run with the operator console (left) and the robot (right). Bystanders are blurred.

A Skill Is a Reward the Agent Writes

The agent writes and debugs the reward. A person reviews it in plain words. Each revision is tested by rerunning the MPC, with no retraining.

    9rounds of feedback in words
    116 kgbarrel with half-fill water load
    0.25 mfinal distance to goal
    1 seedsimulation, segmented run

    What Should the Agent Own?

    In simulation I handed each stage of the pipeline to an LLM agent. It is good at writing programs that run in the loop, and inefficient at being the loop.


    Agent does well

    Writes and fixes the reward

    Real jug roll; 116 kg barrel in simulation.

    Agent does well

    Writes its own control code for the object, with or without MPC tools

    64 of 72 runs: pushes 32/32, uprights 32/40.

    Agent does well

    Hands hard contact to MPC when MPC is offered as a tool

    Better left to a tool

    Being the fast loop

    LLM commands at 1 Hz: 8 of 30. MPC at 20 Hz: 28 of 30. The same MPC at 1 Hz: 16 of 30.

    Better left to a tool

    Walking from raw joint commands

    Only the newest models manage it, on some tasks. The 50 Hz policy already does it.

    LLM issues commands, 1 HzSumo MPC, 1 Hz
    Same rate, same 71 start states: 19 vs 34 successes.
    Three clocks (log scale): an LLM call takes seconds, MPC replans every 50 ms.

    Outlook: a Data Engine

    Not a result yet. Skills the agent authors and MPC executes produce robot data with no human demonstrations, which can later train RL or VLA policies that run fully onboard.


    Baseline

    Sumo

    Sampling MPC over an RL policy; hand-written rewards, motion capture.

    Spot rolling a water jug on grass

    Let's Connect

    This is ongoing work. If you are interested in agents for robot control, or in whole-body manipulation outside the lab, I would be glad to talk.

    Contact
    Built on Sumo (Zhang et al., 2026) and its Judo MPC framework, FoundationPose, and SAM 3. Thanks to collaborators at GE and the University of Pennsylvania.