From Variational Inference, Diffusion Models, Singular Perturbations to Reinforcement Learning
Associate Professor, ECE | The Ohio State University
Co-Director, IITB-OSU Frontier Center
Founder, Ensemble Control Inc. (https://ensemblecontrol.com/)
Humanoid robotics moved from research demos to funded industrial programs:
| Lab | Flagship stack | Action decoder | Motor learning |
|---|---|---|---|
| Physical Intelligence | \(\pi_0\), \(\pi_{0.5}\), \(\pi^*_{0.6}\) + Recap | flow matching | BC \(\to\) RL |
| Figure AI | Helix | regression head | BC (500 h teleop) |
| Skild AI | Skild Brain | omni-body policy | large-scale RL + video |
| NVIDIA | GR00T N1 | diffusion transformer | BC: real + synthetic |
| BD \(\times\) TRI | Atlas LBM (450M) | diffusion/flow DiT | BC (whole-body teleop) |
| Google DeepMind | Gemini Robotics | tokenized decoder | BC on VLM backbone |
| 1X | Redwood + 1XWM | VLA (~160M params) | BC + world-model eval |
Thesis of this talk
Strip the branding: the whole field runs on six classical mathematical ideas.
The labs publish under different names — “vision-language-action models,” “large behavior models,” “robot brains,” “world models” — but the machinery reduces to:
This talk is organized around the mathematics, not the companies.
Every frontier stack is the same triple:
\[ \underbrace{h_t=\Phi(o_{t},\ell)}_{\substack{\text{semantic encoder}\\ \text{(vision-language model)}}} \qquad \underbrace{\hat z_t=\phi(x_{t-k:t},\,a_{t-k:t})}_{\substack{\text{adaptation /}\\ \text{estimation module}}} \qquad \underbrace{a_{t:t+H}\sim p_\psi(\cdot\mid h_t,\hat z_t)}_{\substack{\text{generative}\\ \text{action head}}} \]
The three questions that differentiate the labs
(a) Which generative family implements \(p_\psi\)? (b) Imitation or reinforcement? (c) How are the slow and fast modules coupled?
| Symbol | Meaning |
|---|---|
| \(o_t,\ \ell,\ x_t,\ a_t\) | camera observation, language instruction, robot state, motor action |
| \(a_{t:t+H}\) (or just \(a\)) | action chunk: the next \(H\) actions, generated together |
| \(h_t = \Phi(o_t,\ell)\) | semantic latent from the vision-language model |
| \(p_\psi(a \mid h)\) | the learned action distribution (“policy head”) with parameters \(\psi\) |
| \(\pi_\theta(a \mid x)\) | RL policy with parameters \(\theta\) |
| \(\xi,\ z = e(\xi),\ \hat z_t\) | true dynamics parameters, their encoding, the on-line estimate |
| \(\epsilon \sim \mathcal N(0, I)\) | standard Gaussian noise |
| \(\mathrm{KL}(q\,\Vert\,p) = \mathbb{E}_q\!\left[\log \tfrac{q}{p}\right]\) | Kullback–Leibler divergence (always \(\ge 0\); \(=0\) iff \(q = p\)) |
| \(A^\pi(x,a) = Q^\pi(x,a) - V^\pi(x)\) | advantage: how much better action \(a\) is than average |
One convention throughout: \(\psi\) = generative-head parameters, \(\theta\) = RL-policy parameters, \(\phi\) = estimator.
Every frontier system is pretrained on teleoperation data \(\mathcal D = \{(o_i, \ell_i, a_i)\}\) by behavior cloning (BC):
\[ \min_\psi\ \mathbb{E}_{\mathcal D}\bigl[-\log p_\psi(a \mid o, \ell)\bigr] \;=\; \min_\psi\ \mathrm{KL}\bigl(p_{\mathrm{data}}\ \Vert\ p_\psi\bigr) + \text{const} \]
Why the identity holds: \[ \mathrm{KL}(p_{\mathrm{data}} \Vert p_\psi) = \underbrace{\mathbb{E}_{p_{\mathrm{data}}}[\log p_{\mathrm{data}}]}_{\text{entropy: no } \psi} \;-\; \mathbb{E}_{p_{\mathrm{data}}}[\log p_\psi] \]
so minimizing negative log-likelihood \(\equiv\) projecting the demonstrator onto the model family (an M-projection).
Who uses this
Figure trains Helix on \(\sim\!500\) h of teleop exactly this way; Tesla’s Optimus and 1X’s Redwood are the same recipe at different scales. The architectural race is over which family makes this expressive enough.
Demonstrations are multimodal: to pass an obstacle, half the demos go left, half go right.
Unimodal Gaussian head \(\;p_\psi = \mathcal N(\mu_\psi(x), \sigma^2 I)\)
MLE fits the conditional mean: \[ \mu_\psi(x) \to \mathbb{E}[a \mid x] = \tfrac{1}{2}a_{\text{left}} + \tfrac{1}{2}a_{\text{right}} \]
\(\Rightarrow\) the robot drives into the obstacle. The average of two valid actions need not be valid.
Generative head (diffusion / flow / autoregressive)
Represents the full conditional distribution: \[ p_\psi(a \mid x) \approx \tfrac{1}{2}\delta_{a_{\text{left}}} + \tfrac{1}{2}\delta_{a_{\text{right}}} \]
\(\Rightarrow\) samples commit to one coherent mode per rollout.
This single failure mode explains the entire move to generative action heads — and generative heads are trained by variational inference.
Setup. A latent-variable model \(p(a, z) = p(a \mid z)\, p(z)\). The likelihood \(p(a) = \int p(a \mid z)\, p(z)\, \mathrm{d}z\) is intractable.
Idea. Introduce any distribution \(q(z)\) and use Jensen’s inequality (\(\log\) is concave):
\[ \log p(a) = \log \mathbb{E}_{q(z)}\!\left[\frac{p(a, z)}{q(z)}\right] \;\ge\; \underbrace{\mathbb{E}_{q(z)}\!\left[\log \frac{p(a, z)}{q(z)}\right]}_{\textbf{ELBO}(q)} \]
How tight? Exactly the gap to the true posterior:
\[ \log p(a) \;=\; \mathrm{ELBO}(q) \;+\; \mathrm{KL}\bigl(q(z)\ \Vert\ p(z \mid a)\bigr) \]
The variational principle
Maximizing the ELBO (i) pushes up the data likelihood and (ii) pulls \(q\) toward the true posterior — every generative action head in this talk instantiates this bound.
Split the joint \(p(a,z) = p(a\mid z)\,p(z)\) inside the bound:
\[ \mathrm{ELBO}(q) = \underbrace{\mathbb{E}_{q(z)}\bigl[\log p(a \mid z)\bigr]}_{\text{reconstruction}} \;-\; \underbrace{\mathrm{KL}\bigl(q(z)\ \Vert\ p(z)\bigr)}_{\text{regularization toward the prior}} \]
Diffusion is not a separate theory — it is the ELBO applied to a particularly clever latent hierarchy.
Model the action chunk \(a^0 \equiv a_{t:t+H}\). Define latents \(a^1, \dots, a^S\) by gradually destroying \(a^0\) with Gaussian noise:
\[ q(a^s \mid a^{s-1}) = \mathcal N\!\bigl(\sqrt{1-\beta_s}\; a^{s-1},\ \beta_s I\bigr), \qquad s = 1, \dots, S \]
Gaussians compose, so the marginal has closed form. With \(\alpha_s = 1-\beta_s\), \(\bar\alpha_s = \prod_{r\le s}\alpha_r\):
\[ a^s = \sqrt{\bar\alpha_s}\, a^0 + \sqrt{1-\bar\alpha_s}\,\epsilon, \qquad \epsilon \sim \mathcal N(0, I) \]
Key design trick: because \(q\) is fixed by construction, the only thing left to learn is the reverse denoiser.
The noising chain defines a hierarchical latent-variable model; write its ELBO and it telescopes into per-step KL terms between Gaussians. After the standard reparameterization, the bound collapses to a regression:
\[ \min_\psi\ \mathbb{E}_{s,\,a^0,\,\epsilon}\; \Bigl\|\epsilon - \epsilon_\psi\bigl(\underbrace{\sqrt{\bar\alpha_s}\,a^0 + \sqrt{1-\bar\alpha_s}\,\epsilon}_{\text{noised chunk } a^s},\; s,\; h_t\bigr)\Bigr\|^2 \]
“Given a noised action chunk, predict the noise that was added” — conditioned on the semantic latent \(h_t\).
Connection: score matching
The optimal denoiser estimates the score: \(\ \epsilon_\psi \propto -\nabla_a \log q_s(a^s \mid h_t)\). Training the ELBO \(=\) learning the gradient field of the action distribution; sampling integrates the reverse SDE/ODE along it. And \(\nabla\log q\) represents sets of valid grasps/gaits where a regression head averages them.
Who ships it
Why it wins / what it costs
The latency problem motivates the next idea
If diffusion’s reverse chain is too slow for 50 Hz control, straighten the paths: flow matching.
Physical Intelligence’s \(\pi_0\): a 3B PaliGemma VLM + a 300M flow-matching “action expert.”
Draw \(\tau \in [0,1]\), noise \(\epsilon \sim \mathcal N(0,I)\), and connect noise to data by a straight line:
\[ a^\tau = \tau\, a + (1-\tau)\,\epsilon \qquad\Longrightarrow\qquad \frac{\mathrm{d} a^\tau}{\mathrm{d}\tau} = a - \epsilon \]
Train a network to regress this velocity field, conditioned on semantics \(h_t\):
\[ \min_\psi\ \mathbb{E}_{\tau,\,a,\,\epsilon}\; \bigl\| v_\psi(a^\tau, \tau, h_t) - (a - \epsilon) \bigr\|^2 \]
Inference: integrate the ODE \(\dot a^\tau = v_\psi(a^\tau, \tau, h_t)\) from noise \(\epsilon\) at \(\tau{=}0\) to an action chunk at \(\tau{=}1\) — in a handful of steps (\(\sim\!10\times\) fewer evaluations than diffusion).
That factor of 10 is exactly what makes 50 Hz chunked control feasible on-robot.
| Diffusion | Flow matching | |
|---|---|---|
| Latent path noise \(\to\) data | curved (variance-preserving SDE) | straight line |
| Training target | added noise \(\epsilon\) (score) | velocity \(a - \epsilon\) |
| Theoretical frame | ELBO / denoising score matching | continuous normalizing flow / optimal transport |
| Sampling | many-step reverse SDE/ODE | few-step ODE |
| Frontier users | Atlas LBM, GR00T N1 | \(\pi_0\), \(\pi_{0.5}\), Atlas LBM (FM objective) |
Connection: optimal transport
The straight-line interpolant makes the target field the displacement interpolation of optimal transport between the noise and data measures. Flow matching is the OT-geodesic counterpart of the diffusion ELBO — same endpoints, shortest path.
Google DeepMind’s RT-2 / Gemini Robotics line and \(\pi_0\)-FAST take a third route: make actions look like language.
\[ \min_\psi\ \mathbb{E}_{\mathcal D}\Bigl[-\textstyle\sum_j \log p_\psi\bigl(u_j \mid u_{<j},\, h_t\bigr)\Bigr], \qquad u = \mathrm{tokens}(a_{t:t+H}) \]
— exactly behavior cloning MLE again, with an autoregressive categorical generative family.
The trade
Gain: actions share the VLM’s vocabulary \(\Rightarrow\) co-training with internet-scale data. Cost: expressivity is bounded by the tokenizer, and generation loops token-by-token at chunk rate.
1X trains a video world model (1XWM) as a learned simulator; NVIDIA’s GR00T generates synthetic “dream” trajectories. The underlying object is the latent state-space model ELBO over trajectories:
\[ \log p(o_{1:T} \mid a_{1:T}) \;\ge\; \sum_{t=1}^{T}\; \underbrace{\mathbb{E}_{q}\bigl[\log p(o_t \mid z_t)\bigr]}_{\text{reconstruct observations}} \;-\; \underbrace{\mathrm{KL}\bigl(q(z_t \mid z_{t-1}, a_t, o_t)\ \Vert\ p(z_t \mid z_{t-1}, a_{t-1})\bigr)}_{\text{latent dynamics must predict the posterior}} \]
Use case: score policy checkpoints inside the world model instead of physical A/B tests — an estimated off-policy evaluator.
The cleanest estimation-theoretic idea in the field (RMA \(\to\) extreme parkour \(\to\) Skild Brain).
Phase 1 (simulation, privileged). Train a policy conditioned on the true environment/embodiment parameters \(\xi\) — mass, friction, terrain, motor strength — through a low-dimensional encoding \(z = e(\xi)\): \[ \pi\bigl(a \mid x_t,\ z\bigr), \qquad z = e(\xi) \]
Phase 2 (deployable). The robot can’t see \(\xi\). Train an adaptation module to recover \(z\) from what it can measure — its own proprioceptive history:
\[ \hat z_t = \phi\bigl(x_{t-k:t},\ a_{t-k:t}\bigr), \qquad \min_\phi\ \mathbb{E}_{\xi}\,\mathbb{E}_{\text{rollout}}\ \bigl\|\hat z_t - e(\xi)\bigr\|^2 \]
What this is, classically
Textbook on-line system identification, amortized: the regression is a point estimate of the posterior mean \(\mathbb{E}[z \mid \text{history}]\), and the deployed system is a certainty-equivalent adaptive controller re-estimating its own dynamics tens of times per second.
Skild Brain’s omni-body claim — driving quadrupeds, humanoids, and manipulators it has not trained on, adapting within seconds to payload shifts or damaged joints — is the adaptation-module loop scaled up.
Why history suffices
If your commanded action \(a_{t-1}\) and the resulting state \(x_t\) disagree with the nominal model, the discrepancy is informative about \(\xi\). The module \(\phi\) learns the inverse map
\[ (\text{surprise in dynamics}) \longmapsto \hat z_t \]
Why more sensing helps
The estimate’s error is controlled by the observability Gramian of the excited dynamics:
This is the theoretical reason richer proprioception and force sensing keep appearing on frontier hardware.
Every sim-trained locomotion stack (Unitree-class humanoids, Skild, Tesla sim RL) trains under a distribution over dynamics parameters:
\[ \max_\theta\ \mathbb{E}_{\xi \sim p(\xi)}\; \mathbb{E}_{\pi_\theta,\,\xi}\Bigl[\textstyle\sum_t \gamma^t\, r(x_t, a_t;\, \xi)\Bigr] \]
Robust vs. adaptive — the gap the estimator buys
Conditioning the policy on \(\hat z_t\) converts the robust solution (one controller for all \(\xi\)) into an adaptive one (a \(\xi\)-indexed family). The performance gap between the two is precisely the value of the adaptation module.
Residual dynamics (e.g., ASAP): estimate not \(\xi\) but the sim-to-real discrepancy itself — fit \(\Delta_\eta\) so \(f_{\mathrm{sim}}(x,\, a + \Delta_\eta(x,a)) \approx f_{\mathrm{real}}(x,a)\), then keep training RL in the corrected simulator.
Markov decision process: states \(x\), actions \(a\), reward \(r\), discount \(\gamma \in (0,1)\), policy \(\pi_\theta(a \mid x)\).
\[ J(\theta) = \mathbb{E}_{\pi_\theta}\Bigl[\textstyle\sum_{t\ge 0} \gamma^t\, r(x_t, a_t)\Bigr] \]
The three value objects:
\[ V^\pi(x) = \mathbb{E}_\pi\bigl[\text{return} \mid x_0 = x\bigr], \qquad Q^\pi(x,a) = \mathbb{E}_\pi\bigl[\text{return} \mid x_0 = x,\, a_0 = a\bigr] \]
\[ A^\pi(x,a) = Q^\pi(x,a) - V^\pi(x) \quad\text{= "how much better than my average behavior is } a \text{?"} \]
The core tension of policy optimization
The advantage \(A^{\pi_k}\) is only trustworthy near the policy \(\pi_k\) that generated the data. Improve too aggressively and the estimate is garbage; too timidly and learning stalls. Every method on the next slides is a way of formalizing “improve, but stay close.”
The results that made humanoid locomotion credible (Unitree-class pipelines: 4096 parallel simulated environments, 50 Hz policies) rest on PPO, the practical form of the trust-region problem:
\[ \max_\theta\ \mathbb{E}_{\pi_k}\!\Bigl[\frac{\pi_\theta(a \mid x)}{\pi_k(a \mid x)}\, A^{\pi_k}(x,a)\Bigr] \quad\text{s.t.}\quad \mathbb{E}_{x}\ \mathrm{KL}\bigl(\pi_k(\cdot \mid x)\ \Vert\ \pi_\theta(\cdot \mid x)\bigr) \le \delta \]
Next: solving the Lagrangian of this problem in closed form reveals the deeper structure.
Replace the hard constraint by a KL penalty and solve per state, in closed form:
\[ \pi_{k+1}(\cdot \mid x) = \arg\max_\pi\; \mathbb{E}_{\pi}\bigl[A^{\pi_k}(x, \cdot)\bigr] - \tfrac{1}{\eta}\,\mathrm{KL}\bigl(\pi\ \Vert\ \pi_k(\cdot \mid x)\bigr) \]
\[ \Longrightarrow\qquad \boxed{\ \pi_{k+1}(a \mid x)\ \propto\ \pi_k(a \mid x)\; e^{\eta\, A^{\pi_k}(x,a)}\ } \]
the exponentiated-gradient / multiplicative-weights update — mirror descent on the policy simplex with the KL Bregman divergence.
One update, three famous algorithms
We have the ideal update \(\pi_{k+1} \propto \pi_k\, e^{\eta A}\) — but \(\pi_{k+1}\) must live in our parametric family. Project it back by maximum likelihood (an M-projection, like BC):
\[ \min_\theta\ \mathrm{KL}\bigl(\pi_{k+1}\ \Vert\ \pi_\theta\bigr) \;\;=\;\; \min_\theta\ -\,\mathbb{E}_{(x,a)\sim\mathcal D} \Bigl[\, e^{A(x,a)/\beta}\ \log \pi_\theta(a \mid x)\Bigr] + \text{const} \]
Read it as: behavior cloning, where each datapoint is weighted by the exponentiated advantage — clone the good actions hard, the mediocre ones barely.
BC is the special case \(A \equiv \text{const}\): uniform weights. RL fine-tuning = reweighted imitation.
Physical Intelligence’s Recap is the frontier instantiation of the AWR step — engineered for a multimodal flow-matching head, where per-sample loss weights are awkward.
Recipe:
Conditioning + clamping implements the exponential reweighting without touching the loss — the same KL-proximal improvement, robust to the generative head.
Reported effects
\(\sim\!2\times\) throughput, \(\ge 2\times\) failure reduction on long-horizon tasks (espresso, laundry, box assembly) — the first convincing on-robot RL fine-tuning of a large VLA.
How do you make a diffusion policy maximize return, not just imitate? Two techniques dominate:
1. Advantage-weighted diffusion (offline). Put the AWR weight inside the score-matching loss:
\[ \min_\psi\ \mathbb{E}_{(x,a)\sim\mathcal D,\, s,\, \epsilon}\; \Bigl[\, e^{A(x,a)/\beta}\; \bigl\|\epsilon - \epsilon_\psi\bigl(\sqrt{\bar\alpha_s}\,a + \sqrt{1-\bar\alpha_s}\,\epsilon,\ s,\ x\bigr)\bigr\|^2 \Bigr] \]
— a KL-regularized policy-iteration step that concentrates probability mass on high-return actions while preserving multimodality.
2. Q-guided sampling (at inference). Bias the reverse process with a learned critic. Targeting \(p_\psi \cdot e^{Q/\beta}\) and applying Bayes’ rule to the score:
\[ \tilde{\epsilon}_\psi(a^s, s, x) = \epsilon_\psi(a^s, s, x) - \kappa_s\, \nabla_{a} Q\bigl(x,\ \hat a^0(a^s)\bigr) \]
where \(\hat a^0(a^s)\) is the Tweedie estimate of the clean action and \(\kappa_s\) the guidance strength — steer toward higher return at runtime, no retraining.
Teacher–student steps appear everywhere in these stacks:
\[ \min_\theta\ \mathbb{E}_{x}\ \mathrm{KL}\Bigl(\pi_T\bigl(\cdot \mid x_{\mathrm{priv}}\bigr)\ \Big\Vert\ \pi_\theta\bigl(\cdot \mid x_{\mathrm{obs}}\bigr)\Bigr) \]
— the I-projection of a privileged policy onto a deployable information pattern.
| Instance | Teacher (privileged) | Student (deployable) |
|---|---|---|
| RMA phase 2 | policy seeing true \(\xi\) | policy seeing only history |
| Sim teachers | full-state sim policy | vision-based policy |
| NVIDIA HOVER | many whole-body controllers | one unified controller |
| Gemini Robotics on-device | cloud VLM | on-robot model |
| Helix S2 \(\to\) S1 | 7B semantic latent | 80M motor policy |
Conceptually the same object as the adaptation module: compress privileged information into what the robot can actually observe.
Figure Helix
The same pattern everywhere
\[ \underbrace{h_{t+1} = h_t + \varepsilon\, g(h_t, u_t, o_t)}_{\text{slow semantic state}} \qquad\qquad \underbrace{u_{t+1} = f(u_t, h_t, o_t)}_{\text{fast motor state, } f \text{ contracting}} \]
This is exactly the discrete-time singularly perturbed system of classical control theory.
Tikhonov’s quasi-steady-state argument, applied to robot brains:
Two engineering facts this explains
Why chunking works at all: within a chunk the semantic intent is frozen (slow), and the open-loop chunk is stable because low-level tracking (fast) rejects disturbances.
Why raising the S1 rate pays disproportionately: the suboptimality scales with the timescale ratio \(\varepsilon\), not with either rate alone — Figure’s move to 200 Hz shrank \(\varepsilon\) directly, and dexterity gains followed.
| Method (users) | Mathematical frame | Canonical objective |
|---|---|---|
| Behavior cloning (all labs) | max likelihood / M-projection | \(\min_\psi \mathbb{E}[-\log p_\psi(a\mid o,\ell)]\) |
| Diffusion policy (Atlas, GR00T) | hierarchical VI / score matching | \(\mathbb{E}\Vert\epsilon - \epsilon_\psi(a^s, s, h)\Vert^2\) |
| Flow matching (\(\pi_0\), LBM) | normalizing flows / optimal transport | \(\mathbb{E}\Vert v_\psi(a^\tau,\tau,h) - (a-\epsilon)\Vert^2\) |
| Tokenized actions (RT-2, FAST) | autoregressive MLE + rate–distortion | cross-entropy on DCT tokens |
| World models (1XWM, GR00T) | latent state-space ELBO | reconstruction \(-\) KL per step |
| Rapid motor adaptation (Skild) | on-line parameter estimation | \(\mathbb{E}\Vert\hat z_t - e(\xi)\Vert^2\) |
| Domain randomization (sim-RL) | stochastic robust optimization | \(\max_\theta \mathbb{E}_{\xi}\mathbb{E}_{\pi_\theta}[\sum \gamma^t r]\) |
| PPO locomotion (Unitree-class) | mirror descent / natural gradient | KL-constrained surrogate |
| Recap (\(\pi^*_{0.6}\)) | KL-proximal policy iteration | \(\mathbb{E}[e^{A/\beta}\log\pi_\theta]\) |
| Distillation (HOVER, Helix) | I-projection | \(\min\mathrm{KL}(\pi_T \Vert \pi_\theta)\) |
| Dual-system S2/S1 (Helix, GR00T) | singular perturbation | \(O(\varepsilon)\) decoupling loss |
1 — Variational inference won the action head
All competitive decoders are trained as generative models of action chunks — diffusion/flow ELBOs or autoregressive MLE — because demonstrator distributions are multimodal. The remaining competition is over sampling latency: few-step ODE vs. many-step chain vs. token loop.
2 — Estimation, not scale, is what transfers across bodies
Omni-body claims rest on the RMA estimation loop; observability theory (sensor monotonicity, \(O(m^{-1/2})\)) gives the scaling law for why richer proprioception helps.
3 — Mirror descent is the common abstraction of the RL layer
PPO for locomotion, AWR/Recap for VLA fine-tuning, distillation’s KL projections — all KL-proximal iterations. The frontier trend: moving the mirror-descent step from simulation onto the physical robot, with human corrections as off-policy data.
Inference latency for high-frequency control. Iterative denoising vs. 100–200 Hz balancing: consistency models, latent diffusion, and few-step flow matching are the active fronts
Offline-to-online fine-tuning. The score-matching objective does not directly align with expected return; advantage weighting and Q-guidance bridge the gap without collapsing multimodality — but on-robot RL remains sample-limited
Estimation layer \(\not\leftrightarrow\) generative layer. No public stack conditions the VLA action head on the adaptation latent \(\hat z_t\) — though it conditions locomotion policies, and the belief-embedding formalism says they are the same object
Out-of-distribution robustness. Generative heads capture the support of the demos perfectly, then extrapolate unpredictably; blending robust control priors with generative action heads is open
Certified on-robot policy iteration. Recap is a single conservative mirror-descent step; trust-region methods with certified step sizes would make on-robot iteration routine rather than heroic
One architecture: VLM encoder \(\;+\;\) history-based estimator \(\;+\;\) generative action-chunk head
One training principle for the head: maximize a (variational) likelihood — BC is MLE, diffusion is the ELBO of a noising hierarchy, flow matching is its optimal-transport geodesic twin
One estimation principle for transfer: amortized posterior-mean regression over embodiment parameters — adaptive beats robust by exactly the estimation gap
One RL principle: KL-proximal (mirror-descent) policy iteration — PPO, AWR, and advantage conditioning are the same update in three costumes
One systems principle: timescale separation with \(O(\varepsilon)\) guarantees — chunking and dual-system designs are singular perturbation, engineered deliberately
The vocabulary is new; the mathematics is classical. Knowing the six ideas, you can read any humanoid lab’s technical report — and see exactly which knob they turned.
Abhishek Gupta · The Ohio State University & Ensemble Control Inc. · Mathematics of Humanoid Robotics