SOURCE-LINKED INTELLIGENCE

Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers

arXiv · AI, language, vision and robotics · article · Sep 9, 2026 · UTC

Generated hand-object interaction (HOI) videos provide a controllable way to propose manipulation motions. Simulation-based HOI tracking can translate such kinematic references into feasible low-level control, but its scalability is limited by the lack of reliable reference motions. We therefore combine generated videos with simulation-based HOI grounding: during training, generated videos provide diverse motion references for learning a multi-object, multi-trajectory HOI tracker, and at deployment, the video model produces motion plans that are executed by the learned tracker. In particular,

Read original source ↗ Open in workspace

recordType

paper

region

Global