What Matters for Latent Actions in Robot Learning

隐式动作模型（LAM）已成为一个有前景的范式：通过「隐式动作」作为物理动作的紧凑代理，让机器人学习能够利用大规模无标注视频。尽管进展迅速，LAM 研究仍高度碎片化——现有方法在不相一致的实验设定下各自评估不同的设计选择，难以识别真正决定下游机器人操纵性能的因素。本文提出首个面向机器人操纵的隐式动作学习系统性实证研究：将代表性 LAM 方法统一到一个共同的自编码框架下，沿着三个维度系统考察 41 个 LAM 设计选择（隐式动作建模范式、学习目标与正则化方法、隐式动作集成策略），并检验四个代理指标评估隐式动作质量的能力及其对下游操纵性能的预测力。在三个广泛使用的基准上的大量实验给出强证据：用隐式动作微调视觉-语言模型（VLM）骨干能为下游策略学习提供更强初始化，并在真实机器人操纵任务中进一步得到验证。

Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-language model (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.