OpenVLA: An Open-Source Vision-Language-Action Model

基于互联网规模视觉语言数据和多样化机器人示范进行预训练的大模型，有可能改变机器人学习新技能的方式：我们不再需要从零开始训练新行为，而是可以通过微调 vision-language-action（VLA）模型来获得更稳健、更具泛化性的视觉运动控制策略。然而，VLAs 在机器人领域的广泛采用一直受限于两个问题：其一，现有 VLA 大多是闭源的，公众难以访问；其二，已有工作没有系统探索如何高效微调 VLA 以适配新任务，而这恰恰是实际采用的关键。为了解决这两个问题，作者提出 OpenVLA，一个 70 亿参数、开源的 VLA 模型，训练数据来自 97 万条真实机器人示范。OpenVLA 以 Llama 2 为语言骨干，并结合融合 DINOv2 与 SigLIP 特征的视觉编码器。得益于更丰富的数据和新的模型组件，OpenVLA 在通用操作任务上表现强劲：在跨 29 个任务和多个机器人平台的评测中，它以少 7 倍参数量的代价，在绝对任务成功率上比闭源模型 RT-2-X（55B）高出 16.5%。作者进一步展示，OpenVLA 可以有效微调到新机器人设置上，特别是在多对象、多任务和强语言 grounding 环境中具有较强泛化能力，并比从零训练的 Diffusion Policy 高出 20.4%。此外，作者还讨论了计算效率：OpenVLA 可以通过低秩适配方法在消费级 GPU 上完成微调，并通过量化高效部署而几乎不损失下游成功率。最后，作者开源了模型权重、微调 notebook 和支持大规模 Open X-Embodiment 数据训练的 PyTorch 代码库。

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.