ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

作者提出 ADEPT，一个用于高自由度机器人 embodiment 的大规模强化学习框架，目标是让策略直接从原始视觉-触觉感知解决长程灵巧任务，并能仿真到真实迁移。ADEPT 先在通用 object reposing 任务上预训练灵巧策略，再把该行为作为先验后训练下游策略。这样，多指机器人无需在每个新任务中重新发现相同技能，也能学到从零很难出现的新行为。预训练教师可以零样本完成下游任务的 reposing 阶段，但朴素 RL fine-tuning 会很快破坏这项能力；因此论文提出行为克隆蒸馏、critic warm-up 与保守 on-policy update 的稳定后训练配方。为了安全利用完整运动学灵巧性，作者在策略与机器人之间加入 joint-space Geometric Fabric。最终，后训练教师被蒸馏成感知学生，分别在 23 DoF Kuka-Allegro 双 RGB 相机平台和 29 DoF Flexiv-Sharpa 双 RGB + 五视觉触觉传感器平台上零-shot 迁移，并能以接近人类速度从困难初始状态完成长程任务。

We introduce Accelerating Dexterity via Pre-Training (ADEPT), a large-scale reinforcement learning (RL) framework for learning sim-to-real transferable dexterity across high degree-of-freedom (DoF) robot embodiments that can solve long-horizon tasks directly from raw visuo-tactile perception. ADEPT pretrains a dexterous policy on a generic object reposing task, then post-trains downstream policies with this pretrained behavior as a prior. ADEPT enables learning new behaviors that are otherwise difficult to discover from scratch on multi-fingered robots and avoids learning the same set of skills over again for every new downstream task. The pretrained policy zero-shots the reposing phase of downstream tasks, but naïve RL fine-tuning rapidly degrades this capability during transfer. We address this with a stable post-training recipe combining behavior-cloning distillation, critic warm-up, and conservative on-policy updates. To safely exploit the full kinematic dexterity, we introduce a joint-space Geometric Fabric that mediates between the RL policy and the robot. We distill post-trained teachers into perceptive students that zero-shot sim-to-real transfer on two embodiments: a 23 DoF Kuka-Allegro with two RGB cameras, and a 29 DoF Flexiv-Sharpa with two RGB cameras and five vision-based tactile sensors, and can solve long-horizon tasks from challenging initial states with dexterity at human-level speed.