ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

Overview

ConTrack is a reinforcement learning framework that turns contact-rich human hand-object demonstrations into robot motions.

• Human and robot hands differ in shape and joints, so copying the exact hand pose is the wrong target.
• What the demonstration really specifies is how the object moves.
• ConTrack tracks the object motion as the task and lets the robot hand deviate from the human hand to find its own way to complete the task.

Video

Results

We test ConTrack on bimanual object manipulation (GRAB Dataset), articulated tool use (ARCTIC Dataset), and in-hand rotation (DexterHand Dataset). Each video below is one policy per clip, trained with the same recipe and shown on both the XHand and Sharpa Wave robot hands.

Real Robot Videos

Simulation Videos

Method

ConTrack uses object motion as the task target, then spends the remaining learning capacity on hand motion and contact style.

• Track the object, leave the hand free ConTrack keeps the object close to the reference while allowing the robot hand to use a motion that fits its own kinematics.
• Balance the task-style trade-off online ConTrack uses a scalar controller that compares current object-tracking return with the best return reached on the clip. When tracking falls below the target ratio, training shifts weight to the object; after recovery, capacity moves back to style.
• Practice from reachable failures ConTrack samples resets near early failure boundaries, so the policy practices reachable contact phases more often. During training, the recent failure boundary gradually moves earlier to the start.
• Shape the contact pattern Many robot trajectories can support the same object motion, and not all of them touch the object well. ConTrack uses reference contact events and object-local contact points to encourage more natural contact-rich motion.

BibTeX

@misc{liang2026contrack,
      title={ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control}, 
      author={Liang, Yutong and Peng, Quanquan and Qiu, Ri-Zhao and Wang, Xiaolong},
      year={2026},
      eprint={2606.03177},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2606.03177}, 
}