NIMBLE: A Non-rigid Hand Model with Bones and Muscles

SIGGRAPH 2022


Yuwei Li1,2,3,  Longwen Zhang1,5,  Zesong Qiu1,  Yingwenqi Jiang1,  Nianyi Li4,  Yuexin Ma1,  Yuyao Zhang1,  Lan Xu1,  Jingyi Yu1,6

1School of Information Science and Technology, ShanghaiTech University
2Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences
3University of Chinese Academy of Sciences
4School of Computing, College of Engineering, Clemson University
5Deemos Technology
6DGene Digital Technology


Paper Code Data

Emerging Metaverse applications demand reliable, accurate, and photorealistic reproductions of human hands to perform sophisticated operations as if in the physical world. While real human hand represents one of the most intricate coordination between bones, muscle, tendon, and skin, state-of-the-art techniques unanimously focus on modeling only the skeleton of the hand. In this paper, we present NIMBLE, a novel parametric hand model that includes the missing key components, bringing 3D hand model to a new level of realism. We first annotate muscles, bones and skins on the recent Magnetic Resonance Imaging hand (MRI-Hand) dataset and then register a volumetric template hand onto individual poses and subjects within the dataset. NIMBLE consists of 20 bones as triangular meshes, 7 muscle groups as tetrahedral meshes, and a skin mesh. Via iterative shape registration and parameter learning, it further produces shape blend shapes, pose blend shapes, and a joint regressor. We demonstrate applying NIMBLE to modeling, rendering, and visual inference tasks. By enforcing the inner bones and muscles to match anatomic and kinematic rules, NIMBLE can animate 3D hands to new poses at unprecedented realism. To model the appearance of skin, we further construct a photometric HandStage to acquire high-quality textures and normal maps to model wrinkles and palm print. Finally, NIMBLE also benefits learning-based hand pose and shape estimation by either synthesizing rich data or acting directly as a differentiable layer in the inference network.

Overview


We present NIMBLE, a non-rigid parametric hand model that includes bones and muscles, bringing 3D hand model to a new level of realism. NIMBLE is learnt from a MRI hand dataset with detailed annotation on joint, bone and muscles. By enforcing the inner bones and muscles to match anatomic and kinematic rules, NIMBLE can animate 3D hands to new poses at unprecedented realism.

Pipeline



Overview of building NIMBLE, which includes inner and outer registration and parametric model learning. As well as NIMBLE application for synthetic hand generation and photorealistic rendering. 𝜃, 𝛽, 𝛼 are parameters that control model pose, shape and appearance.

Results



With the NIMBLE model and render engines, we can create an unlimited number of photo-realistic hand images and video sequences with corresponding ground-truth inner and outer geometry, pose and texture maps. All of which can be used for downstream hand-related learning tasks.

Bibtex


@article{10.1145/3528223.3530079, author = {Li, Yuwei and Zhang, Longwen and Qiu, Zesong and Jiang, Yingwenqi and Li, Nianyi and Ma, Yuexin and Zhang, Yuyao and Xu, Lan and Yu, Jingyi}, title = {NIMBLE: A Non-Rigid Hand Model with Bones and Muscles}, year = {2022}, issue_date = {July 2022}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {41}, number = {4}, issn = {0730-0301}, url = {https://doi.org/10.1145/3528223.3530079}, doi = {10.1145/3528223.3530079}, journal = {ACM Trans. Graph.}, month = {jul}, articleno = {120}, numpages = {16} }