Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and
Objects for 3D Hand Pose Estimation under Hand-Object Interaction
release_wmr674bphjdppojrh4vzhfdpda
by
Anil Armagan, Guillermo Garcia-Hernando, Seungryul Baek, Shreyas
Hampali, Mahdi Rad, Zhaohui Zhang, Shipeng Xie, MingXiu Chen, Boshen Zhang,
Fu Xiong, Yang Xiao, Zhiguo Cao, Junsong Yuan (+18 others)
2020
Abstract
In this work, we study how well different type of approaches generalise in
the task of 3D hand pose estimation under hand-object interaction and single
hand scenarios. We show that the accuracy of state-of-the-art methods can drop,
and that they fail mostly on poses absent from the training set. Unfortunately,
since the space of hand poses is highly dimensional, it is inherently not
feasible to cover the whole space densely, despite recent efforts in collecting
large-scale training datasets. This sampling problem is even more severe when
hands are interacting with objects and/or inputs are RGB rather than depth
images, as RGB images also vary with lighting conditions and colors. To address
these issues, we designed a public challenge to evaluate the abilities of
current 3D hand pose estimators~(HPEs) to interpolate and extrapolate the poses
of a training set. More exactly, our challenge is designed (a) to evaluate the
influence of both depth and color modalities on 3D hand pose estimation, under
the presence or absence of objects; (b) to assess the generalisation abilities
\wrt~four main axes: shapes, articulations, viewpoints, and objects; (c) to
explore the use of a synthetic hand model to fill the gaps of current datasets.
Through the challenge, the overall accuracy has dramatically improved over the
baseline, especially on extrapolation tasks, from 27mm to 13mm mean joint
error. Our analyses highlight the impacts of: Data pre-processing, ensemble
approaches, the use of MANO model, and different HPE methods/backbones.
In text/plain
format
Archived Files and Locations
application/pdf 9.0 MB
file_dxd4mirbw5ctxfecenj6f7suaa
|
arxiv.org (repository) web.archive.org (webarchive) |
2003.13764v1
access all versions, variants, and formats of this works (eg, pre-prints)