BodyMap: Learning Full-Body Dense Correspondence Map

1 Moscow Institute of Physics and Technology, 2 Meta AI, 3 Meta Reality Labs Research, Sausalito
*This work was conducted during an internship at Meta Reality Labs Research
CVPR 2022

BodyMap establishes dense surface correspondences for the clothed human body.

Abstract

Dense correspondence between humans carries powerful semantic information that can be utilized to solve fundamental problems for full-body understanding such as in-the wild surface matching, tracking and reconstruction.

In this paper we present BodyMap, a new framework for obtaining high-definition full-body and continuous dense correspondence between in-the-wild images of clothed humans and the surface of a 3D template model. The correspondences cover fine details such as hands and hair, while capturing regions far from the body surface, such as loose clothing. Prior methods for estimating such dense surface correspondence i) cut a 3D body into parts which are unwrapped to a 2D UV space, producing discontinuities along part seams, or ii) use a single surface for representing the whole body, but none handled body details.

Here, we introduce a novel network architecture with Vision Transformers that learn fine-level features on a continuous body surface. BodyMap outperforms prior work on various metrics and datasets, including DensePose-COCO by a large margin. Furthermore, we show various applications ranging from multi-layer dense cloth correspondence, neural rendering with novel-view synthesis and appearance swapping.

BodyMap: Architecture

Given an RGB image as an input we first extract continuous surface embeddings [1] which are then fed to a vision transformer. We combine these features with the features extracted from the respective apperance encoder and feed them to a decoder which establishes dense surface correspondences for each pixel of the clothed human body with high accuracy.

Qualitative Results: Fashion Images





Qualitative Results: Mobile Images



Qualitative Comparisons

Applications: Cloth Swapping

Video

BibTeX

@inproceedings{ianina2022bodymap,
  author    = {Ianina, Anastasia and Sarafianos, Nikolaos and Xu, Yuanlu and Rocco, Ignacio and Tung, Tony},
  title     = {BodyMap: Learning Full-Body Dense Correspondence Map},
  booktitle = {CVPR},
  year      = {2022}
}