Neural Mesh Models for 3D Reasoning

Abstract

Our work is built on feature-level render-and-compare methods, which approximate the analysis-by-synthesis approaches in computer vision. Analysis-by-synthesis approaches have several advantages over purely discriminative methods as it enables efficient learning and largely enhances robustness in image classification, pose estimation, and scene understanding, as well as when objects are viewed from unseen 3D poses. We propose neural mesh models (NeMo) that learns a generative model of nueral feature activiations at each vertex on a dense 3D mesh. Using differentiable rendering we solve 3D objects by minimizing the reconstruction error between a predicted feature representation and a representation rendered from an estimated 3D scene. This codebase implements multiple previous works on neural mesh methods, which targets various tasks (e.g., 3D/6D pose estimation and 3D-aware image classification) in different settings (i.e., few-shot and fully-supervised).

Features

3D pose estimation.

Documentation

See documentation.html.

Paper

NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation

Angtian Wang, Adam Kortylewski, Alan Yuille

ICLR 2021

description PDF

insert_comment BibTeX

Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features

Wufei Ma, Angtian Wang, Adam Kortylewski, Alan Yuille

ECCV 2022

description PDF

insert_comment BibTeX

Citation

@inproceedings{wang2021nemo,
    title={NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation},
    author={Angtian Wang and Adam Kortylewski and Alan Yuille},
    booktitle={International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=pmj131uIL9H}
}
@software{nemo_code_2022,
    title={Neural Mesh Models for 3D Reasoning},
    author={Ma, Wufei and Jesslen, Artur and Wang, Angtian},
    month={12},
    year={2022},
    url={https://github.com/wufeim/NeMo},
    version={1.0.0}
}

Further Information

This repo builds upon several previous works:

• NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation (ICLR 2021)
• Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features (ECCV 2022)

Please also consider citing these papers if you follow our work.

@inproceedings{wang2021nemo,
    title={NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation},
    author={Angtian Wang and Adam Kortylewski and Alan Yuille},
    booktitle={International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=pmj131uIL9H}
}