Neural Mesh Models for 3D Reasoning

Installation

1. Create conda environment:

   conda create -n nemo python=3.9
   conda activate nemo

2. Install PyTorch (see pytorch.org):

   conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=10.2 -c pytorch

3. Install PyTorch3D (see github.com/facebookresearch/pytorch3d):

   conda install -c fvcore -c iopath -c conda-forge fvcore iopath
   conda install -c bottler nvidiacub
   conda install pytorch3d -c pytorch3d

4. Install other dependencies:

   conda install numpy matplotlib scipy scikit-image
   conda install pillow
   conda install -c conda-forge timm tqdm pyyaml transformers
   pip install git+https://github.com/NVlabs/nvdiffrast/
   pip install wget gdown BboxTools opencv-python

5. Install NeMo:

   pip install -e .

Data Preparation

• PASCAL3D+ and Occluded PASCAL3D+.

  python3 prepare_pascal3d.py \
      --config config/datasets/pascal3d.yaml

  To prepare the dataset for specific uses, variants of the configurations files are provided:
  • config/datasets/pascal3d_cls.yaml: for 3D-aware image classification; uniform image sizes for all categories.
  • config/datasets/pascal3d_6d.yaml: training set for 6D pose estimation; data augmentated with varying object scales and background texture padding
• ObjectNet3D.

  python3 prepare_objectnet3d.py \
      --config config/datasets/objectnet3d.yaml

• OOD-CV.

  python3 prepare_ood_cv.py \
      --config config/datasets/ood_cv.yaml

  To prepare the dataset for specific uses, variants of the configurations files are provided:
  • config/datasets/ood_cv_cls.yaml: for 3D-aware image classification; uniform image sizes for all categories.
  • config/datasets/ood_cv_6d.yaml: training set for 6D pose estimation; data augmentated with varying object scales and background texture padding

Training and Evaluation

• Training. There is a general entry point for training baselines and NeMo models. Run the following command with the appropriate configuration and a directory for results:

  CUDA_VISIBLE_DEVICES=0,1,2,3 python3 scripts/train.py \
      --cate aeroplane \
      --config config/pose_estimation_3d.yaml \
      --save_dir exp/pose_estimation_3d_nemo_aeroplane

• Evaluation. After training, evaluate the model by specifying the desired model checkpoint

  CUDA_VISIBLE_DEVICES=0 python3 scripts/inference.py \
      --cate aeroplane \
      --config config/pose_estimation_3d.yaml \
      --save_dir exp/pose_estimation_3d_nemo_aeroplane \
      --checkpoint exp/pose_estimation_3d_nemo_aeroplane/ckpts/model_800.pth

  Model predctions will also be exported to the saving directory (e.g., pascal3d_aeroplane_val.pth) for reuse.

NeMo Models

• NeMo. Neural mesh models (NeMo) for 3D pose estimation.
  • Configuration: config/models/nemo.yaml
  • Configuration for 3D pose estimation on PASCAL3D+: config/pose_estimation_3d.yaml
  • Reference: NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation (ICLR 2021)
  • Weights: https://www.cs.jhu.edu/~wufeim/NeMo/pascal3d/nemo/CATEGORY/model_800.pth, replace CATEGORY with a proper category name (aeroplane, car, etc.).
• NeMo-Cls. Neural mesh models (NeMo) for 3D-aware image classification.
  • Configuration: config/models/nemo_cls.yaml
  • Configuration for 3D pose estimation on PASCAL3D+: config/pose_estimation_3d_nemo_cls.yaml
  • Weights: coming....
• NeMo-6D. Neural mesh models (NeMo) for 6D pose estimation.
  • Configuration: config/models/nemo_6d.yaml
  • Configuration for 6D pose estimation on PASCAL3D+: config/pose_estimation_6d_nemo_6d.yaml
  • Reference: Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features (ECCV 2022)
  • Weights: coming..., replace CATEGORY with a proper category name (aeroplane, car, etc.).

Baseline Models

• ResNet50-General. Regression-based model that formulates 3D pose estimation as a classification problem.
  • Configuration: config/models/resnet50_general.yaml
  • Configuration for 3D pose estimation on PASCAL3D+: config/pose_estimation_3d_resnet50_general.yaml
  • Weights: coming....
• Faster R-CNN. Faster R-CNN extended for 6D pose estimation that formulates pose estimation as a classification problem.
  • Configuration: config/models/faster_rcnn.yaml
  • Configuration for 6D pose estimation on PASCAL3D+: config/pose_estimation_6d_faster_rcnn.yaml
  • Weights: coming....