Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts

We consider the problem of category-level 6D pose estimation from a single RGB image. Our approach represents an object category as a cuboid mesh and learns a generative model of the neural feature activations at each mesh vertex to perform pose estimation through differentiable rendering. A common problem of rendering-based approaches is that they rely on bounding box proposals, which do not convey information about the 3D rotation of the object and are not reliable when objects are partially occluded. Instead, we introduce a coarse-to-fine optimization strategy that utilizes the rendering process to estimate a sparse set of 6D object proposals, which are subsequently refined with gradient-based optimization.

In this work, we introduce ROBIN, a benchmark dataset for diagnosing the robustness of vision algorithms to individual nuisances in real-world images. We provide results for a number of popular baselines and make several interesting observations. We believe our dataset provides a rich testbed to study the OOD robustness of vision algorithms and will help to significantly push forward research in this area.

Research project at Microsoft Research Asia supervised by Dr. Bin Li and Dr. Jiahao Li.

We propose a framework for making group decisions from natural language-based preferences. Experiments on the real world data confirms the efficacy of our method.

Characterize 10 different microstructure representations with image texture features and quantitative metrics from image segmentation. For the microstructure generation task, two schemes are considered: 1) generating high-resolution (1024x1024) microstructure images from random noise; and 2) train a style transfer GAN for image generation conditioned on the segmentation label.

Kinetic modeling of a discontinuous precipitation reaction (5 phases) by 1) deep learning with CNN, and 2) performing image segmentation of various microstructure and quantizing the area fractions.

We first review the application of image-driven machine learning approaches to the field of materials characterization. Then we analyze and discuss the impact of various approaches at each step of the experiments.