UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulation

Overview

Teaser image demonstrating Marigold depth estimation.

We present UrbanCAD, a novel pipeline that automatically builds photorealistic and highly controllable digital twins from a single urban image and a large collection of 3D CAD models and handcrafted materials, supporting various editing operations (top). The produced CAD models can be photorealistically inserted into various background scenes and rendered in novel views, synthesizing challenging out-of-distribution (OOD) scenarios with high fidelity for important downstream applications (bottom).

Demo video

Comparison with baselines

How it works

Given a single view input image, we first perform CAD model retrieval and retrieval-based material optimization to create photorealistic and highly controllable vehicle digital twins (left). Given multi-view background images, we then perform realistic vehicle insertion to create various synthetic data for self-driving system testing (right).

Downstream Applications

Constructing photorealistic urban scenarios

We find our CAD retrieval, material optimization, and lighting estimation modules are all crucial to constructing photorealistic urban scenarios with smaller domain gaps by testing on various perception models.

Constructing OOD situations for perception model testing

Thanks to the high controllability of our generated vehicle models, we can construct various out-of-distribution situations to test the robustness of autonomous perception models. We find the performance of perception models degrades obviously when tested on our constructed door opening scenarios.

BibTeX

@misc{lu2024urbancadhighlycontrollablephotorealistic,
      title={UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulation}, 
      author={Yichong Lu and Yichi Cai and Shangzhan Zhang and Hongyu Zhou and Haoji Hu and Huimin Yu and Andreas Geiger and Yiyi Liao},
      year={2024},
      eprint={2411.19292},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.19292}, 
}