We present HUGSIM, a real-time, photo-realistic and closed-loop simulator for autonomous driving. HUGSIM enables the full closed simulation loop, dynamically updating the ego and actor positions and observations based on control commands. It incorporates 360-degree high-fidelity actor insertion and efficiently generates both normal and aggressive actor behaviors. Moreover, HUGSIM offers a comprehensive benchmark across more than 70 sequences from KITTI-360, Waymo Open Dataset, Nuscenes, and Pandaset, along with over 400 varying scenarios, providing a fair and realistic evaluation platform for existing autonomous driving algorithms.
Our algorithm takes as input posed images of a dynamic urban scene. We decompose the scene into static and dynamic 3D Gaussians, with the motion of dynamic vehicles being modeled via a unicycle model. The 3D Gaussians represent not only appearance but also semantic and flow information, allowing for rendering the RGB images, semantic labels, as well as optical flow through volume rendering.
@article{zhou2024hugsim,
title={HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving},
author={Zhou, Hongyu and Lin, Longzhong and Wang, Jiabao and Lu, Yichong and Bai, Dongfeng and Liu, Bingbing and Wang, Yue and Geiger, Andreas and Liao, Yiyi},
journal={arXiv preprint arXiv:2412.01718},
year={2024}
}