EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI

Tai Wang1*, Xiaohan Mao1,2*, Chenming Zhu1,3*, Runsen Xu1,4, Ruiyuan Lyu1,5, Peisen Li1,5, Xiao Chen1,4, Wenwei Zhang1, Kai Chen1, Tianfan Xue1,4, Xihui Liu1,3, Cewu Lu2, Dahua Lin1,4, Jiangmiao Pang1
1Shanghai AI Laboratory, 2Shanghai Jiao Tong University, 3The University of Hong Kong, 4The Chinese University of Hong Kong, 5Tsinghua University
Paper ArXiv Code Data

EmbodiedScan provides a multi-modal, ego-centric 3D perception dataset and benchmark for holistic 3D scene understanding. Building upon this database, our baseline framework, Embodied Perceptron could process an arbitrary number of multi-modal inputs and demonstrates remarkable 3D perception capabilities, both within the two series of benchmarks we set up and in the wild.

Abstract

In the realm of computer vision and robotics, embodied agents are expected to explore their environment and carry out human instructions. This necessitates the ability to fully understand 3D scenes given their first-person observations and contextualize them into language for interaction. However, traditional research focuses more on scene-level input and output setups from a global view. To address the gap, we introduce EmbodiedScan, a multi-modal, ego-centric 3D perception dataset and benchmark for holistic 3D scene understanding. It encompasses over 5k scans encapsulating 1M ego-centric RGB-D views, 1M language prompts, 160k 3D-oriented boxes spanning over 760 categories, some of which partially align with LVIS, and dense semantic occupancy with 80 common categories. Building upon this database, we introduce a baseline framework named Embodied Perceptron. It is capable of processing an arbitrary number of multi-modal inputs and demonstrates remarkable 3D perception capabilities, both within the two series of benchmarks we set up, i.e., fundamental 3D perception tasks and language-grounded tasks, and in the wild.

Dataset Overview

EmbodiedScan provides a multi-modal, ego-centric 3D perception dataset with massive real-scanned data and rich annotations for indoor scenes. It benchmarks language-grounded holistic 3D scene understanding capabilities for real-world embodied agents.

Framework Pipeline

Embodied Perceptron accepts RGB-D sequence with any number of views along with texts as multi-modal input. It uses classical encoders to extract features for each modality and adopts dense and isomorphic sparse fusion with corresponding decoders for different predictions. The 3D features integrated with the text feature can be further used for language-grounded understanding.

SAM-Assisted Annotation

In-the-Wild Test

Trained with EmbodiedScan, our model shows remarkable 3D perception capabilities both in our established benchmarks and in the wild, even with a different RGB-D sensor in a different environment.

BibTeX

@inproceedings{wang2023embodiedscan,
    title={EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI},
    author={Wang, Tai and Mao, Xiaohan and Zhu, Chenming and Xu, Runsen and Lyu, Ruiyuan and Li, Peisen and Chen, Xiao and Zhang, Wenwei and Chen, Kai and Xue, Tianfan and Liu, Xihui and Lu, Cewu and Lin, Dahua and Pang, Jiangmiao},
    year={2024},
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
}