Unified Geometry-Instance 4D Reconstruction
IGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework.
IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer
1Horizon Robotics
2S-Lab, Nanyang Technological University
3Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
arXiv
Code
Checkpoint
Dataset
Benchmark
The paper, code, checkpoints, dataset, and benchmark will be released soon.
Abstract
Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.
Key Contributions
Streaming Reconstruction & Instance Clustering
Frame-by-frame causal inference and lightweight online clustering enable efficient reconstruction and persistent object identities over long 4D sequences.
InsScene4D-147K Dataset
We curate 147K real/synthetic and static/dynamic sequences, with 4D-consistent instance annotations from an automated geometry-guided pipeline.
InsScene4D-147K Dataset Construction
Beyond existing public datasets, we further curate three complementary data sources for InsScene4D-147K:
RoboTwin 2.0
Using the official RoboTwin 2.0 pipeline, we generate a new collection of RGB-D sequences with temporally consistent instance masks.
RealEstate10K
Using DA3 depth and ground-truth poses, our pipeline reconstructs a TSDF mesh for ID inheritance, then fuses SAM2 auto and box-prompted masks to refine annotations.
HOI4D
Our pipeline removes dynamic regions before static-mesh reconstruction, then updates projected pseudo-labels using provided dynamic-object annotations.
IGGT4D Architecture
Causal Geometry-Instance Transformer
Causal attention and KV caching reuse historical context to build a unified geometry-instance representation frame by frame over long video streams.
Tri-DPT Geometry-Instance Head
The Tri-DPT head jointly decodes depth, rays, and instance features, using geometry-aware attention to inject geometric priors into instance representations.
Streaming Instance Clustering
Per-view cosine-similarity clustering groups instance features into instance masks, matches them to persistent IDs in a global codebook, and updates centers for streaming association.
Streaming Geometry-Instance Predictions
Frame-by-frame visualization of RGB inputs, depth predictions, PCA-projected instance features, and instance masks in streaming video.
Instance-Grounded 4D Scene Understanding
IGGT4D produces 4D-consistent instance features and masks that serve as reusable object-level cues for diverse downstream tasks, including spatial tracking, open-vocabulary segmentation, and LMM-based 4D scene grounding.
Instance Spatial Tracking
Streaming clustering assigns each local mask a persistent global ID. These frame-to-frame correspondences enable dense spatial tracking without an additional association model.
Open-Vocabulary Semantic Segmentation
VLMs such as OpenSeg and LSeg extract per-frame semantics, while instance masks pool them into spatially and temporally consistent object features for open-vocabulary prediction.
4D QA Scene Grounding
4D-consistent instance features and masks provide cues to LMMs such as Qwen3-VL, enabling dynamic-object tracking, grounding, and spatial-temporal question answering.
Experimental Results
Six quantitative tables report geometry reconstruction (camera pose estimation and 3D reconstruction using F1-score and Chamfer Distance), instance spatial tracking, open-vocabulary semantic segmentation, and streaming clustering efficiency.