← Research

PoI: A Filter to Extract Pixel of Interest from Novel Views for Scene Coordinate Regression

Feifei Li, Qi Song, Chi Zhang, Hui Shuai, Rui Huang

Neural View Synthesis (NVS) techniques such as NeRF and 3D Gaussian Splatting (3DGS) have enabled photorealistic rendering from novel viewpoints and are increasingly used to augment training data for visual localization. However, these methods fundamentally rely on observed geometry and radiance; they interpolate existing information but cannot hallucinate unseen 3D structures or recover missing content under sparse or extreme viewpoints. As a result, rendered views often exhibit blur, structural distortion, or incomplete geometry. While such imperfections may be tolerated by Camera Pose Regression (CPR) methods, they severely degrade Scene Coordinate Regression (SCR), which requires accurate per-pixel 3D supervision. To address this limitation, we introduce PoI (Pixel-of-Interest), a framework that enables effective NVS augmentation for SCR-based localization. We first employ 3DGS to render novel views and leverage a single-step diffusion model to refine them, allowing the synthesis of structurally plausible details beyond purely geometry-driven interpolation. However, even diffusion-refined views may contain unreliable pixels. Therefore, we propose a progressive pixel-level filtering strategy based on reprojection error to selectively retain trustworthy synthetic pixels during training while suppressing harmful ones. Extensive experiments on 7Scenes and Cambridge Landmarks demonstrate that our method consistently improves localization accuracy over strong SCR baselines and achieves state-of-the-art performance with competitive training efficiency. Our results reveal that, for SCR, the benefit of novel view augmentation depends not only on generative realism but also on explicit control of pixel-level reliability.

PDF ↗

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace