3DGS-based Interactive Scene Object Manipulation and Animation.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a premier method for high-fidelity scene reconstruction, yet it lacks a complete workflow for object manipulation. The core challenge lies in the structural incompatibility between the novel volumetric primitives of 3DGS and established editing paradigms found in classical geometry processing and modern 2D computer vision works. The focus of this thesis is to address this gap, namely: how can these disparate methodologies be effectively bridged to enable an end-to-end editing pipeline for 3DGS, comprising segmentation, deformation, and recomposition? For segmentation, this thesis focuses on how to transfer the powerful, open-world segmentation capabilities of modern 2D vision models to 3DGS. A dual-stage self-prompting cross-view propagation process is proposed to guide 3D segmentation using 2D models. A critical learning from this investigation is that while training extended representations effectively connects 2D models to 3DGS, a dual-stage design is crucial to include primitives inside objects, which is essential for robust manipulation. For deformation, this thesis focuses on how to adapt classical cage-based deformation techniques to the novel volumetric primitives of 3DGS. The research demonstrates that a proxy point-based representation can bridge this gap and enables real-time, non-rigid deformation on 3DGS or its variants without requiring retraining or architectural modification. An automatic cagebuilding algorithm is also proposed to produce high-quality cages for manipulation. A key insight is that such proxy representation could be effective for adapting classical shape manipulation methods to the novel volumetric primitives of 3DGS. For inpainting and recomposition, this thesis focuses on how to achieve high-fidelity 3D inpainting of 3DGS using 2D image inpainters. This work overcomes the multiview inconsistency issue of 2D inpainters by employing a perceptual-based loss in the scene fine-tuning process. The process is also augmented with a scene content revealing pruning strategy to improve quality. Recomposition is achieved via set union operations. Apart from employing the perceptual loss, a critical learning is that leveraging existing information in the scene is crucial to minimize scene changes and thus improves result quality. Collectively, these contributions formulate an integrated pipeline for selection, manipulation, and recomposition. By answering these questions, this thesis bridges 3DGS with established paradigms and advancements, enhancing the applicability of captured scenes in creative downstream tasks and achieving state-of-the-art quality in evaluations.