Pop-up World: Synthesis of 2.5D Models Based on Monocular Images.

Abstract

For current VR technology, stereoscopic scenes are crucial for an immersive experience. However, the representation of vast 3D scenes in VR poses a significant challenge due to the extensive memory usage and consumption associated with large 3D models. To address this, we propose the use of 2.5D models as a substitute for traditional 3D models in the construction of distant scenes. The method leverages unsupervised clustering algorithm to segment the depth map of the input image. Users can decide the number of segmented layers according to the complexity of the input images and their demands. Each layer is refined using inpainting techniques, especially pre-trained mask-aware transformer models, to ensure a seamless and realistic visual experience. This approach not only reduces the size of the scenes but also maintains the fidelity of the VR experience, striking a balance between technical efficiency and user immersion. The system can effectively handle complex scenes and can be integrated into tools such as Maya for model customization. The source code is available at https://github.com/XChengCode/Synthesis-of2.5D-Models-Based-on-Monocular-Images.