Virtual-Real Object Registration and Shape Completion for Stable and Accurate Augmented Reality Technology.

Abstract

Augmented Reality (AR) technology integrates virtual objects with the real-world scene and has been widely used in many applications. 3D point cloud registration is one of the main processes to correctly align virtual 3D objects with real-world scenes in AR. The higher quality of the underlying 3D point clouds with fewer missing and noisy points, the more accurate the 3D point cloud registration. The goal of this thesis is to develop algorithms, computational frameworks and methods to improve the quality of 3D point clouds and in order to further increase the accuracy of 3D point cloud registration for AR applications. To achieve this goal, firstly, a computational framework is developed for recovering dense and high-quality 3D point clouds from mono-endoscopic images captured by mono-endoscopic sensors. This computational framework contains a monocular depth learning network to generate the 3D point clouds from monocular images and a 3D point cloud completion network to recover the missing data from the generated 3D point clouds. The experimental results show that this computational framework can generate dense 3D point clouds of real endoscopic images and recover high-quality 3D point clouds from incomplete point clouds with 60% missing points. Secondly, in order to improve the quality of 3D point clouds from real-world objects, a learning-based neural network (TreeNet) has been proposed. The experimental results show that TreeNet outperforms five state-of-the-art learning-based methods and also shows good generalization on unknown data. TreeNet is also evaluated in the proposed computational framework for endoscopic scenes as an application, which proves the effectiveness of TreeNet for medical data. Thirdly, in order to improve the accuracy of rigid 3D point cloud registration, an unsupervised learning-based network (Iterative BTreeNet) has been proposed. Iterative BTreeNet has been compared with three traditional and six state-of-the- art learning-based methods and outperforms these methods on partial and noisy point clouds without training them. Iterative BTreeNet also exhibits remarkable generalization to unseen large and dense scenes that are never trained. Finally, a learning-based network (Deform3DNet) has been proposed for non-rigid 3D point cloud registration. Deform3DNet shows how deep learning is used successfully in solving the non-rigid registration and correspondence challenges end-to-end with non-rigid 3D point clouds. The experimental results demonstrate improvement in the quality of the non-rigid registration and correspondence by comparing Deform3DNet with seven state-of-the-art non-rigid 3D point cloud registration and correspondence methods across large deformations, partiality and topological noise.