Geometrically and Perceptually Accurate Facial Mesh Synthesis and Personalised Blendshapes Generation using Graph Neural Networks.

Abstract

The importance of geometric deep learning applications to 3D content creation has in- creased rapidly, driven by significant investments in the next generation Virtual Reality platforms and Visual Effects intensive productions. Generation of high fidelity digital humans became a focal point and one of the fundamental challenges in these applications. The availability of high-quality facial scans and recent advances in deep learning methods applied to mesh processing have led to the development of data driven models. These approaches are at the forefront of content creation technologies, offering artists and users the ability to rapidly generate and edit character assets. Despite major improvements in recent years, the geometric and perceptual quality of 3D facial meshes generated with these techniques do not meet high standards of VFX and AAA video games industry. This research explores geometric deep learning approaches to generation and editing of registered 3D facial meshes and personalised blendshapes. Significant impact of facial shape representations on the quality of reconstructed meshes is demonstrated. Novel methods are proposed to improve the geometric and perceptual accuracy of generated facial meshes and personalised blendshapes. Additionally, user parameters are exposed to independently edit low and high frequency facial deformations. The concept of Deep Spectral Meshes is introduced, which is based on spectral decom- position of meshes in 3D shape representation learning. Using the proposed framework, a parametric model for 3D facial mesh synthesis is built to demonstrate improvements in facial mesh reconstruction in terms of geometric and perceptual error metrics. Additionally, a method is proposed to leverage mutually exclusive objectives of independent control of deformations at different frequencies, and generation of plausible, synthetic examples. A platform is built to compare various deep 3D Morphable Models coupled with different 3D mesh representations and to evaluate them with several distance and perceptual metrics. Using the platform, improvements upon existing state-of-the-art reconstruction results are demonstrated and strengths and weaknesses of 3D mesh representations and preprocessing techniques are further exposed. Subsequently, personalised blendshapes generation with spectral mesh processing method is introduced to improve geometric and perceptual accuracy of synthesised expressions. The proposed method improves personalisation over the deformation transfer, which remains a standard industrial practise. The projects presented in this thesis address professional requirements of industrial partner, Humain Ltd.