Computer Vision with Machine Learning on Smartphones for Beauty Applications.

Abstract

Over the past decade, computer vision has shifted strongly towards deep learning techniques with neural networks, given their relative ease of application to custom tasks, as well as their greatly improved results compared to traditional computer vision techniques. Since the execution of deep learning models is often resource-heavy, this leads to issues when using them on smartphones, as these are generally constrained by their computing power and battery capacity-limited energy consumption. While it is sometimes possible to conduct such resource-heavy tasks on a powerful remote server receiving the smartphone user’s input, this is not possible for real-time augmented reality applications, due to latency constraints. Since smartphones are by far the most common consumer-oriented augmented reality platforms, this makes on-device neural network execution a highly active area of research, as evidenced by Google’s TensorFlow Lite platform and Apple’s CoreML API and Neural Engine-accelerated iOS devices. The overarching goal of the projects carried out in this thesis is to adapt existing desktop computer-oriented computer vision techniques to smartphones, by lowering the computational requirements, or by developing alternative methods. In concordance with the requirements of the placement company, this research contributed to the creation of various beauty-related smartphone and web apps using Unity, as well as TensorFlow Lite and TensorFlowJS for the machine learning components. Beauty is a highly valued market, which has seen increasing adoption of augmented reality technologies to drive user-customized product sales. The projects presented include a novel 6DoF machine learning system for smartphone object tracking, used in a hair care app, an improved wrinkle and facial blemish detection algorithm and implementation in Unity, as well as research on neural architecture search for facial feature segmentation, and makeup style transfer with generative adversarial networks.