Sketch-based retrieval of images and 3D shapes.

Abstract

With the ubiquitous proliferation of touch screens in consumer electronics such as mobile phones and tablet PCs, the demand of consumers for more convenient product search methods is rising. Since sketching meets this need as a form to better express visual intentions, sketch-based retrieval techniques get increasing attention in the computer vision com- munity. Currently, sketch-based retrieval techniques mainly focus on image and 3D shape retrieval, and some issues such as single- and multi-colour sketch based image retrieval and 3D shape retrieval able to deal with a big domain discrepancy between 2D sketches and 3D shapes have not been well investigated. This thesis will address these issues. For the image retrieval, a single-colour sketch based image retrieval (SCSBIR) approach using RGB and HSV colour features is investigated. Previous methods only consider black-and-white sketches and ignore colour matching between sketches and images, which induce a low retrieval precision. To address this problem, the SCSBIR approach is proposed to consider both shape matching and colour matching with a novel ranking method. Since existing methods cannot effectively distinguish images of the same type but different colours, SCSBIR is further extended to multi-colour sketch based image retrieval (MCSBIR) using a two-stage network architecture, in which a new feature embedding for explicably describing the shape and colour information is proposed and a triplet loss function based on a new Euclidean distance, which separates the shape and colour features, is developed. For the 3D shape retrieval, a teacher-student guided and sketch-based 3D shape retrieval (TSS3DSR) approach is presented to tackle the big domain discrepancy between 2D sketches and 3D shapes. The pre-learned semantic features of 3D shapes are first extracted from the teacher network and then used to guide the feature learning of 2D sketches in the student network. A series of experiments have been carried out to demonstrate the effectiveness of the proposed methods in both the image and 3D shape retrieval. A user interface is also developed to facilitate practical applications of the developed colour sketch-based image retrieval and sketch- based 3D shape retrieval in this thesis.