Multi-criteria optimisation for complex learning prediction systems.

Abstract

This work presents a framework for the inclusion of multiple criteria in the design process of supervised learning algorithms; as well as studies the sophisticated interactions among them. The criteria included and tested experimentally in this thesis are: accuracy, model complexity, algorithmic complexity, diversity and robustness. The present thesis addresses important challenges related to considering multiple criteria such as: 1) defining suitable measures for the included criteria, 2) determining effective approaches to optimise the system performance using multiple objectives, 3) finding effective alternative approaches to include such criteria indirectly in the design stages when defining accurate measures is infeasible, and finally 4) analysing the possible interactions among the criteria as well as identifying the main factors/decision points that modulate them. This work introduces a novel Multi-Components, Multi-Layer Predictive System (MCMLPS). This system incorporates mechanisms designed to control the diversity, model complexity and robustness. In the first stage of this thesis, the accuracy, model and algorithmic complexities of the base components for the proposed system have been optimised empirically using two multi-objective optimisation approaches. The first approach consists of a scalarized multi-objective optimisation, where the models are generated from optimising a single cost function that combines the three criteria. The second approach uses a Pareto-based multi objective optimisation which establishes a trade-off among the three criteria to generate a set of selectively balanced models. These first results showed that models generated from Pareto-based multi objective optimisation approach are both more accurate and more diverse than the models generated from scalarized multi-objective optimisation approach. However, the Pareto-based approach is hindered by the high algorithmic complexity required to find the best model and the infeasibility of defining universal measures for some of the above-mentioned criteria. Thus, in later stages of this work these criteria are either presented as constraints or included indirectly in generating the base components for the MCMLPS. In a subsequent stage of this study, the diversity among the base components of the proposed MCMLPS system is encouraged by training them on local regions in the data, were the locality is determined using the similarity of the data features. Each local region contains either disjoint subsets of the data and/or subsets of the features. A range of similarity metrics such as pairwise squared correlation and conditional mutual information of the features are used. Interestingly, the squared correlation method can be applied in supervised as well as unsupervised learning as it does not consider the output class when splitting the data. Meanwhile, the conditional mutual information method can be applied only in supervised learning as it uses the output class in splitting the data. The full MCMLPS architecture is then analysed and its performance is compared to three well-known ensemble methods. Next, the effect of weighing the components of the MCMLPS and combining them is examined using six fusion methods. The results showed that, including the similarity metric used to divide the data into local regions in weighing the system components, of- ten results in the best accuracy compared to the other fusion methods. In the final phase of this study, the robustness of the proposed system in noisy environments is tested and compared to other ensemble methods. The system showed a comparable accuracy to the best performing ensemble and it often has a more robust performance than other ensembles in highly noisy environments. To conclude, the present thesis proposes a multi-component, multi-layer system which simultaneously incorporates multiple criteria in its design cycle. The results of this thesis suggest that the locality in learning and high diversity among the components of the proposed system can be particularly beneficial in designing ensemble learning methods for highly noisy data sets.