Competitive Normalised Least Squares Regression.

Abstract

Online learning has witnessed an increasing interest over the recent past due to its low computational requirements and its relevance to a broad range of streaming applications. In this paper, we focus on online regularised regression. We propose a novel efficient online regression algorithm, called Online Normalised Least Squares (ONLS). We perform theoretical analysis, by comparing the total loss of ONLS against the Normalised Gradient Descent algorithm (NGD) and the best offline LS predictor. We show in particular that ONLS allows for a better bias-variance tradeoff than those state-of-the-art gradient descent-based LS algorithms as well as a better control on the level of shrinkage of the features towards null. Finally, we conduct an empirical study to illustrate the great performance of ONLS against some state-of-the-art algorithms using real-world data.