INTERPRETABLE DATA MODELLING USING SPARSE KERNELS
Mr Jaz Kandola, Department of Electronics and Computer Science, University of Southampton

Abstract: Recent approaches to developing methods that learn from examples have been inspired by the need to develop effective solutions that can discover the implicit and non-trivial relationships that exist in data. Whilst a predictive model is often the ultimate goal of modelling, it is often desirable and often even essential to be able to interpret the final model structure. The easiest way to gain such information is to use models where the parameters and the related 'hyperparameters' have clearly interpretable meanings. The Bayesian method of Automatic Relevance Determination (ARD) has been proposed to perform input selection by using hyperparameters to provide structural information about the problem. This talk will start by reviewing the Bayesian techniques that are typically employed to perform input selection and discuss their associated advantages and disadvantages.

Kernel based methods and Support Vector Machines (SVMs) in particular are a class of learning algorithm that can be used for non-linear regression estimation. The solution is given by a weighted linear summation of kernels, where these kernels are `centred' on the data points. Consequently, the solution is difficult to interpret due to the large number of terms that will typically exist in this expansion. Using the ideas from ARD, this talk will describe an interpretable, advanced non-linear modelling approach that enables the constructed kernel based models to be visualised, allowing model validation and assisting in interpretation. The technique combines the representational advantage of a sparse ANOVA decomposition, with the good generalisation ability of a Support Vector Machine. It achieves this by employing two forms of regularisation: a 1-norm based structural regulariser on the hyperparameters to enforce interpretability, and a 2-norm based regulariser to control smoothness. The resulting model structure can be visualised showing the overall effects of different inputs, their interactions, and the strength of the interactions. The robustness of the technique is illustrated using a range of both artificial and real world datasets. The performance is compared to other modelling techniques, and it is shown to exhibit competitive generalisation performance together with improved interpretability.

This seminar was held at the Department of Computer Science, Royal Holloway, University of London on 12 February 2001.

back