Optimising Kernel Parameters and Regularisation Coefficients for Non-linear Discriminant Analysis

Tonatiuh Peña-Centeno, Neil D. Lawrence
Journal of Machine Learning Research, 7:455-491, 2006.

Abstract

In this paper we consider a novel Bayesian interpretation of Fisher’s discriminiant analysis. We relate Rayleigh’s coefficient to a noise model that minimizes a cost based on the most probable class centres and that abandons the ‘regression to the labels’ assumption used by other algorithms. This yields a direction of discrimination equivalent to Fisher’s discriminant. We use Bayes’ rule to infer the posterior distribution for the direction of discrimination and in this process, priors and constraining distributions are incorporated to reach the desired result. Going further, with the use of a Gaussian process prior we show the equivalence of our model to a regularised kernel Fisher’s discriminant. A key advantage of our approach is the facility to determine kernel parameters and the regularisation coefficient through the optimisation of the marginal log-likelihood of the data. An added bonus of the new formulation is that it enables us to link the regularisation coefficient with the generalisation error.

Cite this Paper

BibTeX
@Article{Pena-fbd04, title = {Optimising Kernel Parameters and Regularisation Coefficients for Non-linear Discriminant Analysis}, author = {Peña-Centeno, Tonatiuh and Lawrence, Neil D.}, journal = {Journal of Machine Learning Research}, pages = {455--491}, year = {2006}, volume = {7}, pdf = {http://www.jmlr.org/papers/volume7/centeno06a/centeno06a.pdf}, url = {http://inverseprobability.com/publications/pena-fbd04.html}, abstract = {In this paper we consider a novel Bayesian interpretation of Fisher’s discriminiant analysis. We relate Rayleigh’s coefficient to a noise model that minimizes a cost based on the most probable class centres and that abandons the ‘regression to the labels’ assumption used by other algorithms. This yields a direction of discrimination equivalent to Fisher’s discriminant. We use Bayes’ rule to infer the posterior distribution for the direction of discrimination and in this process, priors and constraining distributions are incorporated to reach the desired result. Going further, with the use of a Gaussian process prior we show the equivalence of our model to a regularised kernel Fisher’s discriminant. A key advantage of our approach is the facility to determine kernel parameters and the regularisation coefficient through the optimisation of the marginal log-likelihood of the data. An added bonus of the new formulation is that it enables us to link the regularisation coefficient with the generalisation error.} }
Endnote
%0 Journal Article %T Optimising Kernel Parameters and Regularisation Coefficients for Non-linear Discriminant Analysis %A Tonatiuh Peña-Centeno %A Neil D. Lawrence %J Journal of Machine Learning Research %D 2006 %F Pena-fbd04 %P 455--491 %U http://inverseprobability.com/publications/pena-fbd04.html %V 7 %X In this paper we consider a novel Bayesian interpretation of Fisher’s discriminiant analysis. We relate Rayleigh’s coefficient to a noise model that minimizes a cost based on the most probable class centres and that abandons the ‘regression to the labels’ assumption used by other algorithms. This yields a direction of discrimination equivalent to Fisher’s discriminant. We use Bayes’ rule to infer the posterior distribution for the direction of discrimination and in this process, priors and constraining distributions are incorporated to reach the desired result. Going further, with the use of a Gaussian process prior we show the equivalence of our model to a regularised kernel Fisher’s discriminant. A key advantage of our approach is the facility to determine kernel parameters and the regularisation coefficient through the optimisation of the marginal log-likelihood of the data. An added bonus of the new formulation is that it enables us to link the regularisation coefficient with the generalisation error.
RIS
TY - JOUR TI - Optimising Kernel Parameters and Regularisation Coefficients for Non-linear Discriminant Analysis AU - Tonatiuh Peña-Centeno AU - Neil D. Lawrence DA - 2006/02/01 ID - Pena-fbd04 VL - 7 SP - 455 EP - 491 L1 - http://www.jmlr.org/papers/volume7/centeno06a/centeno06a.pdf UR - http://inverseprobability.com/publications/pena-fbd04.html AB - In this paper we consider a novel Bayesian interpretation of Fisher’s discriminiant analysis. We relate Rayleigh’s coefficient to a noise model that minimizes a cost based on the most probable class centres and that abandons the ‘regression to the labels’ assumption used by other algorithms. This yields a direction of discrimination equivalent to Fisher’s discriminant. We use Bayes’ rule to infer the posterior distribution for the direction of discrimination and in this process, priors and constraining distributions are incorporated to reach the desired result. Going further, with the use of a Gaussian process prior we show the equivalence of our model to a regularised kernel Fisher’s discriminant. A key advantage of our approach is the facility to determine kernel parameters and the regularisation coefficient through the optimisation of the marginal log-likelihood of the data. An added bonus of the new formulation is that it enables us to link the regularisation coefficient with the generalisation error. ER -
APA
Peña-Centeno, T. & Lawrence, N.D.. (2006). Optimising Kernel Parameters and Regularisation Coefficients for Non-linear Discriminant Analysis. Journal of Machine Learning Research 7:455-491 Available from http://inverseprobability.com/publications/pena-fbd04.html.

Related Material