Variational Inference for Latent Variables and Uncertain Inputs in Gaussian Processes

Andreas Damianou, Michalis K. Titsias, Neil D. Lawrence
Journal of Machine Learning Research, 17(42):1-62, 2016.

Abstract

The Gaussian process latent variable model (GP-LVM) provides a flexible approach for non-linear dimensionality reduction that has been widely applied. However, the current approach for training GP-LVMs is based on maximum likelihood, where the latent projection variables are maximised over rather than integrated out. In this paper we present a Bayesian method for training GP-LVMs by introducing a non-standard variational inference framework that allows to approximately integrate out the latent variables and subsequently train a GP-LVM by maximising an analytic lower bound on the exact marginal likelihood. We apply this method for learning a GP-LVM from i.i.d. observations and for learning non-linear dynamical systems where the observations are temporally correlated. We show that a benefit of the variational Bayesian procedure is its robustness to over-fitting and its ability to automatically select the dimensionality of the non-linear latent space. The resulting framework is generic, flexible and easy to extend for other purposes, such as Gaussian process regression with uncertain or partially missing inputs. We demonstrate our method on synthetic data and standard machine learning benchmarks, as well as challenging real world datasets, including high resolution video data.

Cite this Paper

BibTeX
@Article{Damianou-variational15, title = {Variational Inference for Latent Variables and Uncertain Inputs in {G}aussian Processes}, author = {Damianou, Andreas and Titsias, Michalis K. and Lawrence, Neil D.}, journal = {Journal of Machine Learning Research}, pages = {1--62}, year = {2016}, volume = {17}, number = {42}, pdf = {https://jmlr.org/papers/volume17/damianou16a/damianou16a.pdf}, url = {http://inverseprobability.com/publications/damianou-variational15.html}, abstract = {The Gaussian process latent variable model (GP-LVM) provides a flexible approach for non-linear dimensionality reduction that has been widely applied. However, the current approach for training GP-LVMs is based on maximum likelihood, where the latent projection variables are maximised over rather than integrated out. In this paper we present a Bayesian method for training GP-LVMs by introducing a non-standard variational inference framework that allows to approximately integrate out the latent variables and subsequently train a GP-LVM by maximising an analytic lower bound on the exact marginal likelihood. We apply this method for learning a GP-LVM from i.i.d. observations and for learning non-linear dynamical systems where the observations are temporally correlated. We show that a benefit of the variational Bayesian procedure is its robustness to over-fitting and its ability to automatically select the dimensionality of the non-linear latent space. The resulting framework is generic, flexible and easy to extend for other purposes, such as Gaussian process regression with uncertain or partially missing inputs. We demonstrate our method on synthetic data and standard machine learning benchmarks, as well as challenging real world datasets, including high resolution video data. } }
Endnote
%0 Journal Article %T Variational Inference for Latent Variables and Uncertain Inputs in Gaussian Processes %A Andreas Damianou %A Michalis K. Titsias %A Neil D. Lawrence %J Journal of Machine Learning Research %D 2016 %F Damianou-variational15 %P 1--62 %U http://inverseprobability.com/publications/damianou-variational15.html %V 17 %N 42 %X The Gaussian process latent variable model (GP-LVM) provides a flexible approach for non-linear dimensionality reduction that has been widely applied. However, the current approach for training GP-LVMs is based on maximum likelihood, where the latent projection variables are maximised over rather than integrated out. In this paper we present a Bayesian method for training GP-LVMs by introducing a non-standard variational inference framework that allows to approximately integrate out the latent variables and subsequently train a GP-LVM by maximising an analytic lower bound on the exact marginal likelihood. We apply this method for learning a GP-LVM from i.i.d. observations and for learning non-linear dynamical systems where the observations are temporally correlated. We show that a benefit of the variational Bayesian procedure is its robustness to over-fitting and its ability to automatically select the dimensionality of the non-linear latent space. The resulting framework is generic, flexible and easy to extend for other purposes, such as Gaussian process regression with uncertain or partially missing inputs. We demonstrate our method on synthetic data and standard machine learning benchmarks, as well as challenging real world datasets, including high resolution video data.
RIS
TY - JOUR TI - Variational Inference for Latent Variables and Uncertain Inputs in Gaussian Processes AU - Andreas Damianou AU - Michalis K. Titsias AU - Neil D. Lawrence DA - 2016/01/01 ID - Damianou-variational15 VL - 17 IS - 42 SP - 1 EP - 62 L1 - https://jmlr.org/papers/volume17/damianou16a/damianou16a.pdf UR - http://inverseprobability.com/publications/damianou-variational15.html AB - The Gaussian process latent variable model (GP-LVM) provides a flexible approach for non-linear dimensionality reduction that has been widely applied. However, the current approach for training GP-LVMs is based on maximum likelihood, where the latent projection variables are maximised over rather than integrated out. In this paper we present a Bayesian method for training GP-LVMs by introducing a non-standard variational inference framework that allows to approximately integrate out the latent variables and subsequently train a GP-LVM by maximising an analytic lower bound on the exact marginal likelihood. We apply this method for learning a GP-LVM from i.i.d. observations and for learning non-linear dynamical systems where the observations are temporally correlated. We show that a benefit of the variational Bayesian procedure is its robustness to over-fitting and its ability to automatically select the dimensionality of the non-linear latent space. The resulting framework is generic, flexible and easy to extend for other purposes, such as Gaussian process regression with uncertain or partially missing inputs. We demonstrate our method on synthetic data and standard machine learning benchmarks, as well as challenging real world datasets, including high resolution video data. ER -
APA
Damianou, A., Titsias, M.K. & Lawrence, N.D.. (2016). Variational Inference for Latent Variables and Uncertain Inputs in Gaussian Processes. Journal of Machine Learning Research 17(42):1-62 Available from http://inverseprobability.com/publications/damianou-variational15.html.

Related Material