Fast Forward Selection to Speed Up Sparse Gaussian Process Regression

[edit]

Matthias Seeger, Amazon
Christopher K. I. Williams
Neil D. Lawrence, University of Sheffield

in Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics

Related Material

Abstract

We present a method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection. Our method is essentially as fast as an equivalent one which selects the “support” patterns at random, yet it can outperform random selection on hard curve fitting tasks. More importantly, it leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically. We demonstrate the model selection capabilities of the algorithm in a range of experiments. In line with the development of our method, we present a simple view on sparse approximations for GP models and their underlying assumptions and show relations to other methods.


@InProceedings{seeger-fast03,
  title = 	 {Fast Forward Selection to Speed Up Sparse Gaussian Process Regression},
  author = 	 {Matthias Seeger and Christopher K. I. Williams and Neil D. Lawrence},
  booktitle = 	 {Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics},
  year = 	 {2003},
  editor = 	 {Christopher M. Bishop and Brendan J. Frey},
  address = 	 {Key West, FL},
  month = 	 {00},
  edit = 	 {https://github.com/lawrennd//publications/edit/gh-pages/_posts/2003-01-01-seeger-fast03.md},
  url =  	 {http://inverseprobability.com/publications/seeger-fast03.html},
  abstract = 	 {We present a method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection. Our method is essentially as fast as an equivalent one which selects the “support” patterns at random, yet it can outperform random selection on hard curve fitting tasks. More importantly, it leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically. We demonstrate the model selection capabilities of the algorithm in a range of experiments. In line with the development of our method, we present a simple view on sparse approximations for GP models and their underlying assumptions and show relations to other methods.},
  crossref =  {Bishop:aistats03},
  key = 	 {Seeger:fast03},
  linkpsgz =  {http://www.thelawrences.net/neil/fastForward.ps.gz},
  group = 	 {shefml,spgp}
 

}
%T Fast Forward Selection to Speed Up Sparse Gaussian Process Regression
%A Matthias Seeger and Christopher K. I. Williams and Neil D. Lawrence
%B 
%C Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics
%D 
%E Christopher M. Bishop and Brendan J. Frey
%F seeger-fast03	
%P --
%R 
%U http://inverseprobability.com/publications/seeger-fast03.html
%X We present a method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection. Our method is essentially as fast as an equivalent one which selects the “support” patterns at random, yet it can outperform random selection on hard curve fitting tasks. More importantly, it leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically. We demonstrate the model selection capabilities of the algorithm in a range of experiments. In line with the development of our method, we present a simple view on sparse approximations for GP models and their underlying assumptions and show relations to other methods.
TY  - CPAPER
TI  - Fast Forward Selection to Speed Up Sparse Gaussian Process Regression
AU  - Matthias Seeger
AU  - Christopher K. I. Williams
AU  - Neil D. Lawrence
BT  - Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics
PY  - 2003/01/01
DA  - 2003/01/01
ED  - Christopher M. Bishop
ED  - Brendan J. Frey	
ID  - seeger-fast03	
SP  - 
EP  - 
UR  - http://inverseprobability.com/publications/seeger-fast03.html
AB  - We present a method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection. Our method is essentially as fast as an equivalent one which selects the “support” patterns at random, yet it can outperform random selection on hard curve fitting tasks. More importantly, it leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically. We demonstrate the model selection capabilities of the algorithm in a range of experiments. In line with the development of our method, we present a simple view on sparse approximations for GP models and their underlying assumptions and show relations to other methods.
ER  -

Seeger, M., Williams, C.K.I. & Lawrence, N.D.. (2003). Fast Forward Selection to Speed Up Sparse Gaussian Process Regression. Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics :-