# Fast variational inference in the Conjugate Exponential family

James Hensman, University of Lancaster
Magnus Rattray, University of Manchester
Neil D. Lawrence, University of Sheffield

in Advances in Neural Information Processing Systems 25

#### Abstract

We present a general method for deriving collapsed variational inference algorithms for probabilistic models in the conjugate exponential family. Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood. We exploit the information geometry of the bound to derive much faster optimization methods based on conjugate gradients for these models. Our approach is very general and is easily applied to any model where the mean field update equations have been derived. Empirically we show significant speed-ups for probabilistic models optimized using our bound.

  @InProceedings{hensman-fast12, title = {Fast variational inference in the Conjugate Exponential family}, author = {James Hensman and Magnus Rattray and Neil D. Lawrence}, booktitle = {Advances in Neural Information Processing Systems}, year = {2012}, editor = {Peter L. Bartlett and Fernando C. N. Pereira and Christopher J. C. Burges and Léon Bottou and Kilian Q. Weinberger}, volume = {25}, address = {Cambridge, MA}, month = {00}, edit = {https://github.com/lawrennd//publications/edit/gh-pages/_posts/2012-01-01-hensman-fast12.md}, url = {http://inverseprobability.com/publications/hensman-fast12.html}, abstract = {We present a general method for deriving collapsed variational inference algorithms for probabilistic models in the conjugate exponential family. Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood. We exploit the information geometry of the bound to derive much faster optimization methods based on conjugate gradients for these models. Our approach is very general and is easily applied to any model where the mean field update equations have been derived. Empirically we show significant speed-ups for probabilistic models optimized using our bound.}, crossref = {Bartlett:nips12}, key = {Hensman:fast12}, linkpdf = {http://papers.nips.cc/paper/4766-fast-variational-inference-in-the-conjugate-exponential-family}, OPTgroup = {} }
 %T Fast variational inference in the Conjugate Exponential family %A James Hensman and Magnus Rattray and Neil D. Lawrence %B %C Advances in Neural Information Processing Systems %D %E Peter L. Bartlett and Fernando C. N. Pereira and Christopher J. C. Burges and Léon Bottou and Kilian Q. Weinberger %F hensman-fast12 %P -- %R %U http://inverseprobability.com/publications/hensman-fast12.html %V 25 %X We present a general method for deriving collapsed variational inference algorithms for probabilistic models in the conjugate exponential family. Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood. We exploit the information geometry of the bound to derive much faster optimization methods based on conjugate gradients for these models. Our approach is very general and is easily applied to any model where the mean field update equations have been derived. Empirically we show significant speed-ups for probabilistic models optimized using our bound. 
 TY - CPAPER TI - Fast variational inference in the Conjugate Exponential family AU - James Hensman AU - Magnus Rattray AU - Neil D. Lawrence BT - Advances in Neural Information Processing Systems PY - 2012/01/01 DA - 2012/01/01 ED - Peter L. Bartlett ED - Fernando C. N. Pereira ED - Christopher J. C. Burges ED - Léon Bottou ED - Kilian Q. Weinberger ID - hensman-fast12 SP - EP - L1 - http://papers.nips.cc/paper/4766-fast-variational-inference-in-the-conjugate-exponential-family UR - http://inverseprobability.com/publications/hensman-fast12.html AB - We present a general method for deriving collapsed variational inference algorithms for probabilistic models in the conjugate exponential family. Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood. We exploit the information geometry of the bound to derive much faster optimization methods based on conjugate gradients for these models. Our approach is very general and is easily applied to any model where the mean field update equations have been derived. Empirically we show significant speed-ups for probabilistic models optimized using our bound. ER - 
 Hensman, J., Rattray, M. & Lawrence, N.D.. (2012). Fast variational inference in the Conjugate Exponential family. Advances in Neural Information Processing Systems 25:-