Approximating Posterior Distributions in Belief Networks using Mixtures

Christopher M. BishopNeil D. LawrenceTommi S. JaakkolaMichael I. Jordan
,  10:416-422, 1998.

Abstract

Exact inference in densely connected Bayesian networks is computationally intractable, and so there is considerable interest in developing effective approximation schemes. One approach which has been adopted is to bound the log likelihood using a mean-field approximating distribution. While this leads to a tractable algorithm, the mean field distribution is assumed to be factorial and hence unimodal. In this paper we demonstrate the feasibility of using a richer class of approximating distributions based on *mixtures* of mean field distributions. We derive an efficient algorithm for updating the mixture parameters and apply it to the problem of learning in sigmoid belief networks. Our results demonstrate a systematic improvement over simple mean field theory as the number of mixture components is increased.

Cite this Paper


BibTeX
@InProceedings{pmlr-v-bishop-mixtures97, title = {Approximating Posterior Distributions in Belief Networks using Mixtures}, author = {Christopher M. Bishop and Neil D. Lawrence and Tommi S. Jaakkola and Michael I. Jordan}, pages = {416--422}, year = {}, editor = {}, volume = {10}, address = {Cambridge, MA}, url = {http://inverseprobability.com/publications/bishop-mixtures97.html}, abstract = {Exact inference in densely connected Bayesian networks is computationally intractable, and so there is considerable interest in developing effective approximation schemes. One approach which has been adopted is to bound the log likelihood using a mean-field approximating distribution. While this leads to a tractable algorithm, the mean field distribution is assumed to be factorial and hence unimodal. In this paper we demonstrate the feasibility of using a richer class of approximating distributions based on *mixtures* of mean field distributions. We derive an efficient algorithm for updating the mixture parameters and apply it to the problem of learning in sigmoid belief networks. Our results demonstrate a systematic improvement over simple mean field theory as the number of mixture components is increased. } }
Endnote
%0 Conference Paper %T Approximating Posterior Distributions in Belief Networks using Mixtures %A Christopher M. Bishop %A Neil D. Lawrence %A Tommi S. Jaakkola %A Michael I. Jordan %B %C Proceedings of Machine Learning Research %D %E %F pmlr-v-bishop-mixtures97 %I PMLR %J Proceedings of Machine Learning Research %P 416--422 %U http://inverseprobability.com %V %W PMLR %X Exact inference in densely connected Bayesian networks is computationally intractable, and so there is considerable interest in developing effective approximation schemes. One approach which has been adopted is to bound the log likelihood using a mean-field approximating distribution. While this leads to a tractable algorithm, the mean field distribution is assumed to be factorial and hence unimodal. In this paper we demonstrate the feasibility of using a richer class of approximating distributions based on *mixtures* of mean field distributions. We derive an efficient algorithm for updating the mixture parameters and apply it to the problem of learning in sigmoid belief networks. Our results demonstrate a systematic improvement over simple mean field theory as the number of mixture components is increased.
RIS
TY - CPAPER TI - Approximating Posterior Distributions in Belief Networks using Mixtures AU - Christopher M. Bishop AU - Neil D. Lawrence AU - Tommi S. Jaakkola AU - Michael I. Jordan BT - PY - DA - ED - ID - pmlr-v-bishop-mixtures97 PB - PMLR SP - 416 DP - PMLR EP - 422 L1 - UR - http://inverseprobability.com/publications/bishop-mixtures97.html AB - Exact inference in densely connected Bayesian networks is computationally intractable, and so there is considerable interest in developing effective approximation schemes. One approach which has been adopted is to bound the log likelihood using a mean-field approximating distribution. While this leads to a tractable algorithm, the mean field distribution is assumed to be factorial and hence unimodal. In this paper we demonstrate the feasibility of using a richer class of approximating distributions based on *mixtures* of mean field distributions. We derive an efficient algorithm for updating the mixture parameters and apply it to the problem of learning in sigmoid belief networks. Our results demonstrate a systematic improvement over simple mean field theory as the number of mixture components is increased. ER -
APA
Bishop, C.M., Lawrence, N.D., Jaakkola, T.S. & Jordan, M.I.. (). Approximating Posterior Distributions in Belief Networks using Mixtures. , in PMLR :416-422

Related Material