Abstract
This paper presents an efficient variational inference framework for a family of structured Gaussian process regression network (SGPRN) models. We incorporate auxiliary inducing variables in latent functions and jointly treat both the distributions of the inducing variables and hyper-parameters as variational parameters. Then we take advantage of the collapsed representation of the model and propose structured variational distributions, which enables the decomposability of a tractable variational lower bound and leads to stochastic optimization. Our inference approach is able to model data in which outputs do not share a common input set, and with a computational complexity independent of the size of the inputs and outputs to easily handle datasets with missing values. Finally, we illustrate our approach on both synthetic and real data.
Chapter PDF
Similar content being viewed by others
Keywords
References
Álvarez, M., Lawrence, N.: Computationally efficient convolved multiple output Gaussian processes. J. Mach. Learn. Res. 12, 1459-1500 (2011)
Goulard, M., Voltz, M.: Linear coregionalization model: tools for estimation and choice of cross-variogram matrix. Math. Geol. 24, 269-286 (1992)
Gneiting, T., Kleiber, W., Schlather, M.: Matérn cross-covariance functions for multivariate random fields. J. Am. Stat. Assoc. 105, 1167-1177 (2010)
Qadir, G., Sun, Y.: Semiparametric estimation of cross-covariance functions for multivariate random fields. Biom. 77, 547-560 (2021)
Gelfand, A., Schmidt, A., Banerjee, S., Sirmans, C.: Nonstationary multivariate process modeling through spatially varying coregionalization. Test. 13, 263-312 (2004)
Meng, R., Soper, B., Lee, H., Liu, V., Greene, J., Ray, P.: Nonstationary multivariate Gaussian processes for electronic health records. J. Biom. Inform. 117, 103698 (2021)
Titsias, M., Lawrence, N.: Bayesian Gaussian process latent variable model. Int. Conf. Artif. Intell. Stat. 844–851 (2010)
Teh, Y., Newman, D., MaxWelling, M.: A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In: Schölkopf, B., Platt, J., Hofmann, T. (eds.) Advances in Neural Information Processing Systems 19, (2006)
Guhaniyogi, R., Finley, A., Banerjee, S., Kobe, R.: Modeling complex spatial dependencies: Low-rank spatially varying cross-covariances with application to soil nutrient data. J. Agric. Biol. Environ. Stat. 18, 274-298 (2013)
Møller, J., Syversveen, A., Waagepetersen, R.: Log Gaussian Cox processes. Scand. J. Stat. 25, 451-482 (1998)
Remes, S., Heinonen, M., Kaski, S.: Non-stationary spectral kernels. Adv. Neural Inf. Process. Syst. 30 (2017), https://proceedings.neurips.cc/paper/2017/file/c65d7bd70fe3e5e3a2f3de681edc193d-Paper.pdf
Nguyen, T., Bonilla, E., et al.: Collaborative multi-output Gaussian processes. Uncertain. Artif. Intell. 643–652 (2014)
Titsias, M., Lázaro-Gredilla, M.: Doubly stochastic variational Bayes for non-conjugate inference. Int. Conf. Mach. Learn. 1971–1979 (2014)
Ranganath, R., Gerrish, S., Blei, D.: Black box variational inference. Int. Conf. Artif. Intell. Stat. 814–822 (2014)
Hoffman, M., Blei, D.,Wang, C., Paisley, J.: Stochastic variational inference. J. Mach. Learn. Res. 14, 1303-1347 (2013)
Rasmussen, C., Kuss, M.: Gaussian processes in reinforcement learning. Adv. Neural Inf. Process. Syst. 751–759 (2004)
Wackernagel, H.: Multivariate geostatistics: an introduction with applications. Springer Science & Business Media (2013)
Nguyen, T., Bonilla, E.: Efficient variational inference for Gaussian process regression networks. Int. Conf. Artif. Intell. Stat. 472–480 (2013)
Liang, X., Zou, T., Guo, B., Li, S., Zhang, H., Zhang, S., Huang, H., Chen, S.: Assessing Beijing’s PM2.5 pollution: severity, weather impact, APEC and winter heating. Proc. R. Soc. A: Math. Phys. Eng. Sci. 471, 20150257 (2015) https://royalsocietypublishing.org/doi/abs/https://doi.org/10.1098/rspa.2015.0257
Snelson, E., Ghahramani, Z.: Sparse Gaussian processes using pseudo-inputs. Adv. Neural Inf. Process. Syst. 1257–1264 (2006), http://papers.nips.cc/paper/2857-sparse-gaussian-processes-using-pseudo-inputs.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2023 The Author(s)
About this paper
Cite this paper
Meng, R., Lee, H.K.H., Bouchard, K. (2023). Stochastic Collapsed Variational Inference for Structured Gaussian Process Regression Networks. In: Brito, P., Dias, J.G., Lausen, B., Montanari, A., Nugent, R. (eds) Classification and Data Science in the Digital Age. IFCS 2022. Studies in Classification, Data Analysis, and Knowledge Organization. Springer, Cham. https://doi.org/10.1007/978-3-031-09034-9_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-09034-9_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09033-2
Online ISBN: 978-3-031-09034-9
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)