Abstract
Nonlinear least squares optimization problems in which the parameters can be partitioned into two sets such that optimal estimates of parameters in one set are easy to solve for given fixed values of the parameters in the other set are common in practice. Particularly ubiquitous are data fitting problems in which the model function is a linear combination of nonlinear functions, which may be addressed with the variable projection algorithm due to Golub and Pereyra. In this paper we review variable projection, with special emphasis on its application to matrix data. The generalization of the algorithm to separable problems in which the linear coefficients of the nonlinear functions are subject to constraints is also discussed. Variable projection has been instrumental for model-based data analysis in multi-way spectroscopy, time-resolved microscopy and gas or liquid chromatography mass spectrometry, and we give an overview of applications in these domains, illustrated by brief case studies.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Bates, D.M., Lindstrom, M.J.: Nonlinear least squares with conditionally linear parameters. In: Proceedings of the Statistical Computing Section, pp. 152–157. American Statistical Association, New York (1986)
Bates, D.M., Watts, D.G.: Nonlinear Regression Analysis and its Applications. Wiley, New York (1988)
Beechem, J., Ameloot, M., Brand, L.: Global and target analysis of complex decay phenomena. Anal. Instrum. 14, 379–402 (1985)
Black, P.E.: “Ragged matrix”. In: Black, P.E. (ed.) Dictionary of Algorithms and Data Structures. U.S. National Institute of Standards and Technology, Gaithersburg (2004)
Garrido, M., Rius, F.X., Larrechi, M.S.: Multivariate curve resolution-alternating least squares (MCR-ALS) applied to spectroscopic data from monitoring chemical reactions processes. Anal. Bioanal. Chem. 390, 2059–2066 (2008)
Gay, D., Kaufman, L.: Tradeoffs in algorithms for separable and block separable nonlinear least squares. In: Vichnevetsky, R., Miller, J.J.H. (eds.) IMACS ’91, Proceedings of the 13th World Congress on Computational and Applied Mathematics, pp. 157–158. Criterion, Dublin (1991)
Gobets, B., van Stokkum, I.H.M., Rogner, M., Kruip, J., Schlodder, E., Karapetyan, N.V., Dekker, J.P., van Grondelle, R.: Time-resolved fluorescence emission measurements of photosystem i particles of various Cyanobacteria: a unified compartmental model. Biophys. J. 81(1), 407–424 (2001)
Godfrey, K.: Compartmental Models and Their Application. Academic, London (1983)
Golub, G.H., LeVeque, R.J.: Extensions and uses of the variable projection algorithm for solving nonlinear least squares problems. In: Proceedings of the 1979 Army Numerical Analysis and Computers Conference, pp. 1–12 (1979)
Golub, G.H., Pereyra, V.: The differentiation of pseudo–inverses and nonlinear least squares problems whose variables separate. Tech. rep., Stanford University, Department of Computer Science (1972)
Golub, G.H., Pereyra, V.: The differentiation of pseudoinverses and nonlinear least squares problems whose variables separate. SIAM J. Numer. Anal. 10, 413–432 (1973)
Golub, G.H., Pereyra, V.: Separable nonlinear least squares: the variable projection method and its applications. Inverse Probl. 19, R1–R26 (2003)
Golub, G.H., van Loan, C.F.: Matrix Computations, 3rd edn. The Johns Hopkins University Press, Baltimore (1996)
de Hoffmann, E., Stroobant, V.: Mass Spectrometry: Principles and Applications, 3rd edn. Wiley, New York (2007)
Holzwarth, A.R.: Data analysis of time-resolved measurements. In: Amesz, J., Hoff, A. (eds.) Biophysical Techniques in Photosynthesis, vol. I, chap. 5, pp. 75–92. Kluwer, Dordrecht (1996)
Istratov, A.A., Vyvenko, O.F.: Exponential analysis in physical phenomena. Rev. Sci. Instrum. 70(2), 1233–1257 (1999)
Jonsson, P., Johansson, A., Gullberg, J., Trygg, J., Jiye, A., Grung, B., Marklund, S., Sjostrom, M., Antti, H., Moritz, T.: High-throughput data analysis for detecting and identifying differences between samples in GC/MS-based metabolomic analyses. Anal. Chem. 77(17), 5635–5642 (2005)
de Juan, A., Tauler, R.: Chemometrics applied to unravel multicomponent processes and mixtures revisiting latest trends in multivariate resolution. Anal. Chim. Acta 500, 195–210 (2003)
Kaufman, L.: A variable projection method for solving separable nonlinear least squares problems. BIT 15, 49–57 (1975)
Krogh, F.T.: Efficient implementation of a variable projection algorithm for nonlinear least squares problems. Commun. Assoc. Comput. Mach. 17(3), 167–169, ISSN 0001-0782 (1974)
Laptenok, S., Mullen, K.M., Borst, J.W., van Stokkum, I.H.M., Apanasovich, V.V., Visser, A.J.W.G.: Fluorescence lifetime imaging microscopy (FLIM) data analysis with TIMP. J. Stat. Softw. 18(8), 1–20 (2007). http://www.jstatsoft.org/v18/i08/
Lawson, C.L., Hanson, R.J.: Solving Least Squares Problems. Prentice Hall, Englewood Cliffs (1974)
Lawton, W.H., Sylvestre, E.A.: Elimination of linear parameters in nonlinear regression. Technometrics 13, 461–467 (1971)
Lukeman, G.G.: Application of the Shen-Ypma algorithm for separable overdetermined nonlinear systems. Master’s thesis, Department of Mathematics and Statistics, Dalhousie University, Canada (1999)
Maus, M., Cotlet, M., Hofkens, J., Gensch, T., de Schryver, F.C., Schaffer, J., Seidel, C.A.M.: An experimental comparison of the maximum likelihood estimation and nonlinear least-squares fluorescence lifetime analysis of single molecules. Anal. Chem. 73(9), 2078–2086 (2001)
Moré, J.J.: The Levenberg-Marquardt algorithm, implementation and theory. In: Watson, G.A. (ed.) Numerical Analysis. Lecture Notes in Mathematics, vol. 630, pp. 105–116. Springer, New York (1978)
Mullen, K.M.: Separable nonlinear models: theory, implementation and applications in physics and chemistry. Ph.D. thesis, Department of Physics and Astronomy, Vrije Universiteit Amsterdam, The Netherlands (2008)
Mullen, K.M., van Stokkum, I.H.M.: TIMP: an R package for modeling multi-way spectroscopic measurements. J. Stat. Softw. 18(3), 1–46 (2007). http://www.jstatsoft.org/v18/i03/
Mullen, K.M., van Stokkum, I.H.M.: TIMP: a problem solving environment for fitting separable nonlinear models in physics and chemistry applications. R package version 1.5.1. http://timp.r-forge.r-project.org/ (2008)
Mullen, K.M., Vengris, M., van Stokkum, I.H.M.: Algorithms for separable nonlinear least squares with application to modelling time-resolved spectra. J. Glob. Optim. 38(2), 201–213 (2007). doi:10.1007/s10898-006-9071-7
Nagle, J.F.: Solving complex photocycle kinetics—theory and direct method. Biophys. J. 59, 476–487 (1991)
Nagle, J.F.: Photocycle kinetics: analysis of Raman data from bacteriorhodopsin. Photochem. Photobiol. 54, 897–903 (1991)
Nagle, J.F., Parodi, L.A., Lozier, R.H.: Procedure for testing kinetic models of the photocycle of bacteriorhodopsin. Biophys. J. 38(2), 161–174 (1982)
Nagle, J.F., Zimanyi, L., Lanyi, J.K.: Testing BR photocycle kinetics. Biophys. J. 68(4), 1490–1499 (1995)
Osborne, M.R.: Separable least squares, variable projection, and the Gauss–Newton algorithm. Electron. Trans. Numer. Anal. 28, 1–15 (2007)
Parks, T.A.: Reducible nonlinear programming problems. Ph.D. thesis, Department of Computational and Applied Mathematics, Rice University, USA (1985)
R Development Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria, ISBN 3-900051-07-0. http://www.R-project.org (2008)
Ruhe, A., Wedin, P.A.: Algorithms for separable nonlinear least squares problems. SIAM Rev. 22(3), 318–337 (1980)
Rust, B.W., Leventhal, M., McCall, S.L.: Evidence for a radioactive decay hypothesis for supernova luminosity. Nature 262, 118–120 (1976)
Seber, G.A.F., Wild, C.J.: Nonlinear Regression. Wiley, New York (2003)
Shen, Y., Ypma, T.J.: Solving rank-deficient separable nonlinear equations. Appl. Numer. Math. 57(5–7), 609–615, ISSN 0168-9274 (2007)
Sima, D.M., Van Huffel, S.: Regularized semiparametric model identification with application to nuclear magnetic resonance signal quantification with unknown macromolecular base-line. J. R. Stat. Soc. B 68(3), 383–409 (2006)
Sima, D.M., Van Huffel, S.: Separable nonlinear least squares fitting with linear bound constraints and its application in magnetic resonance spectroscopy data quantification. J. Comput. Appl. Math. 203, 264–278 (2007)
Sjöberg, J., Viberg, M.: Separable non-linear least-squares minimization-possible improvements for neural net fitting. In: Proceedings of the IEEE Workshop Neural Networks for Signal Processing VII, pp. 345–354, Amelia Island, 24–26 September 1997
van Stokkum, I.H.M.: Global and target analysis of time-resolved spectra, Lecture notes for the Troisième Cycle de la Physique en Suisse Romande. Tech. rep., Department of Physics and Astronomy, Faculty of Sciences, Vrije Universiteit, Amsterdam, The Netherlands, http://www.nat.vu.nl/ivo/lecturenotes.pdf (2005)
van Stokkum, I.H.M., Brouwer, A.M., van Ramesdonk, H., Scherer, T.: Multiresponse parameter estimation and compartmental analysis of time resolved fluorescence spectra: Application to conformational dynamics of charge-separated species in solution. Proc. K. Ned. Akad. Wet. 96, 43–68 (1993)
van Stokkum, I.H.M., Gobets, B., Gensch, T., van Mourik, F., Hellingwerf, K.J., van Grondelle, R., Kennis, J.T.M.: (Sub)-picosecond spectral evolution of fluorescence in photoactive proteins studied with a synchroscan streak camera system. Photochem. Photobiol. 82, 380–388 (2006)
van Stokkum, I.H.M., Larsen, D.S., van Grondelle, R.: Global and target analysis of time-resolved spectra. Biochim. Biophys. Acta 1657, 82–104 (2004) (Erratum 1658, 262)
van Stokkum, I.H.M., Lozier, R.H.: Target analysis of the bacteriorhodopsin photocycle using a spectrotemporal model. J. Phys. Chem. B 106(13), 3477–3485 (2002)
van Stokkum, I.H.M., Mullen, K.M., Mihaleva, V.V.: Global analysis of multiple gas chromatography–mass spectrometry (GC/MS) data sets: a method for resolution of co-eluting components with comparison to MCR-ALS. Chemometr. Intell. Lab. Syst. (in press)
van Stokkum, I.H.M., van Oort, B., van Mourik, F., Gobets, B., van Amerongen, H.: (Sub)-Picosecond Spectral Evolution of Fluorescence Studied with a Synchroscan Streak-Camera System and Target Analysis. In: Aartsma, T.J., Matysik, J. (eds.) Biophysical Techniques in Photosynthesis, Volume II, Series Advances in Photosynthesis and Respiration, vol. 26, pp. 223–240. Springer, Dordrecht, ISBN 978-1-4020-8249-8 (2008)
Suhling, K., French, P.M.W., Phillips, D.: Time-resolved fluorescence microscopy. Photochem. Photobiol. Sci. 4, 13–22 (2005)
Tauler, R.: Multivariate curve resolution applied to second order data. Chemometr. Intell. Lab. Syst. 30, 133–146 (1995)
Tauler, R., Smilde, A., Kowalski, B.: Selectivity, local rank, three-way data analysis and ambiguity in multivariate curve resolution. J. Chemom. 9, 31–58 (1995)
Tsien, R.Y.: The green fluorescent protein. Ann. Rev. Biochem. 67, 509–544 (1998)
van der Veen, J.W.C., de Beer, R., Luyten, P.R., van Ormondt, D.: Accurate quantification of in vivo 31P NMR signals using the variable projection method and prior knowledge. Magn. Reson. Med. 6, 92–98 (1988)
Verveer, P.J., Squire, A., Bastiaens, P.I.H.: Global analysis of fluorescence lifetime imaging microscopy data. Biophys. J. 78(4), 2127–2137 (2000)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Mullen, K.M., Stokkum, I.H.M.v. The variable projection algorithm in time-resolved spectroscopy, microscopy and mass spectrometry applications. Numer Algor 51, 319–340 (2009). https://doi.org/10.1007/s11075-008-9235-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11075-008-9235-2