Abstract
The Bayes factor surface is a new way to present results from experimental searches for new physics. Searches are regularly expressed in terms of phenomenological parameters – such as the mass and cross-section of a weakly interacting massive particle. Bayes factor surfaces indicate the strength of evidence for or against models relative to the background only model in terms of the phenomenological parameters that they predict. They provide a clear and direct measure of evidence, may be easily reinterpreted, but do not depend on choices of prior or parameterization. We demonstrate the Bayes factor surface with examples from dark matter, cosmology, and collider physics.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Experimental searches for new physics are becoming increasingly sophisticated. Nevertheless, experimental results are typically summarized by just a few numbers and plots that express the results of statistical measurements and tests. These results are used by theorists and phenomenologists to understand which models of new physics and which parameters, e.g. masses and couplings of new particles, should be considered ruled out by experiment.
Indeed, there are two major goals in statistics – testing, that is, finding which models are ruled out, and measurement, that is, finding which parameters are ruled out (see e.g., Ref. [1]). Testing and measurement are usually conducted under either the frequentist or Bayesian paradigms of statistics. In frequentist frameworks, one conventionally performs measurement by constructing confidence intervals for an unknown parameter and testing by computing p-values. To construct a confidence interval, one must find a procedure that generates intervals that include the true value of the unknown parameter at a guaranteed rate [2]. The desired rate is known as the confidence level, and \(90\%\) and \(95\%\) are common choices. Procedures that guarantee that the rate is exactly the confidence level are known as exact, whereas those that guarantee that the rate at least as great as the confidence level are known as valid. The confidence interval may be supplemented by estimates of the parameter, such as a best-fit value.
To test in frequentist frameworks, one constructs a procedure that wrongly rejects a null hypothesis at a guaranteed rate. These procedures commonly use a p-value,
That is, the probability under the null hypothesis that the test-statistic, t, is greater than or equal to that observed, \(t^\star \). The p-value allows us to control the error rate: if we reject the null hypothesis when \(p < \alpha \), we wrongly reject the null hypothesis at a rate \(\alpha \). There exists, however, a duality between measurement and testing (see e.g., Refs. [3, 4]). Confidence intervals for a parameter can be constructed by testing possible parameter values.
In the Bayesian framework, given data x, one can perform measurement by considering the posterior, \(p(\theta \,|\,x, M)\), for an unknown parameter \(\theta \) in model M, and summarize it through moments or percentiles. Indeed, the traditional analogue of a confidence interval would be a credible region [5]. This region, \(\mathcal {R}\), contains a specified percentile of posterior probability, e.g., \(95\%\),
As for confidence intervals, this requires an ordering rule. Credible regions predate confidence intervals; indeed, Laplace used a similar construction to report certainty in the mass of Saturn in 1810 [6].
Testing, on the other hand, may be performed by comparing two models, \(M_1\) and \(M_0\), through a Bayes factor [7],
for observed data x. The Bayes factor is a ratio of evidences, \(Z\), and tells us how we must update the relative plausibility of two hypothesis in light of data. As emphasized in Ref. [8], there is no duality between testing and measurement. We cannot infer Bayes factors from credible regions or vice-versa. Measurement conditions on a particular model; testing compares two models. We summarize the frequentist and Bayesian approaches to testing in Table 1.
The duality between testing and measurement in the frequentist setting means that we should reconsider the Bayesian analogue of a confidence interval. We argue that in common cases confidence intervals are used because they test rather than because they measure. In such cases, we should consider the Bayesian analogue of testing not measurement. This occurs particularly when the results of experiments are expressed in terms of phenomenological parameters. Phenomenological parameters are not directly measurable and nor are they fundamental parameters in any theory – experimental results are expressed through them and theories are mapped to them. In other words, they are a representation of the experimental results that can be readily compared between experiments and interpreted in any model.
Whenever confidence intervals (or indeed credible regions) were used to test rather than to measure, and we wish to use a Bayesian method to tackle the same problem, we argue that we should use a Bayesian approach to testing. Specifically, we suggest presenting contours of the Bayes factor – the Bayes factor surface. For example, theories that make predictions that lie outside a Bayes factor contour at \(B = 1 / 100\) must be relatively disfavored by at least a factor 100.
It was previously suggested to use contours of the Bayes factor to visualize dependence on hyperparameters [9] and similar ideas recently appeared independently in mathematical statistics [10], psychological science [11, 12] and searches for gravitational waves [13]. In Ref. [10], Bayes factors are shown as functions of a root mean square effect size (RMSES). References [11, 12] introduce support intervals as an alternative to credible regions and confidence intervals – one-dimensional intervals defined by thresholds of a one-dimensional Bayes factor surface. This was not constructed in the context of searches for new phenomena, and thus compares specific choices of parameter against averaging across a prior. Lastly, the NANOGrav Collaboration internally developed a Bayes factor surface – denoted the K-ratio – and used it to construct one-dimensional upper limits [13]. The details, properties and justification for this procedure were not, however, discussed so far (Fig. 1).
The rest of this work is structured as follows. In Sect. 2 we explain this proposal and discuss properties and interpretation. In Sect. 3, we present examples of experimental results in cosmology, astroparticle physics and collider physics that are expressed in phenomenological parameters and used to test.
2 Bayes factor contours
2.1 Definition
We consider a search for new physics governed by phenomenological parameters \(\theta \) and nuisance parameters \(\phi \). For reporting and displaying results, we would usually consider at most two phenomenological parameters. By convention, we assume that no new physics corresponds to \(\theta = \theta _0\). We construct a Bayes factor as a ratio of evidences,
where the evidence, \(Z\),
for observed data x with likelihood \(p(x \,|\,\theta , \phi )\) and prior for the nuisance parameters \(p(\phi \,|\,\theta )\). In our Bayes factor, the denominator was the evidence for no new physics, which was nested in our model of new physics. Other choices, including non-nested models, are possible and there may be cases where they are preferred.
Having constructed the Bayes factor as a function of phenomenological parameters \(\theta \), we show results by contours. E.g., the contour of \(\theta \) for which \(B_{10}(\theta ) = 1/10\). Because nuisance parameters are marginalized, the Bayes factor surface can be interpreted as a likelihood ratio for two simple hypotheses. In the nested case, the simple hypotheses are two choices of phenomenological parameters, \(\theta \) and \(\theta _0\). As the Bayes factor surface corresponds to a likelihood ratio as a function of phenomenological parameters, it does not depend on a choice of prior for the phenomenological parameters. Thus, unlike credible regions, this construction does not depend a choice of prior for the parameters \(\theta \), a choice of parameterization or a choice of ordering rule.
2.2 Interpretation
Confidence intervals [14] and hypothesis tests [15] may be hard to interpret. On the other hand, the Bayes factor interval admits a simple interpretation: models that make predictions at level \(B_{10}\) are at least \(B_{10}\) times as plausible relative to the background-only model than they were previously.
Sometimes we know the posterior predictive for the phenomenological parameters in a new model, \(M_{X}\). In these cases, we can estimate the partial Bayes factor, \(P_{X0}\), for this new model by averaging \(B_{10}(\theta )\),
We assumed here that \(p(x \,|\,\theta ) = p(x \,|\,\theta , M_X, K)\), that is, that the data, x, are conditionally independent of the model, M, and any background knowledge, K. The partial Bayes factor \(P_{X0}\) tells us how to update the relative plausibility of model \(M_X\) in light of new data, given the old data.
Lastly, Bayes factor surfaces for the same models from different experiments are simple to combine: we simply multiply them. For example, to combine Bayes factor surfaces from experiments \(E_1\) and \(E_2\),
This assumes, however, that the experiments \(E_1\) and \(E_2\) were independent and that there were no common nuisance parameters. Suppose experiments \(E_1\) and \(E_2\) generate data \(x_1\) and \(x_2\), respectively. Without loss of generality, the evidence may be written,
Thus, the posterior for the nuisance parameters from \(E_1\) plays the role of the prior for the nuisance parameters in \(E_2\). Equation (7) may hold approximately if the experiments are independent and at least one of them depends only weakly on the common nuisances. On the other hand, it could hold exactly if the experiments are independent and if the posterior for the nuisance parameters from \(E_1\) was indeed used as the prior in \(E_2\).
2.3 Nuisance parameters and other parameters of interest
There may be unknown parameters that impact an experiment, but that are not of direct interest. These nuisance parameters should be marginalized, that is, averaged over. These parameters are commonly constrained by background knowledge, e.g., by auxiliary measurements, such that the priors are not controversial.
On the other hand, there may be more than two or three parameters of interest. This number of parameters makes it challenging to visualize a Bayes factor surface or any other type of contour or limit. We could eliminate these parameters by marginalization over a prior. This might, however, make it harder to interpret the resulting contour in a different model and introduces a dependence on the choice of prior. We could instead present slices of parameter space or construct at most two new phenomenological parameters that capture the behavior of the model [16].
2.4 Coverage
Bayes factor contours provide direct measures of evidence; they are not constructed for any particular coverage properties. There are, however guarantees on the properties of the Bayes factor. By Kerridge’s theorem [17],
This bound is universal [18] — there are no regularity conditions or asymptotic assumptions. There may, however, be nuisance parameters that were integrated in Eq. (11),
Kerridge’s theorem thus bounds the expected rate of misleading inferences [19].
A Bayes factor contour at, for example, \(B_{10} = 1/10\) would wrongly exclude a parameter at an average rate of no more than \(10\%\). If there are no nuisance parameters the average error rate equals the completely frequentist error rate, and thus Bayes factor contours are valid confidence intervals. Because Eq. (11) only bounds the rate of misleading inferences, Bayes factor contours overcover and are not exact. For example, the long-run coverage of a \(B_{10} = 1/10\) interval would be at least \(90\%\). Any parameter points that are identical to the null hypothesis would show \(B_{10} = 1\) and thus cover at \(100\%\). These coverage properties are similar to those of \(\text {CL}_s\).
2.5 Levels
In the Neyman–Pearson approach to testing, although we may reject a hypothesis, we cannot quantify evidence for a hypothesis. In the Bayesian framework, we can quantify evidence both for and against a hypothesis. Thus, we may plot contours of Bayes factor at \(B_{10} > 1\) and \(B_{10} < 1\). The contours at \(B_{10} < 1\) show models that are disfavored relative to a chosen alternative; whereas the contours at \(B_{10} > 1\) show models that are favored.
We thus suggest reporting several contours of the Bayes factor, perhaps in logarithmic spacing, spanning the range of possible Bayes factors. For example, if on a plane of parameters the Bayes factor spanned 1/100 to 100, Bayes factor contours could be shown at 1/100, 1/10, 1, 10 and 100. In the absence of evidence for a new theory or effect, the contours at 1/100 and 1/10 would show theories that were disfavored or ruled out at those thresholds. In the presence of evidence, contours at 10 and 100 would show theories that were favored.
2.6 Spurious exclusion
When parameters lie outside a confidence interval, it could indicate either that (i) the tested parameters are false or (ii) the tested parameters are true and there was a fluctuation away from the predictions of the tested parameters.
When worrying about spurious exclusion, we worry about the second case, that is, that exclusion indicates a fluctuation. This possibility may be considered disturbing, especially when the test of the parameters was under-powered, that is, \(1 - \beta \lesssim \alpha \) for power \(\beta \) at error rate \(\alpha \) [20].
This was a concern in high-energy physics. In this context, the size of a new physics contribution to a measurement can be parameterized by \(\mu \ge 0\). The background only model corresponds to \(\mu = 0\). Traditional confidence intervals could be found from a choice of test-statistic, t, with a p-value,
The parameter value \(\mu \) is excluded at \(1 - \alpha \) if \(p(\mu ) < \alpha \). This enables one to find an exact confidence interval.
There were two proposed modifications to confidence intervals to address this spurious exclusion of \(\mu = 0\). First, a \(\text {CL}_s\) construction [21, 22]. This construction considered the ratio
The parameter value \(\mu \) is excluded at \(1 - \alpha \) if \(\text {CL}_s (\mu ) < \alpha \). Since \(\text {CL}_s (\mu ) \ge p(\mu )\), this results in a valid confidence interval that can never exclude signal models that are sufficiently similar to the background only model.
Second, a power constrained limit [23]. This tackled spurious exclusion by requiring
for a chosen threshold \(\beta \), where the power,
Whereas \(p(\mu )\) depends on the observed data, the power is a property of the experimental design and does not depend on the observed data. Power-constrained limits censure tests of parameter values for which the experiment lacked power.
The spurious exclusion problem is hard to formulate in a Bayesian context. In the Bayesian case, we directly consider the relative plausibility of the tested parameters. That comparison is made through the Bayes factor. In any case, when the tested parameters are identical to the null hypothesis, \(B_{10} = 1\) and so the tested parameters cannot lie outside any \(B_{10} < 1\) contour.
2.7 Experimental design
In a frequentist setting, one would design optimal experiments or analyses by maximizing power [24]. By duality, optimal procedures for measurement are optimal procedures for testing [3]. In a Bayesian setting, optimality may be defined through information theory [25]. That is, optimal experiments maximize the information that we learn. As there is no duality between measurement and testing in the Bayesian framework, however, measurement – learning about a parameter – and testing – learning about models – are distinct design goals. If we are concerned by testing, e.g. discovering new phenomena, we may design experiments that are expected to lead to compelling evidence [26].
2.8 Computation
The Bayes factor surface could be computationally challenging, as it may involve challenging evidence integrals [27]. In Eq. (4), we considered nested models. For nested models, the Bayes factor surface could be found through Savage–Dickey density ratios (SDDR; [28]),
where
As Eq. (17) can be written in terms of the posterior and prior density, we avoid direct computation of any challenging evidence integrals. Estimating Eq. (17) through SDDRs requires a choice of prior \(\pi (\theta )\), even though ultimately Eq. (17) should be independent of that choice. We leave discussions of optimal choices of prior for computational efficiency to future works.
There may be a residual challenge; if we want to compute Bayes factor contours at e.g., \(B_{10} = 1/100\) through an SDDR, we could require an estimate of the posterior density in the tails of the posterior. In the context of Markov Chain Monte Carlo (MCMC), an accurate estimate of tails could require a substantial effective sample size and thus a long run or multiple chains. Nested sampling (NS; [29, 30]) could be an attractive alternative to MCMC, because it compresses through a sequence of constrained priors resulting in weighted samples from the tails of the posterior.
Lastly, non-nested models are even more challenging as we cannot avoid evidence integrals. Using the SDDR, however, we avoid computing the Bayes factor surface across sites on a grid. The Bayes factor surface can be expressed as
Thus the numerator can be found through an SDDR. The denominator, however, requires evidence integrals or an estimate of their ratio. The NS algorithm – or any algorithm that returns posterior and evidence at the same time – could be used to compute both the SDDR and the evidence \(Z\). A second run could be required for \(Z_0\), if it involves marginalization of nuisance parameters.
3 Examples
3.1 Cosmology – Planck measurements of the CMB
The standard model of cosmology posits that our Universe underwent a period of accelerated expansion (see e.g., Ref. [31]). This period – called inflation – could solve the flatness and horizon problems in cosmology and leave measurable imprints in the cosmic microwave background (CMB).
Models of inflation are thus constrained by measurements of the CMB by the Planck experiment [32]. The results are commonly expressed on a plane of phenomenological parameters that impact the CMB: the scalar-to-tensor ratio, r, and the spectral tilt, \(n_s\). In the Planck presentation, \(68\%\) and \(95\%\) credible regions are shown on the \((n_s, r)\) plane and predictions from theories of inflation are superimposed. The goal here is testing: testing those fundamental theories against experimental results.
Since the goal is testing, we show instead the Bayes factor surface in Fig. 2: we show contours of a Bayes factor for a model that predicts a specific \((n_s, r)\) versus the base (\(r = 0\)) model defined by Planck. We computed the surface from Markov Chain Monte Carlo (MCMC) samples produced from cosmoMC [33] that are available in the Planck legacy archive [34].Footnote 1 Specifically, we computed:
where each factor was expressed through an SDDR. We estimated the posterior densities \(p(r=0)\) and \(p(r, n_s)\) using kernel density estimation routines from arViz [36]. There were no MCMC samples in the shaded regions in Fig. 2 and we do not extrapolate our contours there. Through the SDDR, we marginalize five parameters in the base + r model (the dark matter and dark energy densities, the Thomson scattering optical depth, the power of the primordial curvature perturbation, and the angular size of sound horizon).
We see in Fig. 2 that the \(95\%\) credible region contourFootnote 2 corresponds to a Bayes factor of only around 1. Models that make predictions inside the \(68\%\) credible region, on the other hand, are favored by more than 10. The maximum Bayes factor was 23. Beyond the \(95\%\) credible region, we see contours showing models disfavored by 10 and 100. From the superimposed model predictions, we see that Starobinsky inflation may be favored by about 10 compared to \(r = 0\); 2/3 and 4/3 mononomial potentials are favored by about 1–10 and cubic inflation is disfavored by about 10 or more. Natural inflation makes a range of predictions that sweep across the contours, though only a fraction of predictions are favored by 10 or more. In summary, in contrast to the credible region, the Bayes factor surface shows us the evidence for or against different models of inflation.
3.2 Collider physics – searches for electroweakinos
Supersymmetry (SUSY) is a popular extension of the Standard Model (SM) of particle physics that predicts a supersymmetric particle for every known SM particle (see e.g., Ref. [37]). The fermionic partners of the photon, W boson and Higgs bosons mix to form electroweakinos. There are four neutral electroweakinos, called neutralinos and denoted \(\chi ^0\), and two charged ones, called charginos and denoted \(\chi ^\pm \). The predictions of a model depend on the masses and mixing angles of the electroweakinos. Results are typically shown on a two-dimensional plane of phenomenological parameters, using simplifying assumptions for the parameters that are not plotted (though see Ref. [16]). These phenomenological parameters are ubiquitous in collider searches and include e.g., signal strength and coupling modifiers in Higgs searches (see e.g., Ref. [38]) and effective field theory coefficients (see e.g., Ref. [39]).
In Refs. [40, 41], a likelihood was constructed for several searches for electroweakinos. To show models that were worse than the background, a capped likelihood ratio versus the background only model was constructed;
for a model that predicted s signal and b background events, and likelihood \(\mathcal {L}\). This was explored by profiling a four-dimensional parameter space to two dimensions. This construction was similar to the Bayes factor surface.
The ATLAS experiment at the large hadron collider (LHC) searched for neutralinos and charginos that were produced from proton collisions and decayed through,
The Higgs and W were assumed to decay to a pair of b-quarks and leptonically, respectively [42]. The observed events were consistent with the background only model and \(95\%\) \(\text {CL}_s\) limits were placed on masses of the electroweakinos, assuming that the lightest chargino and next-to-lightest neutralino are mass-degenerate winos, \(m_{\chi ^\pm _1} = m_{\chi ^0_2}\), and that the lightest neutralino is a bino.
We reanalyze this search using pyhf [43, 44] and the publicly available histfactory models for this search [45]. In Fig. 3 we show the Bayes factor surface as well as the \(95\%\) \(\text {CL}_s\) limit on the \((m_{\chi ^\pm _1} = m_{\chi ^0_2}, m_{\chi ^0_1})\) plane. The Bayes factor surface shows the Bayes factor for a specific choice of masses versus the background only model. We see that a chunk of the plane is excluded at \(95\%\) \(\text {CL}_s\). The Bayes factor surface reveals that this includes models, however, that are favored versus the background only model, \(B_{10} > 1\). This discrepancy occurs because \(\text {CL}_s\) is ratio of tail probabilities whereas the Bayes factor is a ratio of probability densities of the observed data. The contours indicate regions of masses are disfavored by Bayes factors of less than one and less than 20. Lastly, excesses in the observed data of around \(2\sigma \) (local) lead to a region that is favored by between 10 and 100; the existence of this region or the strength of the evidence was not revealed by the \(\text {CL}_s\) contour.
There are 125 nuisance parameters in this histfactory model, and it is suggested that 124 of them are allowed to vary. Marginalizing so many nuisance parameters would be computationally expensive and there is no specialized functionality for this in pyhf.Footnote 3 We thus fix all nuisance parameters to their central values and compute the Bayes factor surface as a likelihood ratio. For consistency, we fix all nuisance parameters in the \(\text {CL}_s\) contour. These contours should thus be considered somewhat approximate, though the \(\text {CL}_s\) contour is quantitatively similar to that in ref. [42].
3.3 Astroparticle physics – direct detection of dark matter
Weakly interacting massive particles (WIMPs) are popular candidates for dark matter (DM) that could be detected through elastic scattering in direct detection (DD) experiments [49, 50]. The signal depends, among other things, on the unknown mass, m and scattering cross section, \(\sigma \), of the WIMP. Thus, results of DD experiments are presented on the phenomenological \((m, \sigma )\) plane. The results are summarized by contours on this plane, traditionally through a confidence interval. These intervals are not uniquely defined by the statistical framework and depend on choices and conventions that differ between experiments. Because of perceived deficiencies in the simplest approaches to setting confidence limits, limits are often set by following specialized techniques including power constrained limits [51].
In Fig. 4 we show the Bayes factor surface versus the background only model and one-dimensional confidence limits from the LZ 2022 DD experiment [48] on the \((m, \sigma )\) plane. They were computed using the likelihood implemented in DDCalc [52]. We follow the convention that astrophysical nuisance parameters are fixed to benchmark values rather than varied [51] such that the Bayes factor surface reduces to a likelihood ratio. We compute one-dimensional confidence limits using an asymptotic approximation and they are numerically similar to the official confidence limits. For comparison, we show predictions from a scalar singlet model of WIMP dark matter [53] found from publicly available MCMC chains [54]. These are posterior predictions; they take into account constraints from previous dark matter searches.
The Bayes factor contours on the \((m, \sigma )\) plane versus the background only model are similar shapes to the confidence limits. They show, however, that the \(99\%\) and \(90\%\) confidence limits correspond to Bayes factors of only about 2 and 10. The contours intersect the predictions from scalar singlet dark matter and indicate the extent to which the model was disfavored by the experimental result. The strength of evidence cannot be deduced from the confidence limit.
In Fig. 5 we show how the contours would look if the LZ experiment had observed a signal of dark matter. We injected \(3 \sqrt{b}\) events into the observed events in each bin. By construction, the one-dimensional confidence limits permit the entire mass range. Reference [51] suggests that in the presence of a signal two-dimensional limits could be drawn as well. The Bayes factor contours show the extent to which models that predicted cross sections that were too large would be disfavored, as well as the evidence for models that reproduce the injected signal. The maximum Bayes factor is about 237, lying at a WIMP mass of \(10\,\text {TeV}\). Confidence limits, on the other hand, cannot indicate the strength of evidence in favor of a signal.
4 Conclusions
We presented the Bayes factor surface – a new way to present experimental searches for new physics. The results of these searches are commonly expressed in terms of phenomenological parameters. We argued that measuring phenomenological parameters themselves is not of direct interest; rather, we are interested in testing models of new physics. In frequentist paradigms, this point is moot as there is a duality between testing and measurement.
In Bayesian statistics, there is no such duality. We thus argued that the Bayesian analogue of frequentist confidence intervals need not be a credible region or a summary based on the posterior. Rather, if we are interested in the testing aspect of confidence intervals, the Bayesian analogue should be built from a Bayes factor. This led us to the Bayes factor surface. The Bayes factor surface shows the strength of evidence for or against a model relative to the background only model in terms of the phenomenological parameters that it predicts. We demonstrated the Bayes factor surface with examples from Planck measurements of the cosmic microwave background, a collider search for supersymmetric particles, and a direct search dark matter.
The Bayes factor surface provides a clear and direct measure of evidence, may be easily reinterpreted, but does not depend on choices of prior or parameterization.
Data availability
Data will be made available on reasonable request. [Authors’ comment: The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.]
Notes
For this example we used the base_r_plikHM_TT_lowl_lowE_lensing_{1,2,3,4}txt chains from COM_CosmoParams_fullGrid_R3.01.zip in Ref. [35].
For consistency, we recompute credible regions using the same tool-chain as that used for the Bayes factor surface. The credible regions are quantitatively similar to the published ones.
References
J.K. Kruschke, T.M. Liddell, The Bayesian new statistics: hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychon. Bull. Rev. 25, 178–206 (2017). https://doi.org/10.3758/s13423-016-1221-4
J. Neyman, Outline of a theory of statistical estimation based on the classical theory of probability. Philos. Trans. R. Soc. Lond. Math. Phys. Sci. 236, 333–380 (1937). https://doi.org/10.1098/rsta.1937.0005
M.G. Kendall, A. Stuart, J.K. Ord, Kendall’s Advanced Theory of Statistics (Oxford University Press, Oxford, 1987)
E. Lehmann, J.P. Romano, Testing Statistical Hypotheses. Springer International Publishing (2022). https://doi.org/10.1007/978-3-030-70578-7
E.T. Jaynes, Confidence Intervals vs Bayesian Intervals (1976), in Papers on Probability, Statistics and Statistical Physics, ed. by R.D. Rosenkrantz, pp. 149–209 (Springer Netherlands, Dordrecht, 1989). https://doi.org/10.1007/978-94-009-6581-2_9
P.-S. Laplace, Mémoire sur les intégrales définies et leur application aux probabilités, et spécialementa la recherche du milieu qu’il faut choisir entre les résultats des observations. Mem. Acad. Sci.(I), XI, Section V, p. 375 (1810). https://en.wikisource.org/wiki/Page%3AA_philosophical_essay_on_probabilities_Tr._Truscott%2C_Emory_1902.djvu/89wikisource
R.E. Kass, A.E. Raftery, Bayes factors. J. Am. Stat. Assoc. 90, 773 (1995). https://doi.org/10.1080/01621459.1995.10476572
R.D. Cousins, Lectures on Statistics in Theory: Prelude to Statistics in Practice. arXiv:1807.05996
C.T. Franck, R.B. Gramacy, Assessing Bayes factor surfaces using interactive visualization and computer surrogate modeling. Am. Stat. 74, 359 (2020). https://doi.org/10.1080/00031305.2019.1671219. arXiv:1809.05580
V.E. Johnson, S. Pramanik, R. Shudde, Bayes factor functions for reporting outcomes of hypothesis tests. Proc. Natl. Acad. Sci. 120, (2023). https://doi.org/10.1073/pnas.2217331120. arXiv:2210.00049
E.-J. Wagenmakers, Q.F. Gronau, F. Dablander, A. Etz, The support interval. Erkenntnis 87, 589–601 (2020). https://doi.org/10.1007/s10670-019-00209-z. https://doi.org/10.31234/osf.io/zwnxb psyarxiv/zwnxb
S. Pawel, A. Ly, E.-J. Wagenmakers, Evidential calibration of confidence intervals. Am. Stat. (2023). https://doi.org/10.1080/00031305.2023.2216239, 1-11 [arXiv: 2206.12290]
NANOGrav collaboration, The NANOGrav 15 yr data set: search for signals from new physics. Astrophys. J. Lett. 951, L11 (2023). https://doi.org/10.3847/2041-8213/acdc91. arXiv: 2306.16219
R.D. Morey, R. Hoekstra, J.N. Rouder, M.D. Lee, E.-J. Wagenmakers, The fallacy of placing confidence in confidence intervals. Psychon. Bull. Rev. 23, 103 (2016). https://doi.org/10.3758/s13423-015-0947-8
S. Greenland, S.J. Senn, K.J. Rothman, J.B. Carlin, C. Poole, S.N. Goodman et al., Statistical tests, p values, confidence intervals, and power: a guide to misinterpretations. Eur. J. Epidemiol. 31, 337–350 (2016). https://doi.org/10.1007/s10654-016-0149-3
M. van Beekveld, P. Grace, A. Kvellestad, A. Leinweber, M. White, Simple, but not simplified: A new approach for optimising beyond-Standard Model physics searches at the Large Hadron Collider. arXiv: 2305.01835
D. Kerridge, Bounds for the frequency of misleading Bayes inferences. Ann. Math. Stat. 34, 1109–1110 (1963). https://doi.org/10.1214/aoms/1177704038
L. Wasserman, A. Ramdas, S. Balakrishnan, Universal inference. Proc. Natl. Acad. Sci. 117, 16880–16890 (2020). https://doi.org/10.1073/pnas.1922664117. arXiv:1912.11436
A. Fowlie, Neyman–Pearson lemma for Bayes factors. Commun. Stat. Theory Method 52, 5379 (2023). https://doi.org/10.1080/03610926.2021.2007265. arXiv:2110.15625
V.L. Highland, Estimation of upper limits from experimental data, Tech. Rep (1986). https://inspirehep.net/files/02763d65a8565e814ea9eb1b39d6fd2d COO-3539-38
A.L. Read, Modified frequentist analysis of search results (the \(\text{CL}_s\) method), in Workshop on Confidence Limits, pp. 81–101 (2000). https://doi.org/10.5170/CERN-2000-005.81
A.L. Read, Presentation of search results: the \(\text{ CL}_s\) technique. J. Phys. G 28, 2693 (2002). https://doi.org/10.1088/0954-3899/28/10/313
G. Cowan, K. Cranmer, E. Gross, O. Vitells, Power-Constrained Limits. arXiv: 1105.3166
R. Fisher, The Design of Experiments (Oliver and Boyd, 1971)
D.V. Lindley, On a measure of the information provided by an experiment. Ann. Math. Stat. 27, 986–1005 (1956). https://doi.org/10.1214/aoms/1177728069
F.D. Schönbrodt, E.-J. Wagenmakers, Bayes factor design analysis: planning for compelling evidence. Psychon. Bull. Rev. 25, 128–142 (2017). https://doi.org/10.3758/s13423-017-1230-y
A. Gelman, X.-L. Meng, Simulating normalizing constants: from importance sampling to bridge sampling to path sampling. Stat. Sci. 13, 163 (1998). https://doi.org/10.1214/ss/1028905934
J.M. Dickey, The weighted likelihood ratio, linear hypotheses on normal location parameters. Ann. Math. Stat. (1971). https://doi.org/10.1214/aoms/1177693507
J. Skilling, Nested sampling for general Bayesian computation. Bayesian Anal. 1, 833 (2006). https://doi.org/10.1214/06-BA127
G. Ashton et al., Nested sampling for physical scientists. Nature (2022). https://doi.org/10.1038/s43586-022-00121-x. arXiv: 2205.15570
D. Baumann, Inflation, in Theoretical Advanced Study Institute in Elementary Particle Physics: Physics of the Large and the Small, pp. 523–686 (2011). https://doi.org/10.1142/9789814327183_0010. arXiv:0907.5424
Planck collaboration, Planck 2018 results. VI. Cosmological parameters. Astron. Astrophys. 641, A6 (2020). https://doi.org/10.1051/0004-6361/201833910. arXiv: 1807.06209
A. Lewis, S. Bridle, Cosmological parameters from CMB and other data: a Monte Carlo approach. Phys. Rev. D 66, 103511 (2002). https://doi.org/10.1103/PhysRevD.66.103511. arXiv:astro-ph/0205436
X. Dupac, C. Arviset, M. Fernandez Barreiro, M. Lopez-Caniego, J. Tauber, The Planck Legacy Archive, in Science Operations 2015: Science Data Management, p. 1 (2015). https://doi.org/10.5281/zenodo.34639
Planck collaboration, PR3 Cosmology Products, Dataset. https://doi.org/10.5270/esa-gb3sw1aPR3 (2018)
R. Kumar, C. Carroll, A. Hartikainen, O. Martin, ArviZ a unified library for exploratory analysis of Bayesian models in Python. J. Open Source Softw. 4, 1143 (2019). https://doi.org/10.21105/joss.01143
S.P. Martin, A supersymmetry primer. Adv. Ser. Direct. High Energy Phys. 18, 1 (1998). https://doi.org/10.1142/9789812839657_0001. arXiv:hep-ph/9709356
CMS collaboration, A portrait of the Higgs boson by the CMS experiment ten years after the discovery. Nature607, 60 (2022). https://doi.org/10.1038/s41586-022-04892-x. arXiv:2207.00043
CMS collaboration, Search for physics beyond the standard model in top quark production with additional leptons in the context of effective field theory. JHEP 12, 068 (2023). https://doi.org/10.1007/JHEP12(2023)068. arXiv:2307.15761
GAMBIT collaboration, Combined collider constraints on neutralinos and charginos. Eur. Phys. J. C 79, 395 (2019). https://doi.org/10.1140/epjc/s10052-019-6837-x. arXiv:1809.02097
GAMBIT collaboration, Collider constraints on electroweakinos in the presence of a light gravitino. Eur. Phys. J. C 83, 493 (2023). https://doi.org/10.1140/epjc/s10052-023-11574-z. arXiv: 2303.09082
ATLAS collaboration, Search for direct production of electroweakinos in final states with one lepton, missing transverse momentum and a Higgs boson decaying into two \(b\)-jets in \(pp\) collisions at \(\sqrt{s}=13\) TeV with the ATLAS detector. Eur. Phys. J. C 80, 691 (2020). https://doi.org/10.1140/epjc/s10052-020-8050-3. arXiv: 1909.09226
L. Heinrich, M. Feickert, G. Stark, pyhf, version https://doi.org/10.5281/zenodo.1169739 0.7.5
L. Heinrich, M. Feickert, G. Stark, K. Cranmer, pyhf: pure-python implementation of histfactory statistical models. J. Open Source Softw. 6, 2823 (2021). https://doi.org/10.21105/joss.02823
ATLAS collaboration, “1Lbb-likelihoods-hepdata.tar.gz” of “Search for direct production of electroweakinos in final states with one lepton, missing transverse momentum and a Higgs boson decaying into two \(b\)-jets in (pp) collisions at \(\sqrt{s}=13\) TeV with the ATLAS detector” (version 3), HEPData (2020). https://doi.org/10.17182/hepdata.90607.v3/r3
M. Feickert, L. Heinrich and M. Horstmann, Bayesian methodologies with pyhf, in 26th International Conference on Computing in High Energy & Nuclear Physics (2023). arXiv:2309.17005
R.M. Neal, Monte Carlo Implementation, in Lecture Notes in Statistics, pp. 55–98 (Springer, New York (1996). https://doi.org/10.1007/978-1-4612-0745-0_3
LZ collaboration, First dark matter search results from the LUX-ZEPLIN (LZ) experiment. Phys. Rev. Lett. 131, 041002 (2023). https://doi.org/10.1103/PhysRevLett.131.041002. arXiv:2207.03764
M.W. Goodman, E. Witten, Detectability of certain dark matter candidates. Phys. Rev. D 31, 3059 (1985). https://doi.org/10.1103/PhysRevD.31.3059
G. Jungman, M. Kamionkowski, K. Griest, Supersymmetric dark matter. Phys. Rep. 267, 195 (1996). https://doi.org/10.1016/0370-1573(95)00058-5. arXiv:hep-ph/9506380
D. Baxter et al., Recommended conventions for reporting results from direct dark matter searches. Eur. Phys. J. C series 81, 907 (2021). https://doi.org/10.1140/epjc/s10052-021-09655-y. [arXiv: 2105.00599]
GAMBIT collaboration, DarkBit: a GAMBIT module for computing dark matter observables and likelihoods. Eur. Phys. J. C 77, 831 (2017). https://doi.org/10.1140/epjc/s10052-017-5155-4. arXiv: 1705.07920
GAMBIT collaboration, Status of the scalar singlet dark matter model. Eur. Phys. J. C 77, 568 (2017). https://doi.org/10.1140/epjc/s10052-017-5113-1. arXiv: 1705.07931
GAMBIT collaboration, Supplementary data: status of the scalar singlet dark matter model . Zenodo (2017). https://doi.org/10.5281/zenodo.846860. arXiv:1705.07931
Acknowledgements
We thank Valen Johnson, Kai Schmitz, Samuel Pawel and Ken Olum for discussions.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Code availability
Code/software will be made available on reasonable request. [Authors’ comment: The codes/software generated during and/or analysed during the current study are available from the corresponding author on reasonable request.]
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Funded by SCOAP3.
About this article
Cite this article
Fowlie, A. The Bayes factor surface for searches for new physics. Eur. Phys. J. C 84, 426 (2024). https://doi.org/10.1140/epjc/s10052-024-12792-9
Received:
Accepted:
Published:
DOI: https://doi.org/10.1140/epjc/s10052-024-12792-9