Flavour and CP symmetries in the inverse seesaw

Abstract

We consider an inverse seesaw mechanism of neutrino mass generation in which the Standard Model is extended by $3+3$ (heavy) sterile states, and endowed with a flavour symmetry $G_f$, $G_f=\Delta (3 \, n^2)$ or $G_f=\Delta (6 \, n^2)$, and a CP symmetry. These symmetries are broken in a peculiar way, so that in the charged lepton sector a residual symmetry $G_\ell$ is preserved, while the neutral fermion sector remains invariant under the residual symmetry $G_\nu =Z_2 \times CP$. We study the concrete setup, where the Majorana mass term for three of the sterile states conserves $G_\nu$, while the remaining mass terms (i.e. couplings of left-handed leptons and heavy sterile states, as well as the Dirac-type couplings among the latter) do not break the flavour or CP symmetry. We perform a comprehensive analysis of lepton mixing for different classes of residual symmetries, giving examples for each of these, and study in detail the impact of the additional sterile states on the predictions for lepton mixing. We further confront our results with those obtained in the model-independent scenario, in which the light neutrino mass matrix leaves the residual symmetry $G_\nu$ intact. We consider the phenomenological impact of the inverse seesaw mechanism endowed with flavour and CP symmetries, in particular concerning effects of non-unitarity of the lepton mixing matrix (which strongly constrain the parameter space of the scenario), prospects for neutrinoless double beta decay and for charged lepton flavour violating processes.

1 Introduction

The Standard Model (SM) of particle physics can successfully explain a plethora of experimental observations. Yet, the existence of three generations of SM fermions, the origin of neutrino masses, the features of lepton and quark mixing, as well as the striking differences between these remain open issues. Symmetries acting on flavour space can address the first and the third point [1,2,3,4], while different types of new particles can be added to the SM in order to generate at least two non-vanishing neutrino masses [5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21].

In the present study, we choose a non-abelian discrete symmetry $G_f$ combined with a CP symmetry, both acting non-trivially on flavour space. This combination has proven to be highly constraining [22,23,24,25,26,27,28,29,30] since, as long as $G_f$ and CP are broken to different residual symmetries $G_\ell$ among charged leptons and $G_\nu =Z_2 \times CP$ among the neutral states, the Pontecorvo–Maki–Nakagawa–Sakata (PMNS) mixing matrix depends on a single free parameter. We select $G_f$ to be a member of the series of groups $\Delta (3 \, n^2)$ [31] and $\Delta (6 \, n^2)$ [32], n integer, because these have shown to lead to several interesting mixing patterns [33,34,35,36,37,38,39,40,41,42,43,44]. Four of these, called Case 1), Case 2), Case 3 a) and Case 3 b.1), have been identified in [33].

Among the different realisations of the Weinberg operator (including the well-known type-I, type-II and type-III seesaw mechanisms – as well as their variants), the so-called inverse seesaw (ISS) mechanism [5,6,7,8] emerges as another interesting possibility. In particular, the ISS mechanism offers a direct connection between the smallness of neutrino masses and the breaking of lepton number (LN) conservation: when compared to the canonical type-I seesaw, a potentially tiny LN violating (LNV) dimensionful coupling $\mu _S$ provides an additional source of suppression for the light neutrino masses, while being technically natural in the sense of ’t Hooft [45] (in the limit in which the LNV couplings vanish, LN conservation is restored as an accidental symmetry of the ISS Lagrangian). The ISS mechanism thus allows to accommodate light neutrino masses for natural values of the Dirac neutrino Yukawa couplings ($\sim {\mathcal {O}}(1)$) at comparatively low scales (TeV or below).

In addition to being a theoretically well-motivated framework, the ISS mechanism can have an important phenomenological impact: as a consequence of the sizeable mixing between active neutrinos and the comparatively light additional sterile states (possibly within collider reach), extensive contributions to numerous observables can occur. Among the latter, one can mention several charged lepton flavour violation (cLFV) processes [46,47,48,49,50,51,52], CP violating observables such as the electric dipole moment (EDM) of the electron [53], or neutrinoless double beta ($0\nu \beta \beta$) decays [54, 55]. The impact of the ISS mechanism regarding the Higgs sector (for instance concerning the one-loop effects of the heavy sterile states on the triple Higgs coupling) has been also explored, and found to be non-negligible (see, for instance [56]).

Flavour (and CP) symmetries have been studied in association with several scenarios of neutrino mass generation, see, e.g., [37,38,39,40,41,42,43,44, 57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73].

In this study, we endow an ISS framework with a flavour symmetry $G_f$ and a CP symmetry. We focus on the so-called (3, 3) ISS framework, in which the SM field content is extended by $3+3$ heavy sterile states, $N_i$ and $S_j$. We note that different realisations of the ISS mechanism with flavour (and CP) symmetries have been considered in the literature, see, e.g., [63,64,65,66,67,68,69,70,71,72,73]. The main features of the present ISS framework are the following: left-handed (LH) lepton doublets, and the sterile states $N_i$ and $S_j$ all transform as irreducible triplets of $G_f$, while right-handed (RH) charged leptons are assigned to singlets, so that the three different charged lepton masses can be easily accommodated. While the source of breaking of $G_f$ and CP to the residual symmetry $G_\ell$ is unique in the charged lepton sector (corresponding to the charged lepton mass terms), the breaking to $G_\nu$ among the neutral states can be realised in different ways. Indeed, we can consider three minimal options, depending on which of the neutral fermion mass terms encodes the symmetry breaking. In this study, we use an option (henceforth called “option 1”), in which only the Majorana mass matrix $\mu _S$ breaks $G_f$ and CP to $G_\nu$. In this way, $\mu _S$ is the unique source of lepton flavour and LN violation in the neutral sector. Similar to what is found for the charged lepton masses, light neutrino masses are not constrained in this scenario, and their mass spectrum can follow either a normal ordering (NO) or an inverted ordering (IO). The mass spectrum of the heavy sterile states is instead strongly restricted, since they combine to form three approximately degenerate pseudo-Dirac pairs (to a very high degree).

We show analytically and numerically that the impact of these heavy sterile states on lepton mixing (i.e., results for lepton mixing angles, predictions for CP phases as well as (approximate) sum rules) is always small, with relative deviations below $1\%$ from the results previously obtained in the model-independent scenario [33]. This is a consequence of effects arising due to deviations from unitarity of the PMNS mixing matrix,¹ which are subject to stringent experimental limits. The matrix encoding these effects is of a peculiar form in our scenario, being both flavour-diagonal and flavour-universal. Due to their pseudo-Dirac nature, the heavy states’ contribution to $0\nu \beta \beta$ decay is always strongly suppressed. As we will discuss, and in stark contrast to typical ISS models, new contributions to cLFV are also negligible. Our scenario thus complies with all experimental limits for masses of the heavy sterile states as low as 500 GeV and Dirac neutrino Yukawa couplings of order 0.1, and successfully reproduces the results for lepton mixing obtained in the model-independent scenario.

The remainder of the paper is organised as follows: in Sect. 2 we present the chosen approach to lepton mixing, first in the model-independent scenario, and then in the (3, 3) ISS framework. Section 3 is devoted to a brief summary of the main results for lepton mixing in the model-independent scenario. The impact of the heavy sterile states of the (3, 3) ISS framework on lepton mixing is analytically evaluated in Sect. 4. The results of the numerical study are discussed in depth in Sect. 5, using an explicit example for each of the different cases, Case 1) through Case 3 b.1), and emphasising the impact of the deviations from unitarity of the PMNS mixing matrix. Sections 6 and 7 are devoted to the results concerning $0\nu \beta \beta$ decays, and prospects for cLFV, respectively. We briefly summarise and give an outlook in Sect. 8. Additional information and complementary discussions are collected in several appendices.

2 Approach to lepton mixing

We assume the existence of a flavour symmetry $G_f=\Delta (3 \, n^2)$ or $G_f=\Delta (6 \, n^2)$ and a $Z_3$ symmetry $Z_3^{(\mathrm {aux})}$, as well as a CP symmetry in the theory.² These are broken (without specifying the breaking mechanism) to a residual $Z_3$ symmetry $G_\ell$, corresponding to the diagonal subgroup of a $Z_3$ group contained in $G_f$ and $Z_3^{(\mathrm {aux})}$,³ in the charged lepton sector and to $G_\nu = Z_2 \times CP$ (with $Z_2$ being a subgroup of $G_f$) among the neutral states. The $Z_2$ symmetry is given by the generator Z, denoted as $Z({\mathbf{r}})$ in the representation ${\mathbf{r}}$. The CP symmetry is described by a CP transformation X in flavour space. In the different representations ${\mathbf{r}}$ of $G_f$, $X ({\mathbf{r}})$ corresponds to a unitary matrix fulfilling

$$\begin{aligned} X ({\mathbf{r}}) \, X({\mathbf{r}})^\star = X({\mathbf{r}})^\star \, X ({\mathbf{r}}) = \mathbb {1}\end{aligned}$$

(1)

so that X is always represented as a symmetric matrix.⁴ A consistent definition of a theory with $G_f$ and CP necessitates the fulfilment of the consistency condition

$$\begin{aligned} X ({\mathbf{r}}) \, g ({\mathbf{r}})^\star \, X ({\mathbf{r}})^\star = g^\prime ({\mathbf{r}}) \end{aligned}$$

(2)

with g and $g^\prime$ being elements of $G_f$ and $g^{(\prime )} ({\mathbf{r}})$ their representation matrices in the representation ${\mathbf{r}}$. This condition must be fulfilled for all representations ${\mathbf{r}}$, or at least for the representations used for charged leptons and the neutral states. Since the product $Z_2 \times CP$ is direct, $Z({\mathbf{r}})$ and $X({\mathbf{r}})$ commute

$$\begin{aligned} X ({\mathbf{r}}) \, Z ({\mathbf{r}})^\star - Z ({\mathbf{r}}) \, X ({\mathbf{r}}) =\mathbb {0}\end{aligned}$$

(3)

for all representations ${\mathbf{r}}$. The flavour and CP symmetries, together with their residuals, determine the lepton mixing pattern. Since we follow the approach to lepton mixing presented in [33], we further assume that the index of $G_f$ is not divisible by three, i.e. $3 \not \mid n$. All choices of CP symmetries and residual $Z_2$ groups in the sector of the neutral states fulfil the conditions in Eqs. (1, 2, 3). For convenience, we summarise in Appendix A the relevant group theory aspects of $G_f$, i.e. the generators and their form in the chosen irreducible representations ${\mathbf{r}}$ of $G_f$. Details about the form of the CP transformation $X ({\mathbf{r}})$ can also be found in Appendix A.

In the following, we first review the implementation of these symmetries and their residuals in the model-independent scenario that has been considered in [33], and then turn to the (3, 3) ISS framework, focusing on one particular implementation, called option 1. We comment on two other minimal options at the end of this section.

2.1 Model-independent scenario

In the model-independent scenario, we consider the mass terms

$$\begin{aligned} -{\bar{\ell }}_{\alpha L} \, (m_\ell )_{\alpha \beta } \, \ell _{\beta R} - \frac{1}{2} \, {\overline{\nu }}^c_{\alpha L} \, (m_\nu )_{\alpha \beta } \, \nu _{\beta L} + \mathrm {h.c.} \end{aligned}$$

(4)

for charged leptons, $m_\ell$, and for neutrinos, $m_\nu$, and with indices $\alpha , \beta = e, \mu , \tau$. While charged leptons acquire their (Dirac) masses from the Yukawa couplings to the Higgs, the LNV neutrino mass term can be effectively generated by means of the Weinberg operator,

$$\begin{aligned} -(y_\ell )_{\alpha \beta } \, {\overline{L}}_\alpha \, H \, \ell _{\beta R} + \frac{1}{\Lambda _{\mathrm {LN}}} \, (y_\nu )_{\alpha \beta } \, \Big ( {\overline{L}}^c_\alpha \, H \Big ) \, \Big ( L_\beta \, H \Big ) + \mathrm {h.c.}\nonumber \\ \end{aligned}$$

(5)

with LH lepton doublets defined as $L_\alpha = \Big ( \begin{array}{c} \nu _{\alpha L} \\ \ell _{\alpha L} \end{array} \Big ) \sim (\mathbf{2}, - \frac{1}{2})$, RH charged leptons $\ell _{\alpha R} \sim (\mathbf{1}, -1)$ and the Higgs doublet $H \sim (\mathbf{2}, \frac{1}{2})$ under $SU(2)_L \times U(1)_Y$. $\Lambda _\mathrm {LN}$ defines the scale at which LN is broken and Majorana neutrino masses are generated. After electroweak symmetry breaking, $\langle H \rangle = \Big ( \begin{array}{c} 0 \\ \frac{v}{\sqrt{2}} \end{array} \Big )$ with $v \approx 246 \, \mathrm {GeV}$, the mass matrices $m_\ell$ and $m_\nu$ are given by

$$\begin{aligned} m_\ell = y_\ell \, \frac{v}{\sqrt{2}} \;\; \text{ and } \;\; m_\nu = y_\nu \, \frac{v^2}{\Lambda _{\mathrm {LN}}} \; . \end{aligned}$$

(6)

The physical (mass) basis, denoted by $^{\wedge }$, is related to the interaction basis by the unitary transformations

$$\begin{aligned} \ell _{L} = U_\ell \, {\hat{\ell }}_{ L}, \;\; \ell _{R} = U_R \, {\hat{\ell }}_{R} \;\; \text{ and } \;\; \nu _{ L} = U_\nu \, {\hat{\nu }}_{ L}. \end{aligned}$$

(7)

The mass matrices $m_\ell$ and $m_\nu$ are then diagonalised as follows

$$\begin{aligned} U_\ell ^\dagger \, m_\ell \, U_R= & {} m_\ell ^\mathrm {diag} = \mathrm {diag}\left( m_e, m_\mu , m_\tau \right) \;\; \text{ and } \;\; \nonumber \\ U_\nu ^T \, m_\nu \, U_\nu= & {} m_\nu ^\mathrm {diag} = \mathrm {diag}\left( m_1, m_2, m_3 \right) \end{aligned}$$

(8)

and the (unitary) PMNS mixing matrix⁵$U_{\mathrm {PMNS}}$ appears in the charged current interactions

$$\begin{aligned} - \frac{g}{\sqrt{2}} \, \overline{{\hat{\ell }}}_{L} \, /\!\!\!\!W^{-} \, U_{\mathrm {PMNS}} \, {\hat{\nu }}_L \;\; \text{ with } \;\; U_{\mathrm {PMNS}} = U_\ell ^\dagger \, U_\nu . \end{aligned}$$

(9)

When it comes to the implementation of $G_f$ and CP, and of the residual symmetries $G_\ell$ and $G_\nu$, we first specify the assignment of LH lepton doublets $L_\alpha$ and RH charged leptons $\ell _{\alpha R}$. In order to constrain as much as possible the resulting lepton mixing pattern, we assign $L_\alpha$ to an irreducible, faithful (complex)⁶ three-dimensional representation $\mathbf{3}$ of $G_f$. This representation can be chosen without loss of generality (see [74] for details) as the representation $\mathbf{3_{(n-1, 1)}}$ and $\mathbf{3_{1 \, (1)}}$ in the convention of [31] and [32], respectively. Right-handed charged leptons $\ell _{\alpha R}$ transform as the trivial singlet $\mathbf{1}$ of $G_f$. In order to distinguish the different flavours, we employ the $Z_3$ symmetry $Z_3^{(\mathrm {aux})}$ and assign $\ell _{eR} \sim 1$, $\ell _{\mu R} \sim \omega$ and $\ell _{\tau R} \sim \omega ^2$ with $\omega = e^{\frac{2 \, \pi \, i}{3}}$. Left-handed lepton doublets $L_\alpha$ do not carry a non-trivial charge under $Z_3^{(\mathrm {aux})}$.

The residual symmetry $G_\ell$ is fixed to the diagonal subgroup of the $Z_3$ group, arising from the generator a of $G_f$, see Eqs. (137, 138) in Appendix A, and $Z_3^{(\mathrm {aux})}$. Since $a (\mathbf{3})$ is diagonal, see Eq. (137), the mass matrix $m_\ell$ of charged leptons is diagonal. In our analysis, we assume that charged lepton masses are canonically ordered⁷ so that the contribution to lepton mixing from the charged lepton sector is trivial, i.e.

$$\begin{aligned} U_\ell = \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad 1 &{}\quad 0\\ 0 &{}\quad 0 &{}\quad 1 \end{array} \right) . \end{aligned}$$

(10)

The lepton mixing pattern depends on the choice of $G_f$, the CP symmetry and the residual $Z_2$ symmetry among the neutral states. In general, the light neutrino mass matrix $m_\nu$ is constrained by the conditions [22]

$$\begin{aligned} Z (\mathbf{3})^T \, m_\nu \, Z (\mathbf{3}) = m_\nu \;\; \text{ and } \;\; X (\mathbf{3}) \, m_\nu \, X (\mathbf{3}) = m_\nu ^\star \; . \end{aligned}$$

(11)

The CP transformation $X (\mathbf{3})$ can be written as

$$\begin{aligned} X (\mathbf{3}) = \Omega (\mathbf{3}) \, \Omega (\mathbf{3})^T \end{aligned}$$

(12)

with $\Omega (\mathbf{3})$ being unitary; furthermore $\Omega (\mathbf{3})$ can be chosen such that

$$\begin{aligned} \Omega (\mathbf{3})^\dagger \, Z (\mathbf{3}) \, \Omega (\mathbf{3}) \;\; \text{ is } \text{ diagonal. } \end{aligned}$$

(13)

In this basis, rotated by $\Omega (\mathbf{3})$, the light neutrino mass matrix is block-diagonal and real. Since $Z (\mathbf{3})$ generates a $Z_2$ symmetry, two of its eigenvalues are equal. This explains why the resulting matrix is block-diagonal and why a rotation around a free angle $\theta$, encoded in the rotation matrix $R_{fh} (\theta )$ (with the indices f and h determined by the pair of degenerate eigenvalues of $Z (\mathbf{3})$), is necessary in order to arrive at a basis in which $m_\nu$ is diagonal. Furthermore, positive semi-definiteness of the light neutrino masses is ensured by a diagonal matrix $K_\nu$, with entries taking values $\pm 1$ and $\pm i$. Hence, $U_\nu$ is given by

$$\begin{aligned} U_\nu = \Omega (\mathbf{3}) \, R_{fh} (\theta ) \, K_\nu . \end{aligned}$$

(14)

The explicit form of $\Omega (\mathbf{3})$ and the value of the indices f and h in the different cases, Case 1) through Case 3 b.1), will be presented in Sect. 3. Since the charged leptons’ physical basis coincides with the interaction basis, see Eq. (10), we have $U_\ell = \mathbb {1}$ and thus $U_{\mathrm {PMNS}}=U_\nu$. The angle $\theta$ can take values between 0 and $\pi$ and is fixed by accommodating the measured lepton mixing angles as well as possible.

2.2 (3, 3) ISS framework

In the (3, 3) ISS framework six neutral states, singlets under the SM gauge group, are added to the SM field content. In the following, these are denoted by $N_i$ and $S_j$ with $i,j = 1, 2, 3$. The Lagrangian giving rise to masses for the neutral particles (i.e. light neutrinos and heavy sterile states) reads

$$\begin{aligned}&- (y_D)_{\alpha i} \, {\overline{L}}^c_\alpha \, H \, N^c_i - (M_{NS})_{ij} \, {\overline{N}}_i \, S_j \nonumber \\&\quad - \frac{1}{2} \, (\mu _S)_{kl} \, {\overline{S}}^c_k \, S_l + \mathrm {h.c.} \end{aligned}$$

(15)

with $\alpha =e, \mu , \tau$ and $i,j,k,l=1,2,3$. In the basis $\left( \nu _{\alpha L}, N^c_i, S_j \right)$,⁸ the mass matrix is of the form

$$\begin{aligned} {\mathcal {M}}_{\mathrm {Maj}} = \left( \begin{array}{ccc} \mathbb {0}&{}\quad m_D &{}\quad \mathbb {0}\\ m_D^T &{} \quad \mathbb {0}&{}\quad M_{NS} \\ \mathbb {0}&{}\quad M_{NS}^T &{}\quad \mu _S \end{array} \right) \;\; \text{ with } \;\; m_D = y_D \, \frac{v}{\sqrt{2}}\,\text{. }\nonumber \\ \end{aligned}$$

(16)

In the limit $|\mu _S| \ll |m_D| \ll |M_{NS}|$ the light neutrino mass matrix is given at leading order in $(|m_D|/|M_{NS}|)^2$ by

$$\begin{aligned} m_\nu = m_D \, \Big ( M_{NS}^{-1} \Big )^{T} \, \mu _S \, M_{NS}^{-1} \, m_D^T\,\text{. } \end{aligned}$$

(17)

The contribution at subleading order reads [75]⁹

$$\begin{aligned} \!\!m_\nu ^1= & {} - \frac{1}{2} \, m_D \, \Big ( M_{NS}^{-1} \Big )^T \, \Big [ \mu _S \, M_{NS}^{-1} \, m_D^T \, m_D^\star \, \Big ( M_{NS}^{-1} \Big )^\dagger \nonumber \\&+ \Big ( M_{NS}^{-1} \Big )^\star \, m_D^\dagger \, m_D \, \Big ( M_{NS}^{-1} \Big )^T \, \mu _S \Big ] \, M_{NS}^{-1} \, m_D^T. \end{aligned}$$

(18)

The source of LN breaking in the ISS framework is $\mu _S$ and light neutrino masses vanish in the limit $\mu _S \, \rightarrow \, 0$, upon which LN conservation is restored.

The matrix ${\mathcal {M}}_{\mathrm {Maj}}$ is diagonalised as

$$\begin{aligned} {\mathcal {U}}^ T \, {\mathcal {M}}_{\mathrm {Maj}} \, {\mathcal {U}} = {\mathcal {M}}_{\mathrm {Maj}}^{\mathrm {diag}} \end{aligned}$$

(19)

with

$$\begin{aligned} {\mathcal {U}} = \left( \begin{array}{cc} {\widetilde{U}}_\nu &{}\quad S \\ T &{}\quad V \end{array} \right) \end{aligned}$$

(20)

in which ${\widetilde{U}}_\nu$ is a three-by-three, S a three-by-six, T a six-by-three and V a six-by-six matrix. The mass spectrum contains the three light (mostly active) neutrinos and six heavy (mostly sterile) states; their masses are denoted by $m_i$, with $i = 1, 2, 3$ corresponding to the light neutrinos, and $i = 4, \ldots , 9$ regarding the heavy neutral mass eigenstates. For $|\mu _S| \ll |M_{NS}|$, the heavy masses are given to good approximation by $M_{NS}$, with $\mu _S$ determining the mass splitting between the states forming pseudo-Dirac pairs.

We note that at leading order ${\widetilde{U}}_\nu$ approximately diagonalises the light neutrino mass matrix (c.f. Eq. (17)) as

$$\begin{aligned} {\widetilde{U}}_\nu ^T m_\nu {\widetilde{U}}_\nu \approx \,\mathrm {diag} (m_1, m_2, m_3). \end{aligned}$$

(21)

While ${\mathcal {U}}$ is unitary, ${\mathcal {U}} \, {\mathcal {U}}^\dagger ={\mathcal {U}}^\dagger \, {\mathcal {U}} = \mathbb {1}$, none of the matrices ${\widetilde{U}}_\nu$, S, T and V has a priori this property. We can define the (in general non-unitary) PMNS mixing matrix as

$$\begin{aligned} {\widetilde{U}}_{\mathrm {PMNS}} = U_\ell ^\dagger \, {\widetilde{U}}_\nu . \end{aligned}$$

(22)

The non-unitarity of ${\widetilde{U}}_{\mathrm {PMNS}}$, induced by the mixing of the active neutrinos with the (heavy) sterile states, can be conveniently captured in the matrix $\eta$, with flavour indices $\alpha , \beta =e, \mu , \tau$. It is defined as¹⁰

$$\begin{aligned} {\widetilde{U}}_{\mathrm {PMNS}} = \Big ( \mathbb {1}- \eta \Big ) \, U_0 \end{aligned}$$

(23)

with $\eta$ hermitian and $U_0$ unitary. Note that

$$\begin{aligned} {\widetilde{U}}_{\mathrm {PMNS}} \, {\widetilde{U}}_{\mathrm {PMNS}}^\dagger \approx \mathbb {1}- 2 \, \eta . \end{aligned}$$

(24)

For $U_\ell =\mathbb {1}$, which is always the case in our analysis, the following equality also holds

$$\begin{aligned} {\widetilde{U}}_{\nu } = \Big ( \mathbb {1}- \eta \Big ) \, U_0. \end{aligned}$$

(25)

The size of $\eta$ and its form in flavour space are given at leading order by

$$\begin{aligned} \eta = \frac{1}{2} \, m_D^\star \, \Big ( M_{NS}^{-1} \Big )^\dagger \, M_{NS}^{-1} \, m_D^T. \end{aligned}$$

(26)

We can estimate the form of the matrix T as

$$\begin{aligned} T = \left( \begin{array}{c} \mathbb {0}\\ - M_{NS}^{-1} \, m_D^T \, {\widetilde{U}}_\nu \end{array} \right) \approx \left( \begin{array}{c} \mathbb {0}\\ - M_{NS}^{-1} \, m_D^T \, U_0 \end{array} \right) , \end{aligned}$$

(27)

while for S one has

$$\begin{aligned} S = \left( \mathbb {0}, \;\; m_D^\star \, \Big ( M_{NS}^{-1} \Big )^\dagger \right) \, V, \end{aligned}$$

(28)

and V approximately diagonalises the lower six-by-six matrix of ${\mathcal {M}}_{\mathrm {Maj}}$, i.e. 

$$\begin{aligned} V^T \, \left( \begin{array}{cc} \mathbb {0}&{}\quad M_{NS} \\ M_{NS}^T &{}\quad \mu _S \end{array} \right) \, V \approx \,\mathrm {diag}\left( m_4, \ldots , m_9\right) . \end{aligned}$$

(29)

The matrix $\mu _S$, a complex symmetric matrix, is itself diagonalised by

$$\begin{aligned} U_S^T \, \mu _S \, U_S = \left( \begin{array}{ccc} \mu _1 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad \mu _2 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad \mu _3 \end{array} \right) , \end{aligned}$$

(30)

with $\mu _i$ real and positive semi-definite, and $U_S$ unitary.

Like in the model-independent scenario, the charged lepton sector leaves the residual symmetry $G_\ell$ invariant. For this reason, we assign the three generations of LH lepton doublets $L_\alpha$ and of RH charged leptons $\ell _{\alpha R}$ to the same representations under $G_f$, the $Z_3$ group $Z_3^{(\mathrm {aux})}$ and the CP symmetry as in the model-independent scenario. As a consequence, also in the (3, 3) ISS framework the charged lepton mass matrix $m_\ell$ is diagonal and the contribution to the lepton mixing matrix is $U_\ell =\mathbb {1}$. The group $G_\nu =Z_2 \times CP$ is the residual symmetry among the neutral states. In the (3, 3) ISS framework, we also have to assign the heavy sterile states, $N_i$ and $S_j$ with $i,j= 1,2,3$, to representations of $G_f$, $Z_3^{(\mathrm {aux})}$ and CP. In the following, we identify three minimal options to choose these representations.

2.2.1 Option 1

For option 1, we assume that $N_i$ and $S_j$ each transform like the LH lepton doublets $L_\alpha$, namely as the triplet $\mathbf{3}$ under $G_f$. Furthermore, the heavy sterile states are neutral under $Z_3^{(\mathrm {aux})}$. As a consequence of this assignment, the Dirac neutrino Yukawa matrix $y_D$, and consequently the mass matrix $m_D$ as well as the matrix $M_{NS}$, are non-vanishing in the limit of unbroken $G_f$, $Z_3^{(\mathrm {aux})}$ and CP. They take a particularly simple form

$$\begin{aligned} m_D = y_0 \, \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad 1 &{}\quad 0\\ 0 &{}\quad 0 &{}\quad 1 \end{array} \right) \, \frac{v}{\sqrt{2}} \;\; \text{ with } \;\; y_0 > 0 \end{aligned}$$

(31)

and

$$\begin{aligned} M_{NS} = M_0 \, \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad 1 &{}\quad 0\\ 0 &{}\quad 0 &{}\quad 1 \end{array} \right) \;\; \text{ with } \;\; M_0 > 0. \end{aligned}$$

(32)

Thus, the only source of $G_f$ and CP breaking in the sector of the neutral states is the matrix $\mu _S$. In order to preserve the residual symmetry $G_\nu$, the matrix $\mu _S$ is constrained by the following equations

$$\begin{aligned} Z (\mathbf{3})^T \, \mu _S \, Z (\mathbf{3}) = \mu _S \;\; \text{ and } \;\; X (\mathbf{3}) \, \mu _S \, X (\mathbf{3}) = \mu _S^\star , \end{aligned}$$

(33)

implying that $\mu _S$ has to fulfil the same relations as $m_\nu$ (cf. Eq. (11)). Hence, the matrix $U_S$, which diagonalises $\mu _S$, is of the same form as $U_\nu$, see Eq. (14),

$$\begin{aligned} U_S = \Omega (\mathbf{3}) \, R_{fh} (\theta _S). \end{aligned}$$

(34)

Note that we do not mention explicitly a matrix equivalent to $K_\nu$ in Eq. (14), as we assume for concreteness in our analysis that it is the identity matrix.

For option 1, $\mu _S$ is the unique source of LN violation and lepton flavour violation. Nevertheless, LN, $G_f$ and CP can be broken in different ways, explicitly or spontaneously, and at vastly different scales in concrete models.

Plugging $m_D$, $M_{NS}$ and $\mu _S$ from Eqs. (30, 31, 32, 34) into the form of $m_\nu$ in Eq. (17), we find at leading order

$$\begin{aligned} m_\nu = \frac{y_0^2 \, v^2}{2 \, M_0^2} \, \mu _S = \frac{y_0^2 \, v^2}{2 \, M_0^2} \, U_S^\star \, \left( \begin{array}{ccc} \mu _1 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad \mu _2 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad \mu _3 \end{array} \right) \, U_S^\dagger . \end{aligned}$$

(35)

Consequently, the matrix ${\widetilde{U}}_\nu$, which diagonalises $m_\nu$ at leading order (neglecting the correction $\eta$ that encodes the deviation from unitarity of ${\widetilde{U}}_\nu$), is given by

$$\begin{aligned} {\widetilde{U}}_\nu \approx U_0 = U_S = \Omega (\mathbf{3}) \, R_{fh} (\theta _S), \end{aligned}$$

(36)

and the light neutrino masses read

$$\begin{aligned} m_i = \frac{y_0^2 \, v^2}{2 \, M_0^2} \, \mu _i \;\; \text{ for } \;\; i=1,2,3. \end{aligned}$$

(37)

Assuming $y_0 \sim 1$ and $M_0 \sim 1000 \, \text{ GeV }$, we can estimate the size of $\mu _i$ to be of the order of $\mathrm {eV}$. The ratio between $m_D$ and $M_{NS}$, evaluating the impact of the heavy sterile states, is then $\frac{y_0 \, v}{\sqrt{2} \, M_0} \sim 0.17$. Since the mass squared differences of neutrinos have been determined from neutrino oscillation data and the sum of neutrino masses is constrained by cosmological measurements, see Appendix C, the values of $\mu _i$ are further restricted. Since $U_\ell =\mathbb {1}$ and ${\widetilde{U}}_\nu$ is at leading order of the form given in Eq. (36), we have for the PMNS mixing matrix

$$\begin{aligned} {\widetilde{U}}_{\mathrm {PMNS}}\approx \Omega (\mathbf{3}) \, R_{fh} (\theta _S) \end{aligned}$$

(38)

with $\theta _S$ being constrained by the measured values of the lepton mixing angles, like $\theta$ in Eq. (14). We note that we consider the free angle $\theta _S$ to vary in the range 0 and $\pi$. The results in the (3, 3) ISS framework (at leading order) are thus identical to those obtained in the model-independent scenario. However, they can be altered by two effects: the inclusion of the subleading contribution $m_\nu ^1$ to the light neutrino mass matrix in Eq. (18) and effects of non-unitarity of ${\widetilde{U}}_\nu$, see Eqs. (23, 26). This is studied in detail analytically in Sect. 4 and numerically in Sect. 5 (the experimental constraints on $\eta$ are discussed in Sect. 5.1).

We briefly discuss the form of the matrices S, T and V, as well as the mass spectrum of the heavy sterile states analytically. With Eqs. (27, 31, 32, 36) the matrix T reads at leading order

$$\begin{aligned} T = \left( \begin{array}{c} \mathbb {0}\\ - \frac{y_0 \, v}{\sqrt{2} \, M_0} \, U_S \end{array} \right) . \end{aligned}$$

(39)

From the definition of V in Eq. (29) and with the form of $M_{NS}$ in Eq. (32) and $\mu _S$ in Eqs. (30, 34), we find at leading order for V

$$\begin{aligned} V = \frac{1}{\sqrt{2}} \, \left( \begin{array}{cc} i \, U_S^\star &{}\quad U_S^\star \\ -i \, U_S &{}\quad U_S \end{array} \right) , \end{aligned}$$

(40)

while the matrix S in Eq. (28) reads

$$\begin{aligned} S = \frac{y_0 \, v}{2 \, M_0} \, \left( - i \, U_S, \;\; U_S \right) . \end{aligned}$$

(41)

We note that the approximate analytical results for ${\widetilde{U}}_\nu$, S, T and V have been compared to the numerical ones for one choice of parameters for Case 1) and we find good agreement in form and magnitude of their entries. The mass spectrum of the heavy sterile states (arising from the diagonalisation through V in Eq. (40)) is at leading order

$$\begin{aligned} m_{3 + i}&= M_0 - \frac{\mu _i}{2} \;\; \text{ and } \;\; m_{6+i}= M_0 + \frac{\mu _i}{2}\nonumber \\&\quad \;\; \text{ with } \;\; i=1,2,3. \end{aligned}$$

(42)

All heavy sterile states are thus degenerate in mass to a very high degree for typical choices of $M_0$ and $\mu _S$, e.g. $M_0 \sim 1000 \, \mathrm {GeV}$ and $\mu _S \lesssim 1 \, \mathrm {keV}$.

Beyond option 1, there are two further minimal options, option 2 and option 3, in which only one of the mass matrices $m_D$, $M_{NS}$ and $\mu _S$ carries non-trivial flavour information. These options share a common feature: in both the matrix $\mu _S$ has a trivial flavour structure. Thus, for these options the sources of lepton flavour and LN violation are decoupled. For option 2, $m_D$ contains all flavour information, while $M_{NS}$ is flavour-diagonal and flavour-universal, so that the mass spectrum of the heavy sterile states will be degenerate to a high degree, like for option 1. Instead, for option 3 the entire flavour structure is encoded in the matrix $M_{NS}$, while $m_D$ is flavour-diagonal and flavour-universal. In this way, the heavy sterile states have in general different masses. We note that the realisation of option 2 and option 3 requires in general that (at least) the assignment of the three sterile states $S_i$, $i=1,2,3$, under the flavour symmetry $G_f$ be altered compared to option 1, in order to ensure that the matrix $\mu _S$ is non-vanishing in the limit of unbroken $G_f$, $Z_3^{(\mathrm {aux})}$ and CP. However, this can always be achieved by an appropriate choice of $G_f$. Obviously, one can also consider less minimal options in which two of the three mass matrices $m_D$, $M_{NS}$ and $\mu _S$, if not all three of them, have a non-trivial flavour structure.

3 Lepton mixing in the model-independent scenario

In this section, we revisit the four different types of lepton mixing patterns, Case 1) through Case 3 b.1), that have been identified in the study of [33]. We mention for each case the generator Z, the CP transformation X and the expressions for $\sin ^2\theta _{ij}$, $J_{\mathrm {CP}}$, $I_1$ and $I_2$ and, where available, (approximate) formulae for the sines of the CP phases as well as (approximate) sum rules among the lepton mixing parameters. We remind that the residual symmetry in the charged lepton sector, $G_\ell$, is always chosen as the $Z_3$ group which corresponds to the diagonal subgroup of the $Z_3$ symmetry, contained in $G_f$ and arising from the generator a, and the $Z_3$ symmetry $Z_3^{(\mathrm {aux})}$. As discussed, this leads to a diagonal charged lepton mass matrix and, consequently, to no contribution to lepton mixing from the charged lepton sector, see Eq. (10).

3.1 Case 1)

For Case 1), the generator Z of the residual $Z_2$ symmetry and the CP transformation X are given by

$$\begin{aligned} Z=c^{n/2} \;\; \text{ and } \;\; X=a\, b \, c^s \, d^{2 \, s} \, X_0 \;\; \text{ with } \;\; 0 \le s \le n-1.\nonumber \\ \end{aligned}$$

(43)

Note that the index n has to be even. The explicit form of the generators and of $X_0$ can be found in Appendix A. The matrix $\Omega (\mathbf{3})$ and the indices f and h of the rotation $R_{fh} (\theta )$, appearing in Eq. (14), are

$$\begin{aligned} \Omega (\mathbf{3})= & {} e^{i \, \phi _s} \, U_{\mathrm {TB}} \, \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad e^{-3 \, i \, \phi _s} &{}\quad 0\\ 0 &{}\quad 0 &{}\quad -1 \end{array} \right) \;\; \text{ and } \;\;\nonumber \\ R_{13} (\theta )= & {} \left( \begin{array}{ccc} \cos \theta &{}\quad 0 &{}\quad \sin \theta \\ 0 &{}\quad 1 &{}\quad 0\\ -\sin \theta &{}\quad 0 &{}\quad \cos \theta \end{array} \right) \end{aligned}$$

(44)

with

$$\begin{aligned} U_{\mathrm {TB}} = \left( \begin{array}{ccc} \sqrt{\frac{2}{3}} &{}\quad \frac{1}{\sqrt{3}} &{}\quad 0\\ -\frac{1}{\sqrt{6}} &{}\quad \frac{1}{\sqrt{3}} &{}\quad \frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{6}} &{}\quad \frac{1}{\sqrt{3}} &{} \quad -\frac{1}{\sqrt{2}} \end{array} \right) \end{aligned}$$

(45)

and

$$\begin{aligned} \phi _s=\frac{\pi \, s}{n}. \end{aligned}$$

(46)

The matrix $K_\nu$, present in Eq. (14), is set to the identity matrix for concreteness.

The main results of Case 1) are the following:

(a):

the solar mixing angle is constrained by

$$\begin{aligned} \sin ^2\theta _{12} \gtrsim \frac{1}{3} \; , \end{aligned}$$

(47)

(b):

none of the mixing angles depends on the parameters n and s

$$\begin{aligned} \sin ^2\theta _{13}= & {} \frac{2}{3} \, \sin ^2\theta , \;\; \sin ^2\theta _{12} = \frac{1}{2+\cos 2 \theta }, \;\; \nonumber \\ \sin ^2 \theta _{23}= & {} \frac{1}{2} \, \Big ( 1+ \frac{\sqrt{3} \, \sin 2 \theta }{2+\cos 2 \theta } \Big ), \end{aligned}$$

(48)

(c):

the size of the free angle $\theta$ is (mainly) fixed by the measured value of the reactor mixing angle $(\theta _{13})$ and $\theta$ takes two different values in the interval between 0 and $\pi$,

$$\begin{aligned} \theta \approx 0.18 \;\; \text{ and } \;\; \theta \approx 2.96, \end{aligned}$$

(49)

(d):

two approximate sum rules among the mixing angles can be established

$$\begin{aligned}&\sin ^2 \theta _{12} \approx \frac{1}{3} \, \left( 1 + \sin ^2\theta _{13} \right) \;\; \text{ and } \;\;\nonumber \\&\quad \sin ^2\theta _{23} \approx \frac{1}{2} \, \left( 1 \pm \sqrt{2} \, \sin \theta _{13} \right) \end{aligned}$$

(50)

with ± depending on $\theta \lessgtr \pi /2$,

(e):

the Dirac phase $\delta$ and the Majorana phase $\beta$ are both trivial, $\sin \delta =0$ and $\sin \beta =0$,

(f):

the Majorana phase $\alpha$ only depends on the parameter s (the ratio s/n) and its sine reads

$$\begin{aligned} \sin \alpha = - \sin 6 \, \phi _s, \end{aligned}$$

(51)

(g):

for $s=0$ and $s=\frac{n}{2}$, CP is not violated.

3.2 Case 2)

The residual $Z_2$ symmetry in the sector of the neutral states is the same as in Case 1), while the CP transformation X depends on two different parameters

$$\begin{aligned} Z=c^{n/2} \;\; \text{ and } \;\; X=c^s \, d^t \, X_0, \;\; \text{ with } \;\; 0 \le s, t \le n-1.\nonumber \\ \end{aligned}$$

(52)

Like for Case 1), the index n of $G_f$ has to be even. A more convenient choice of parameters than s and t are u and v, which are related to the former by

$$\begin{aligned} u=2 \, s-t \;\; \text{ and } \;\; v=3 \, t. \end{aligned}$$

(53)

The matrix $\Omega (\mathbf{3})$ and the indices f and h of the rotation matrix $R_{fh} (\theta )$ in Eq. (14) read

$$\begin{aligned}&\Omega (\mathbf{3}) = e^{i \, \phi _v/6} \, U_{\mathrm {TB}} \, R_{13} \, \left( -\frac{\phi _u}{2} \right) \, \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad e^{-i \, \phi _v/2} &{}\quad 0\\ 0 &{}\quad 0 &{}\quad -i \end{array} \right) \;\; \text{ and } \;\; \nonumber \\&R_{13} (\theta ) = \left( \begin{array}{ccc} \cos \theta &{}\quad 0 &{}\quad \sin \theta \\ 0 &{}\quad 1 &{}\quad 0\\ -\sin \theta &{}\quad 0 &{}\quad \cos \theta \end{array} \right) \end{aligned}$$

(54)

with

$$\begin{aligned} \phi _u=\frac{\pi \, u}{n} \;\; \text{ and } \;\; \phi _v=\frac{\pi \, v}{n}. \end{aligned}$$

(55)

For the definition of $U_{\mathrm {TB}}$ see Eq. (45). Like for Case 1) we set $K_\nu$ to the identity matrix.

The main features of the mixing pattern of Case 2) are:

(a):

the solar mixing angle has a lower limit identical to the one of Case 1), see Eq. (47),

(b):

the lepton mixing angles depend on the parameters u and n as well as the free angle $\theta$

$$\begin{aligned} \sin ^2 \theta _{13}= & {} \frac{1}{3} \, (1-\cos \phi _u \, \cos 2 \theta ), \;\; \nonumber \\ \sin ^2 \theta _{12}= & {} \frac{1}{2+\cos \phi _u \cos 2 \theta }, \;\; \nonumber \\ \sin ^2 \theta _{23}= & {} \frac{1}{2} \, \left( 1 + \frac{\sqrt{3} \, \sin \phi _u \cos 2\theta }{2+\cos \phi _u \cos 2 \theta } \right) , \end{aligned}$$

(56)

(c):

the size of $\cos \phi _u \, \cos 2 \theta$ (and thus of u/n and $\theta$) is constrained by the measured value of the reactor mixing angle. Taking into account symmetries of the formulae in $(u,\theta )$, discussed in [33], it is sufficient to consider small values of u/n and $\cos 2\theta \approx 1$. The choice $u=0$ is associated with distinctive features (see point (g) below).

(d):

the mixing angles fulfil two (approximate) sum rules: the one already found for Case 1), see first approximate equality in Eq. (50), and

$$\begin{aligned}&6 \, \sin ^2 \theta _{23} \, (1-\sin ^2 \theta _{13}) \nonumber \\&\quad = 3+\sqrt{3} \, \tan \phi _u - 3 \, \left( 1+\sqrt{3} \, \tan \phi _u \right) \, \sin ^2 \theta _{13},\nonumber \\ \end{aligned}$$

(57)

(e):

the Dirac phase $\delta$ and the Majorana phase $\beta$ depend on the parameters u and n as well as on the free angle $\theta$. Information on them is most conveniently given in terms of the CP invariants $J_{\mathrm {CP}}$ and $I_2$ (see Appendix B)

$$\begin{aligned} J_{\mathrm {CP}} = - \frac{\sin 2\theta }{6 \, \sqrt{3}} \;\; \text{ and } \;\; I_2= \frac{1}{9} \, \sin 2 \,\phi _u \, \sin 2 \theta , \end{aligned}$$

(58)

(f):

the Majorana phase $\alpha$ depends, to very good accuracy, only on the parameters v and n (through the ratio v/n)

$$\begin{aligned} \sin \alpha \approx -\sin \phi _v, \end{aligned}$$

(59)

(g):

for the choice $u=0$, the atmospheric mixing angle and the Dirac phase are both maximal, $\sin ^2 \theta _{23}=1/2$ and $|\sin \delta |=1$, while the Majorana phase $\beta$ is trivial, $\sin \beta =0$, and the Majorana phase $\alpha$ exactly fulfils the approximate equality in Eq. (59).

(h):

if $v=0$ is permitted, this leads to a trivial Majorana phase $\alpha$, $\sin \alpha =0$,

(i):

furthermore, three symmetry transformations of the formulae of the lepton mixing parameters (in the parameters u and $\theta$) have been found in [33]. Two of them are independent, e.g.

$$\begin{aligned}&u \; \rightarrow \; u+n,\quad \theta \; \rightarrow \; \frac{\pi }{2} - \theta {:}\nonumber \\&\sin ^2 \theta _{ij}, \, J_{\mathrm {CP}}, \, I_2 \; \text{ are } \text{ invariant } \text{ and } \;\; I_1 \; \text{ changes } \text{ sign; }\nonumber \\&u \; \rightarrow \; 2\, n-u, \theta \; \rightarrow \; \pi -\theta {:}\nonumber \\&\sin ^2 \theta _{13}, \, \sin ^2 \theta _{12}, \, I_1, \, I_2 \; \text{ are } \text{ invariant, } \;\; J_{\mathrm {CP}} \; \text{ changes } \text{ sign }\nonumber \\&\mathrm {and} \; \sin ^2\theta _{23} \; \rightarrow \;\; 1-\sin ^2\theta _{23}. \end{aligned}$$

(60)

3.3 Case 3 a) and Case 3 b.1)

Case 3 a) and Case 3 b.1) are based on a different residual $Z_2$ symmetry in the sector of the neutral states than that of Case 1) and Case 2). This $Z_2$ symmetry depends on the parameter m. Similarly, the CP transformation X depends on the parameter s. The explicit form of the generator Z and of X is

$$\begin{aligned}&Z=b \, c^m \, d^m \;\; \text{ and } \;\; \nonumber \\&X=b \, c^s \, d^{n-s} \, X_0 \;\; \text{ and } \;\; 0 \le m, \, s \le n-1. \end{aligned}$$

(61)

Since Z contains the generator b, Case 3 a) and Case 3 b.1) can only be realised with the flavour symmetry $G_f=\Delta (6 \, n^2)$. The value of the parameter m and, consequently, the choice of the residual $Z_2$ symmetry are strongly constrained by the measured values of the lepton mixing angles.

A possible form of $\Omega (\mathbf{3})$ and the matrix $R_{fh} (\theta )$ are given by

$$\begin{aligned}&\Omega (\mathbf{3}) = e^{i \, \phi _s}\, \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad \omega &{}\quad 0\\ 0 &{}\quad 0 &{}\quad \omega ^2 \end{array} \right) \, U_{\mathrm {TB}} \, \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad e^{-3 \, i \, \phi _s} &{}\quad 0\\ 0 &{}\quad 0 &{}\quad -1 \end{array} \right) \, R_{13} (\phi _m)\nonumber \\&\text{ and } \;\; \nonumber \\&R_{12} (\theta ) = \left( \begin{array}{ccc} \cos \theta &{}\quad \sin \theta &{}\quad 0\\ -\sin \theta &{}\quad \cos \theta &{}\quad 0\\ 0 &{}\quad 0 &{}\quad 1 \end{array} \right) \end{aligned}$$

(62)

with

$$\begin{aligned} \phi _m=\frac{\pi \, m}{n}. \end{aligned}$$

(63)

Again, the matrix $K_\nu$ is set to the identity matrix.

Two viable types of mixing patterns are found [33]: in Case 3 a) the parameter m fixes the values of the atmospheric and reactor mixing angles, while in Case 3 b.1) the parameter m is around n/2 in order to successfully accommodate the solar mixing angle. We first recapitulate the results for Case 3 a) and then those for Case 3 b.1).

3.3.1 Case 3 a)

The relevant properties of the mixing pattern of Case 3 a) are:

(a):

the value of m/n is strongly constrained by the measured value of the reactor mixing angle. This value has to be either close to 0 or to 1. Not only $\sin ^2\theta _{13}$ is fixed by m/n, but also the value of the atmospheric mixing angle

$$\begin{aligned} \sin ^2\theta _{13}= & {} \frac{2}{3} \, \sin ^2 \phi _m \;\; \text{ and } \;\; \nonumber \\ \sin ^2 \theta _{23}= & {} \frac{1}{2} \, \left( 1+ \frac{\sqrt{3} \, \sin 2 \, \phi _m}{2+\cos 2 \, \phi _m} \right) , \end{aligned}$$

(64)

(b):

due to this strong correlation a sum rule can be derived for $\sin ^2 \theta _{13}$ and $\sin ^2 \theta _{23}$

$$\begin{aligned} \sin ^2\theta _{23} \approx \frac{1}{2} \, \left( 1 \pm \sqrt{2} \, \sin \theta _{13} \right) \end{aligned}$$

(65)

with ± for m/n close to 0 or 1, respectively,

(c):

the solar mixing angle depends on the parameter s and on the free angle $\theta$ as well

$$\begin{aligned} \sin ^2\theta _{12} = \frac{1+\cos 2\,\phi _m\,\sin ^2\theta +\sqrt{2}\,\cos \phi _m\,\cos 3\,\phi _s \, \sin 2\theta }{2+\cos 2 \, \phi _m}.\nonumber \\ \end{aligned}$$

(66)

Note that the solar mixing angle can be accommodated to its measured best-fit value for most of the choices of the parameter s. In particular, $\sin ^2\theta _{12}$ is no longer constrained to be larger than 1/3, as for Case 1) and Case 2). For most choices of s two values of the free angle $\theta$, one close to 0 or $\pi$ and another depending on the parameter s, permit an acceptable fit to the measured value of $\sin ^2\theta _{12}$.

(d):

the CP invariants $J_{\mathrm {CP}}$, $I_1$ and $I_2$ depend in general on all parameters, n, m, s and $\theta$,

$$\begin{aligned} J_{\mathrm {CP}}= & {} - \frac{1}{6 \, \sqrt{6}} \, \sin 3 \, \phi _m \, \sin 3 \, \phi _s \, \sin 2 \theta , \nonumber \\ I_1= & {} -\frac{1}{9} \, \cos \phi _m \, \sin 3 \, \phi _s \, \left( 4 \, \cos \phi _m \, \cos 3 \, \phi _s \, \cos 2 \theta \right. \nonumber \\&\left. + \sqrt{2} \, \cos 2 \, \phi _m \, \sin 2 \theta \right) , \nonumber \\ I_2= & {} \frac{4}{9} \, \sin ^2\phi _m \, \sin 3 \, \phi _s \, \sin \theta \, \left( \cos 3\, \phi _s \, \sin \theta \right. \nonumber \\&\left. -\sqrt{2} \, \cos \phi _m \, \cos \theta \right) , \end{aligned}$$

(67)

(e):

approximate values can be found for the sines of the CP phases when the constraints on m/n and $\theta$, arising from accommodating the lepton mixing angles, are used. These are

$$\begin{aligned} |\sin \alpha | \approx |\sin 6 \, \phi _s| \; , \end{aligned}$$

(68)

and

$$\begin{aligned}&\text{ for } \; \theta \approx 0, \, \pi \, : \,\sin \delta \approx 0 \;\; \text{ and } \;\; \sin \beta \approx 0, \nonumber \\&\text{ for } \; \theta \not \approx 0, \, \pi \, : \,|\sin \delta | \approx \left| \frac{3 \, \sin 6 \,\phi _s}{5+ 4 \, \cos 6 \, \phi _s}\right| \;\; \text{ and } \;\; \nonumber \\&\quad |\sin \beta |\approx 2 \, |\sin 6 \, \phi _s| \, \left| \frac{2+\cos 6 \, \phi _s}{5+4 \, \cos 6 \, \phi _s} \right| . \end{aligned}$$

(69)

Note that the magnitude of $\sin \beta$ has an upper limit, $|\sin \beta | \lesssim 0.87$.

(f):

if two values of the free angle $\theta$ permit an acceptable fit to the measured lepton mixing angles for a certain choice of s, the sine of the Majorana phase $\alpha$ for the two different values of $\theta$ has the same magnitude, but opposite sign. If only one value of $\theta$ leads to a good fit to the experimental data, the Majorana phase $\alpha$ is trivial, $\sin \alpha =0$,

(g):

for $s=0$, all CP phases are trivial, i.e. $\sin \alpha =\sin \beta =\sin \delta =0$.

(h):

for the choice $s=\frac{n}{2}$, the free angle $\theta$, that leads to the best accommodation of the measured values of the lepton mixing angles, is $\theta =0$. Consequently, the solar mixing angle is bounded from below, i.e. $\sin ^2\theta _{12} \gtrsim 1/3$, and all CP phases are trivial,

(i):

like for Case 2), three symmetry transformations of the formulae of the lepton mixing parameters in the parameters m, s and the free angle $\theta$ have been found in [33]. Two of them are independent, e.g. 

$$\begin{aligned}&s\; \rightarrow \; n-s,\quad \theta \; \rightarrow \; \pi -\theta \! :\nonumber \\&\sin ^2\theta _{ij} \; \text{ are } \text{ invariant } \text{ and } \; J_{\mathrm {CP}}, \, I_1, \, I_2 \; \text{ change } \text{ sign; } \nonumber \\&m \; \rightarrow \; n-m,\quad \theta \; \rightarrow \; \pi -\theta \! :\nonumber \\&\sin ^2\theta _{13}, \, \sin ^2\theta _{12}, \, I_1, \, I_2 \; \text{ are } \text{ invariant, } \; J_{\mathrm {CP}} \; \text{ changes } \text{ sign } \nonumber \\&\text{ and } \; \sin ^2\theta _{23} \; \rightarrow 1-\sin ^2\theta _{23}. \end{aligned}$$

(70)

3.3.2 Case 3 b.1)

The lepton mixing pattern of Case 3 b.1) arises from the matrices $\Omega (\mathbf{3})$ and $R_{12} (\theta )$ in Eq. (62), if these are multiplied from the right with the cyclic permutation matrix $P_{\mathrm {cyc}}$

$$\begin{aligned}&P_{\mathrm {cyc}} = \left( \begin{array}{ccc} 0 &{}\quad 1 &{}\quad 0\\ 0 &{}\quad 0 &{}\quad 1\\ 1 &{}\quad 0 &{}\quad 0 \end{array} \right) ,\; \nonumber \\&\text{ i.e. } \;\; U_{\mathrm {PMNS}}=U_\nu =\Omega (\mathbf{3}) \, R_{12} (\theta ) \, P_{\mathrm {cyc}}. \end{aligned}$$

(71)

This cyclic permutation corresponds to a re-ordering of the columns of the PMNS mixing matrix. The properties of the lepton mixing pattern of Case 3 b.1) can be summarised as follows:

(a):

all lepton mixing parameters depend on n, m, s and the free angle $\theta$,

$$\begin{aligned}&\sin ^2 \theta _{13}= \frac{1}{3} \, \nonumber \\&\quad \times \left( 1+ \cos 2 \, \phi _m \, \sin ^2\theta +\sqrt{2} \, \cos \phi _m \, \cos 3 \, \phi _s \, \sin 2 \theta \right) , \nonumber \\&\sin ^2 \theta _{23}= \frac{1}{2} \, \nonumber \\&\quad \times \left( 1+ \frac{2 \, \sqrt{3}\, \sin \phi _m\,\sin \theta \, (\sqrt{2}\,\cos 3 \, \phi _s \, \cos \theta -\cos \phi _m \, \sin \theta )}{2-\cos 2 \, \phi _m \,\sin ^2\theta -\sqrt{2}\,\cos \phi _m\,\cos 3 \, \phi _s \, \sin 2 \theta } \right) , \nonumber \\&\sin ^2\theta _{12} = 1 - \frac{2 \, \sin ^2 \phi _m}{2-\cos 2 \, \phi _m \,\sin ^2\theta -\sqrt{2}\,\cos \phi _m \, \cos 3 \, \phi _s \, \sin 2\theta }\nonumber \\ \end{aligned}$$

(72)

and

$$\begin{aligned} J_{\mathrm {CP}}= & {} -\frac{1}{6 \, \sqrt{6}} \, \sin 3 \, \phi _m \, \sin 3 \, \phi _s \, \sin 2 \theta , \nonumber \\ I_1= & {} -\frac{4}{9} \, \sin ^2 \phi _m \, \sin 3 \, \phi _s \, \sin \theta \, \left( \cos 3 \, \phi _s \, \sin \theta \right. \nonumber \\&\left. -\sqrt{2} \, \cos \phi _m \, \cos \theta \right) , \nonumber \\ I_2= & {} -\frac{4}{9} \, \sin ^2 \phi _m \, \sin 3 \, \phi _s \, \cos \theta \, \left( \cos 3 \, \phi _s \, \cos \theta \right. \nonumber \\&\left. +\sqrt{2} \, \cos \phi _m \, \sin \theta \right) , \end{aligned}$$

(73)

(b):

the parameter m is strongly constrained by the measured value of $\sin ^2\theta _{12}$, i.e. $m\approx \frac{n}{2}$,

(c):

for $m=\frac{n}{2}$, two approximate sum rules among the lepton mixing angles are found

$$\begin{aligned}&\sin ^2\theta _{12} \approx \frac{1}{3} \, \left( 1-2 \, \sin ^2\theta _{13} \right) \;\; \text{ and } \;\;\nonumber \\&\quad \sin ^2\theta _{23} \approx \frac{1}{2} \left( 1+\sqrt{\frac{2}{3}} \, \frac{\cos 3 \, \phi _s \, \sin 2 \theta _0}{1-\sin ^2 \theta _{13}} \right) \end{aligned}$$

(74)

with $\theta _0 \approx 1.31$ or $\theta _0 \approx 1.83$, constrained by the measured value of the reactor mixing angle,

(d):

for $m= \frac{n}{2}$ and $s= \frac{n}{2}$, the atmospheric mixing angle is maximal, $\sin ^2\theta _{23}= \frac{1}{2}$,

(e):

for $m= \frac{n}{2}$, the Majorana phases only depend on the parameter s (the ratio s/n), and have the same magnitude,

$$\begin{aligned} \sin \alpha =\sin \beta =-\sin 6 \, \phi _s \end{aligned}$$

(75)

and the Dirac phase fulfils the approximate relation

$$\begin{aligned} \sin \delta \approx \pm \sin 3 \,\phi _s \;\; \text{ with } \; \pm \; \text{ referring } \text{ to } \;\; \theta \lessgtr \pi /2. \end{aligned}$$

(76)

Taking into account the constraints on the free angle $\theta$ and the parameter s, arising from the experimental data on lepton mixing angles, the magnitude of the sine of the Dirac phase is bounded from below, $|\sin \delta | \gtrsim 0.71$,

(f):

for $m= \frac{n}{2}$ and $s= \frac{n}{2}$, the Dirac phase is maximal, $|\sin \delta |=1$, while both Majorana phases are trivial, $\sin \alpha =0$ and $\sin \beta =0$,

(g):

for $s=0$, CP is not violated,

(h):

for Case 3 b.1) the same symmetry transformations hold as for Case 3 a), see point (i) in Sect. 3.3.1, Eq. (70).

4 Impact of heavy sterile states of the (3, 3) ISS on lepton mixing

As already mentioned in Sect. 2.2, there are two possible effects that can have an impact on lepton mixing: the inclusion of the subleading contribution $m_\nu ^1$ to the light neutrino mass matrix $m_\nu$ and effects of non-unitarity of ${\widetilde{U}}_\nu$, which are encoded in $\eta _{\alpha \beta }$. A numerical analysis of examples for each case, Case 1) through Case 3 b.1), can be found in Sect. 5 and confirms the analytical results, which we proceed to discuss.

4.1 Subleading contribution to the light neutrino mass matrix

When plugging in the form of the matrices $m_D$, $M_{NS}$ and $\mu _S$ for option 1, see Eqs.  (30, 31, 32, 34), the subleading contribution to the light neutrino mass matrix, shown in Eq. (18), takes a simple form:

$$\begin{aligned} m_\nu ^1 = - \frac{y_0^4 \, v^4}{4 \, M_0^4} \, \mu _S = - \frac{y_0^4 \, v^4}{4 \, M_0^4} \, \, U_S^\star \, \left( \begin{array}{ccc} \mu _1 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad \mu _2 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad \mu _3 \end{array} \right) \, U_S^\dagger .\nonumber \\ \end{aligned}$$

(77)

Comparing with the leading order contribution $m_\nu$, found in Eq. (35), we see that $m_\nu ^1$ has exactly the same form in flavour space and is suppressed by a factor $\frac{y_0^2 \, v^2}{2 \, M_0^2}$. Thus, this subleading contribution does not introduce any change in the lepton mixing parameters and only slightly corrects the values of the light neutrino masses, e.g. for $y_0 \sim 1$ and $M_0 \sim 1000 \, \mathrm {GeV}$ the correction is around 0.03 with respect to the leading order result, see Eq. (37). Such a correction can be compensated by re-adjusting the values of the parameters $\mu _i$.

4.2 Effects of non-unitarity of ${\widetilde{U}}_\nu$

The deviation from unitarity of ${\widetilde{U}}_\nu$ is encoded in $\eta$, see Eqs. (23, 26). For option 1, the form of $\eta$ turns out to be flavour-diagonal and flavour-universal, since both $m_D$ and $M_{NS}$ have this property, see Eqs. (31, 32)

$$\begin{aligned} \eta = \frac{y_0^2 \, v^2}{4 \, M_0^2} \, \mathbb {1}\equiv \eta _0 \, \mathbb {1}. \end{aligned}$$

(78)

Furthermore, it is independent of the particular case, Case 1) through Case 3 b.1), which we confirm numerically.

For $y_0 \sim 1$ and $M_0 \sim 1000 \, \mathrm {GeV}$ we have $\eta _0 \sim 0.015$, while for $y_0 \sim 0.1$ it is suppressed by further two orders of magnitude (at constant $M_0$). The features of being flavour-diagonal and flavour-universal are numerically confirmed. The size of $\eta$ and its dependence on $y_0^2$ as well as $\frac{1}{M_0^2}$ are also very well fulfilled. For further details and a comparison with experimental bounds on $\eta _{\alpha \beta }$ see Sect. 5.1.

Since $\eta$ is flavour-diagonal as well as flavour-universal and $\eta _0$ is positive, the presence of $\eta$ effectively leads to a suppression of all elements of $U_0 = U_S$, see Eqs. (23, 36). We can thus easily estimate the deviations expected in the results for the lepton mixing parameters (mixing angles and CP invariants/CP phases) extracting them in the same way as for the unitary case, i.e. the (3, 3) ISS framework at leading order and the model-independent scenario (MIS).¹¹ We consider relative deviations between the non-unitary results, $(\sin ^2\theta _{ij})_{\mathrm {ISS}}$, $(J_{\mathrm {CP}})_{\mathrm {ISS}}$ and $(I_i)_{\mathrm {ISS}}$, and the unitary ones, $(\sin ^2\theta _{ij})_{\mathrm {MIS}}$, $(J_{\mathrm {CP}})_{\mathrm {MIS}}$ and $(I_i)_{\mathrm {MIS}}$,¹²

$$\begin{aligned} \Delta \sin ^2\theta _{ij}= & {} \frac{(\sin ^2\theta _{ij})_{\mathrm {ISS}} - (\sin ^2\theta _{ij})_{\mathrm {MIS}}}{(\sin ^2\theta _{ij})_{\mathrm {MIS}}}, \;\; \nonumber \\ \Delta J_{\mathrm {CP}}= & {} \frac{(J_{\mathrm {CP}})_{\mathrm {ISS}}-(J_{\mathrm {CP}})_{\mathrm {MIS}}}{(J_{\mathrm {CP}})_{\mathrm {MIS}}} \;\; \text{ and } \;\; \nonumber \\ \Delta I_i= & {} \frac{(I_i)_{\mathrm {ISS}}-(I_i)_{\mathrm {MIS}}}{(I_i)_{\mathrm {MIS}}} \end{aligned}$$

(79)

and alike for the sines of the CP phases $\delta$, $\alpha$ and $\beta$. In doing so, we can find formulae for the relative deviations that are valid for all cases, Case 1) through Case 3 b.1). The exact numerical values of these deviations can in general (slightly) depend on the chosen case and other parameters, such as the index of $G_f$, the choice of the residual $Z_2$ symmetry in the sector of the neutral states, and the value of the free angle $\theta _S$. We comment on this in the numerical analysis to be carried in Sect. 5.

For $\Delta \sin ^2 \theta _{ij}$ we have

$$\begin{aligned}&\Delta \sin ^2 \theta _{13} \approx -2 \, \eta _0, \nonumber \\&\Delta \sin ^2 \theta _{12} \approx - \frac{2 \, \eta _0}{1-|U_{e3}|^2} \approx -2.04 \, \eta _0,\nonumber \\&\Delta \sin ^2 \theta _{23} \approx - \frac{2 \, \eta _0}{1-|U_{e3}|^2} \approx -2.04 \, \eta _0 \end{aligned}$$

(80)

for $|U_{e3}|^2 \approx 0.022$ [76]. For the CP invariants $J_{\mathrm {CP}}$, $I_1$ and $I_2$ we find

$$\begin{aligned} \Delta J_{\mathrm {CP}} \approx - 4 \, \eta _0, \;\; \Delta I_1 \approx - 4 \, \eta _0, \;\; \Delta I_2 \approx - 4 \, \eta _0. \end{aligned}$$

(81)

With this information we can also extract $\Delta \sin \delta$, $\Delta \sin \alpha$ and $\Delta \sin \beta$ and arrive at

$$\begin{aligned}&\Delta \sin \delta \approx -2.82 \, \eta _0, \;\; \Delta \sin \alpha \approx -2.95 \, \eta _0, \;\;\nonumber \\&\quad \Delta \sin \beta \approx -2.95 \, \eta _0 \end{aligned}$$

(82)

for $|U_{e2}|^2 \approx 0.30$, $|U_{e3}|^2 \approx 0.022$ and $|U_{\mu 3}|^2 \approx 0.56$ [76]. For $y_0 \sim 1$ and $M_0 \sim 1000 \, \mathrm {GeV}$ we expect

$$\begin{aligned}&\Delta \sin ^2 \theta _{ij} \approx -0.03 \; , \;\; \Delta J_{\mathrm {CP}} \approx \Delta I_i \approx -0.06, \;\; \nonumber \\&\quad \Delta \sin \delta \approx -0.042, \; \Delta \sin \alpha \approx \Delta \sin \beta \approx -0.044. \end{aligned}$$

(83)

Due to the suppression of all elements of $U_0 = U_S$, all relative deviations are expected to be negative. Furthermore, their size slightly depends on the considered quantity and is generally not expected to exceed values of a few percent. These estimates are confirmed numerically, as we discuss in Sect. 5. It is important to note that certain features, like the vanishing of the sine and the periodicity of some of the CP phases in terms of the group theory parameters, remain preserved exactly, since the flavour structure of the light neutrino mass matrix is not changed and the deviation from unitarity only amounts to a common rescaling of all elements of the PMNS mixing matrix.

Furthermore, we can estimate the deviations in the (approximate) sum rules induced by effects of non-unitarity of the lepton mixing matrix, such as the ones in Eq. (50). These are discussed in turn for each of the cases, Case 1) through Case 3 b.1).

4.3 Case 1)

Two approximate sum rules have been found for Case 1), see Eq. (50). The effects of non-unitarity of the lepton mixing matrix on these are expected to be as follows: for the first sum rule, relating the solar and the reactor mixing angle, using the best-fit value $|U_{e 3}|^2\approx 0.022$ [76], we have

$$\begin{aligned} \Delta \Sigma _1 \approx -2 \, \eta _0 \, \left( \frac{1+|U_{e3}|^4}{1-|U_{e3}|^4} \right) \approx -2 \, \eta _0 \end{aligned}$$

(84)

with $\Delta \Sigma _1$ corresponding to the relative deviation of the non-unitary result from the unitary one and defined as

$$\begin{aligned} \Delta \Sigma _1= & {} \frac{\left( \frac{3 \, \sin ^2 \theta _{12}}{1 + \sin ^2\theta _{13}} \right) _{\mathrm {ISS}} - \left( \frac{3 \, \sin ^2 \theta _{12}}{1 + \sin ^2\theta _{13}} \right) _{\mathrm {MIS}}}{\left( \frac{3 \, \sin ^2 \theta _{12}}{1 + \sin ^2\theta _{13}} \right) _{\mathrm {MIS}}} \;\; \;\; \nonumber \\&\quad \text{ with }\left( \frac{3 \, \sin ^2 \theta _{12}}{1 + \sin ^2\theta _{13}} \right) _{\mathrm {MIS}} \approx 1 \;\; \text{ from } \text{ Eq. } ~(50),\nonumber \\ \end{aligned}$$

(85)

while the deviation for the second sum rule, the one involving the atmospheric and the reactor mixing angle, is of the form

$$\begin{aligned} \Delta \Sigma _2\approx & {} - \sqrt{2} \, \eta _0 \, \left( \frac{\sqrt{2} \pm |U_{e3}|\, (1+|U_{e3}|^2)}{(1\pm \sqrt{2} \, |U_{e3}|)\, (1-|U_{e3}|^2)} \right) \nonumber \\\approx & {} -1.87 \, (-2.31) \, \eta _0 \end{aligned}$$

(86)

for $+ (-)$. $\Delta \Sigma _2$ is defined analogously to $\Delta \Sigma _1$ with the help of the second approximate sum rule in Eq. (50). The different signs refer to the different signs in the sum rule. We note that none of the relative deviations, $\Delta \Sigma _1$ and $\Delta \Sigma _2$, depends on the parameters n, s or on the precise value of the free angle $\theta _S$, up to the sign in $\Delta \Sigma _2$. For $y_0 \sim 1$ and $M_0 \sim 1000$ GeV we expect these to be

$$\begin{aligned} \Delta \Sigma _1 \approx -0.03 \;\; \text{ and } \;\; \Delta \Sigma _2 \approx -0.028 \, (-0.035) \end{aligned}$$

(87)

for $+ (-)$ from the expression for $\Delta \Sigma _2$ in Eq. (86).

4.4 Case 2)

For Case 2) we also have two approximate sum rules: one which coincides with the first sum rule of Case 1) and another one, relating the atmospheric and the reactor mixing angles, shown in Eq. (57). The effects of non-unitarity (of the PMNS mixing matrix) on the latter one are estimated to be of the order of

$$\begin{aligned} \Delta \Sigma _3 \approx - 2 \, \eta _0 \, \left( \frac{\sqrt{3}+\tan \phi _u}{\sqrt{3}\, (1-|U_{e3}|^2) + (1-3\, |U_{e3}|^2) \, \tan \phi _u} \right) ,\nonumber \\ \end{aligned}$$

(88)

where $\Delta \Sigma _3$ is defined in the analogous way as $\Delta \Sigma _1$. The form of $\Delta \Sigma _3$ can be simplified by remembering that u/n is required to be small and thus we expand in $\phi _u=\frac{\pi \, u}{n}$ up to the linear order. At the same time, we use the best-fit value for $|U_{e3}|^2 \approx 0.022$ [76] so that we have

$$\begin{aligned} \Delta \Sigma _3 \approx -2.05 \, \eta _0 \, (1+ 0.026 \, \phi _u). \end{aligned}$$

(89)

This shows that there is only a very mild dependence of $\Delta \Sigma _3$ on $\phi _u = \frac{\pi \, u}{n}$. Furthermore, there is no explicit dependence of $\Delta \Sigma _3$ on the parameter v and the free angle $\theta _S$. Numerically we find for $y_0 \sim 1$ and $M_0 \sim 1000$ GeV that

$$\begin{aligned} \Delta \Sigma _3 \approx -0.031, \end{aligned}$$

(90)

which is of a size very similar to the other relative deviations.

4.5 Case 3 a) and Case 3 b.1)

For Case 3 a) the approximate sum rule, found in Eq. (65), is actually identical to the second sum rule for Case 1), see Eq. (50), taking into account the different signs in both of them. We thus expect very similar results also for Case 3 a).

For Case 3 b.1) two approximate sum rules are derived for $m=\frac{n}{2}$, see Eq. (74). For the first of these two, we find as relative deviation of the non-unitary result from the unitary one

$$\begin{aligned} \Delta \Sigma _4 \approx -2 \, \eta _0 \, \left( \frac{1-2 \, |U_{e3}|^4}{1-3 \, |U_{e3}|^2+2 \, |U_{e3}|^4} \right) \approx -2.14 \, \eta _0,\nonumber \\ \end{aligned}$$

(91)

while for the second one we have

$$\begin{aligned} \Delta \Sigma _5\approx & {} - 2 \, \eta _0 \, \left( \frac{\sqrt{3}+ \sqrt{2} \, \cos 3 \, \phi _s \, \sin 2 \, \theta _0}{\sqrt{3} \, (1-|U_{e3}|^2) + \sqrt{2} \, \cos 3 \, \phi _s \, \sin 2 \, \theta _0} \right) \nonumber \\\approx & {} -2.05 \, \eta _0 {\mp } 0.019 \, \eta _0 \, \cos 3 \, \phi _s, \end{aligned}$$

(92)

where we have again used $|U_{e3}|^2 \approx 0.022$ [76] and $\theta _0 \approx \frac{\pi }{2} \pm \epsilon$ with $\epsilon \approx 0.26$, cf. text below Eq. (74). We thus see that the relative deviation $\Delta \Sigma _5$ only weakly depends on the value of the parameter s, related to the chosen CP transformation X. Furthermore, we infer that neither $\Delta \Sigma _4$ nor $\Delta \Sigma _5$ depends strongly on the parameter n or the free angle $\theta _S$. Using $y_0 \sim 1$ and $M_0 \sim 1000$ GeV, we have for the two relative deviations

$$\begin{aligned} \Delta \Sigma _4 \approx -0.032 \;\; \text{ and } \;\; \Delta \Sigma _5 \approx -0.031. \end{aligned}$$

(93)

5 Numerical analysis

In this section, we study numerically the impact of the heavy sterile states of the (3, 3) ISS framework on the results for the lepton mixing parameters, and if available, on the approximate sum rules among these. We do so for each of the different cases, Case 1) through Case 3 b.1), for some viable choices of the group theory parameters, e.g. the index n of the flavour symmetry $G_f$. We also compare these findings to the analytical estimates, presented in Sect. 4.

Before detailing results for the different cases in Sects. 5.2–5.5, we present constraints on the Dirac neutrino Yukawa coupling $y_0$, the mass scale $M_0$ of the heavy sterile states, as well as on the parameters $\mu _i$, emphasising the role of the bounds on the unitarity of the PMNS mixing matrix.

5.1 Symmetry endowed (3,3) ISS: setup and unitarity constraints on ${\widetilde{U}}_\nu$

We begin by briefly discussing the constraints arising from the violation of unitarity of the PMNS mixing matrix ${\widetilde{U}}_\nu$, as encoded in the matrix $\eta$. As can be seen from Eq. (78), $\eta$ is determined by the chosen regimes for $y_0$ and $M_0$, which characterise the impact of the heavy sterile states on the lepton mixing parameters. Thus, the experimental limits on the quantities $\eta _{\alpha \beta }$, $\alpha ,\beta =e,\mu ,\tau$, are at the source of the most important constraints on the present (3,3) ISS framework.

Before discussing how the limits on $\eta _{\alpha \beta }$ crucially constrain $y_0$ and hence the combination of $y_0$ and $M_0$, let us first emphasise two points: we have checked numerically that the form of the quantities $\eta _{\alpha \beta }$ does not depend on the specific case, Case 1) through Case 3 b.1), as expected from the analytical estimate in Eq. (78); furthermore, we also confirm numerically that the matrix $\eta$ is flavour-diagonal and flavour-universal, and that $\eta _0$ is proportional to $y_0^2$ (and inversely proportional to $M_0^2$), as can be also seen from Eq. (78). The (indirect) experimental constraints on $\eta _{\alpha \beta }$ are taken from [77] and are given by¹³

$$\begin{aligned}&\left| \eta _{\alpha \beta }\right| \le \left( \begin{array}{ccc} 1.3 \times 10^{-3} &{}\quad 1.2 \times 10^{-5} &{}\quad 1.4 \times 10^{-3}\\ 1.2 \times 10^{-5} &{}\quad 2.2 \times 10^{-4} &{}\quad 6.0 \times 10^{-4}\\ 1.4 \times 10^{-3} &{}\quad 6.0 \times 10^{-4} &{}\quad 2.8 \times 10^{-3} \end{array} \right) \;\; \nonumber \\&\quad \text{ at } \text{ the } 1\,\sigma \text{ level. } \end{aligned}$$

(94)

As can be verified, the diagonal element subject to the strongest experimental bounds is $\eta _{\mu \mu }$, $\left| \eta _{\mu \mu }\right| \le 2.2 \, (4.4) \, [6.6] \times 10^{-4}$ at the $1 (2) [3]\, \sigma$ level. We thus use this limit in the subsequent analysis.

The maximal size of the Yukawa coupling $y_0$, compatible with the experimental constraints on $\eta _{\alpha \beta }$, can be read from the left plot in Fig. 1: for $y_0$ as small as $y_0=0.1$, the unitarity constraints can be evaded for values of $M_0$ as low as $M_0 \gtrsim 500$ GeV (at the $3 \, \sigma$ level); larger values, $y_0=0.5$, already require $M_0 \gtrsim 2400$ GeV, and for $y_0=1$ one must have $M_0 \gtrsim 4800$ GeV in order to be in agreement with the bounds of Eq. (94) at the $3 \, \sigma$ level, i.e. $|\eta _{\mu \mu }|\lesssim 6.6\times 10^{-4}$. This is illustrated in the right panel of Fig. 1 by an exclusion plot in the ($M_0-y_0$) plane. The subsequent numerical analyses will rely on regimes of $y_0$ and $M_0$ compatible with experimental data at the $3\, \sigma$ level,¹⁴ and regimes in conflict with experimental bounds on $\eta _{\alpha \beta }$ will be clearly indicated in the discussion.

Fig. 1

Constraints from unitarity of ${\widetilde{U}}_\nu$. Left plot: $\eta _{\alpha \alpha }$ with respect to the mass scale $M_0$ (in GeV) for different values of the Yukawa coupling: $y_0=0.05$ (purple line), $y_0=0.1$ (orange line), $y_0=0.3$ (red line), $y_0=0.5$ (blue line) and $y_0=1$ (green line). The grey-shaded regions denote the areas excluded by the strongest constraint on the flavour-diagonal entries of $\eta$ (arising from $\eta _{\mu \mu }$) at $1 \, \sigma$ level (light grey) [77] and $3 \, \sigma$ level (dark grey). Right plot: Disfavoured regions of the ($M_0-y_0$) plane, with $M_0$ given in GeV, due to conflict with experimental bounds on $\eta _{\alpha \alpha }$, at $1 \, \sigma$, $2 \, \sigma$ and $3 \, \sigma$ (respectively denoted by light, medium and dark blue)

In view of the above, we will in general assume that the mass scale $M_0$ varies in the range

$$\begin{aligned} 500 \, \mathrm {GeV} \lesssim M_0 \lesssim 5000 \, \mathrm {GeV}. \end{aligned}$$

(95)

Although mostly lying beyond future collider reach [78], the chosen range for $M_0$ (and thus for $M_{NS}$ and the heavy mass spectrum) is motivated by its phenomenological interest, as it is in general associated with extensive observational imprints, being thus indirectly accessible in numerous dedicated facilities [46,47,48,49,50,51,52].

Concerning the Yukawa coupling $y_0$, and following the results displayed in Fig. 1, we will in general illustrate our results for two different values of the Yukawa coupling $y_0$,

$$\begin{aligned} y_0= 0.5 \;\; \text{ and } \;\; y_0=0.1. \end{aligned}$$

(96)

Nevertheless, we will exceptionally consider larger values of the Yukawa coupling $y_0=1$, in order to better illustrate the effects of the deviations from unitarity of the PMNS mixing matrix. These cases will be clearly identified in the discussion; unless otherwise stated, disfavoured regimes associated with bounds on $\eta _{\alpha \beta }$ will be indicated by a grey-shaded area in the corresponding plots.

Finally, we consider the free parameters $\mu _i$. As can be seen from Eq. (37), in the case of option 1, $\mu _i$ are directly proportional to the light neutrino masses $m_i$. Thus, they are experimentally constrained by the measured mass squared differences and by the bound on the sum of the light neutrino masses coming from cosmology. The latest experimental data are collected in Appendix C. We notice that in our numerical study, the two mass squared differences are always adjusted to their experimental best-fit value [76].

A few comments are still in order concerning the light neutrino mass spectrum – the value of the lightest neutrino mass $m_0$, and the ordering (NO vs. IO). Regarding $m_0$, we have verified that the results for the lepton mixing parameters are always independent of its choice. Throughout this section, we have thus fixed its value to

$$\begin{aligned} m_0 = 0.001 \, \mathrm {eV}. \end{aligned}$$

(97)

Furthermore, we note that we have performed the numerical analysis for both NO and IO, and no (numerically significant) differences were found, neither for the relative deviations of the lepton mixing parameters, nor for the (approximate) sum rules. Accordingly, all the results of this section will be only illustrated for the case of a NO light neutrino mass spectrum. However, notice that upon discussion of the prospects of the current framework concerning $0\nu \beta \beta$ decay in Sect. 6, we will consider both orderings of the mass spectrum, and also vary $m_0$.

Leading to the fits presented in the following subsections, we only consider experimental constraints on the lepton mixing angles and the two mass squared differences, but not on the CP phase $\delta$, since the latter is only very mildly experimentally constrained (a summary of the relevant neutrino oscillation data is given in Appendix C). Additional information on the numerical fit procedure can be found in Appendix D.

5.2 Case 1)

In order to scrutinise the effects of the (3, 3) ISS framework and its heavy states on the lepton mixing parameters, we choose a value of the index n that allows studying several different values of the parameter s (and thus CP transformations X) for Case 1). In this way, the behaviour of the Majorana phase $\alpha$, see Eq. (51), can be studied systematically. Concretely, in the following we use

$$\begin{aligned} n=26 \;\; \text{ and } \;\; 0 \le s \le 25. \end{aligned}$$

(98)

Based on the results obtained in the model-independent scenario (see Sect. 3.1), and the analytical estimates of the effects due to the heavy sterile states of the (3, 3) ISS framework carried in Sect. 4, only the CP phase $\alpha$ is expected to show a dependence on the parameters n and s (through the ratio s/n). This is confirmed by our numerical analysis. Without loss of generality we thus set $s=1$ to study the relative deviations $\Delta \sin ^2\theta _{12}$ and $\Delta \sin ^2\theta _{23}$. These are shown in Fig. 2, respectively in the left and right plots, as a function of $M_0$, which determines the scale of the heavy mass spectrum.

Fig. 2

Case 1). Relative deviation of $\sin ^2\theta _{12}$ (left) and $\sin ^2\theta _{23}$ (right) as obtained for option 1 of the (3, 3) ISS from the corresponding values derived in the model-independent scenario, as a function of $M_0$ (in GeV). For concreteness, we have fixed $s=1$ and $n=26$. The curves are associated with distinct values of the Yukawa coupling $y_0$: the orange (dashed) curve corresponds to $y_0=0.1$ and the blue (solid) one to $y_0=0.5$. A grey-shaded area denotes regimes disfavoured due to conflict with experimental bounds (see detailed discussion in Sect. 5.1)

We notice that their sign and size is consistent with the estimate found in Eq. (83).¹⁵ The relative deviation of the reactor mixing angle, $\Delta \sin ^2\theta _{13}$, is not shown and does not fulfil the expectations from the analytical estimate, since it turns out to be positive and always below $0.5\%$ for values of $y_0 \lesssim 0.5$ and $M_0 \gtrsim 500$ GeV. This is a consequence of having $\theta _{13}$ driving the fit to determine $\theta _S$, due to its associated experimental precision, see Appendix C. Consequently, we find for $\theta _S$ values around 0.19, which are slightly larger than those obtained in the model-independent scenario, see Eq. (49). We note that in the plots shown here, we always have $\theta _S < \pi /2$, since this leads to a much better agreement with the experimentally preferred value of the atmospheric mixing angle: $\sin ^2 \theta _{23} \approx 0.604$ to be compared to the experimental values $\sin ^2 \theta _{23} = 0.570^{+0.018} _{-0.024}$ for light neutrinos with NO and $\sin ^2 \theta _{23} = 0.575^{+0.017} _{-0.021}$ for light neutrinos with IO [76].

Fig. 3

Case 1). Left plot: Relative deviation of $\sin \alpha$ as obtained for option 1 of the (3, 3) ISS framework from the corresponding model-independent prediction, with respect to $M_0$ (in GeV). Line and colour code as in Fig. 2. Right plot: $\sin \alpha$ with respect to s/n (fixing $n=26$ and continuously varying $0\le s < 26$). The black (dashed) curve displays the result for $\sin \alpha$ obtained in the model-independent scenario, see Eq. (51). The coloured (solid) curves refer to distinct values of $M_0$: blue for $M_0=500$ GeV, orange for $M_0=1000$ GeV and green for $M_0=5000$ GeV. We have chosen $y_0=1$ in order to better display the deviation from the model-independent scenario (notice that such a value of $y_0$ requires $M_0 \gtrsim 4800$ GeV to comply with the experimental bounds on $\eta _{\alpha \alpha }$ at the $3\, \sigma$ level, cf. Sect. 5.1)

Moving on to the relative deviation of the Majorana phase $\alpha$, we note that also in this case the size, sign and behaviour of the relative deviation $\Delta \sin \alpha$ (depending on $y_0$ and $M_0$) does not depend on the actual choice of the parameter s. Thus, we have again taken $s=1$. In the left plot in Fig. 3, we present the relative deviation of $\sin \alpha$ as obtained for option 1 of the (3, 3) ISS framework from the corresponding model-independent prediction, with respect to $M_0$ (in GeV). Comparing the maximal size of the relative deviation of $\sin \alpha$ ($\Delta \sin \alpha$) with the ones of the solar and the atmospheric mixing angles, $\Delta \sin ^2\theta _{12}$ and $\Delta \sin ^2\theta _{23}$, previously displayed in Fig. 2, we confirm that the latter are slightly smaller than the former, as expected from the analytical estimate in Eq. (83). The right plot in Fig. 3 illustrates the suppression of the value of $\sin \alpha$ depending on s/n for three different values of $M_0$, $M_0=500$ GeV, 1000 GeV and 5000 GeV, and these are compared to the result expected in the model-independent scenario, see Eq. (51). We have chosen here $y_0=1$ in order to enhance the visibility of the deviations between the model-independent scenario and the (3,3) ISS presented in this plot, although such a large value of the Yukawa coupling requires $M_0$ to be at least $M_0 \gtrsim 4800$ GeV in order to comply with the experimental bounds on the quantities $\eta _{\alpha \beta }$, see Sect. 5.1. Beyond this suppression of the value of $\sin \alpha$, we note that the periodicity in s/n is still the same, independently of the effects of non-unitarity of ${\widetilde{U}}_\nu$, confirming the analytical estimates of Sect. 4.2. We have also numerically verified the analytical expectation that the Dirac phase $\delta$ as well as the Majorana phase $\beta$ remain trivial, i.e. $\sin \delta =0$ and $\sin \beta =0$.

Fig. 4

Case 1). Validity check of approximate sum rules for option 1 of the (3, 3) ISS framework with respect to the mass $M_0$ (in GeV). Line and colour code as in Fig. 2. We note that for the second sum rule (right plot) we focus on the approximate sum rule with a plus sign, since we present results for $\theta _S < \pi /2$, see Eq. (50) and below

Finally, we address the validity of the two approximate sum rules, see Eq. (50). As can be seen from the plots in Fig. 4, deviations do not exceed the level of $-3\%$, in agreement with the analytical estimate. Furthermore, we numerically confirm that the maximally achieved relative deviation is slightly larger for the first sum rule than for the second, for $\theta _S < \pi /2$. We also note that for large values of $M_0$, where effects of the non-unitarity of ${\widetilde{U}}_\nu$ should be suppressed, both ratios related to the two different sum rules become slightly larger than one. This is consistent with the fact that these sum rules only hold approximately.

5.3 Case 2)

Fig. 5

Case 2). Relative deviations $\Delta \sin \delta$ (left plot) and $\Delta \sin \beta$ (right plot), as obtained for option 1 of the (3, 3) ISS framework, from the values obtained in the model-independent scenario, with respect to the mass $M_0$ (in GeV). The concrete choice of v is irrelevant and thus we have set $v=3$. Line and colour code as in Fig. 2

In our numerical study, we choose as representative values of the index n and of the parameter u

$$\begin{aligned} n=14 \;\; \text{ and } \;\; u=1, \end{aligned}$$

(99)

also commenting on results for the choices $u=-1$, $u=15$ as well as $u=0$ in order to comprehensively analyse the features of Case 2). For the parameter v, we consider all permitted values according to the relations in Eqs. (52, 53) and the chosen value of u, e.g. for $u=1$ we have

$$\begin{aligned} v= 3, \, 9, \, 15, \, 21, \, 27, \, 33, \, 39. \end{aligned}$$

(100)

We start by discussing the relative deviations of $\sin ^2\theta _{ij}$. The results for $\Delta \sin ^2\theta _{12}$ and $\Delta \sin ^2\theta _{23}$ are consistent with the analytical expectations, see Eq. (80). Indeed, the plots for $\Delta \sin ^2\theta _{12}$ and $\Delta \sin ^2\theta _{23}$ look very similar to those presented in Fig. 2 for Case 1). However, the relative deviation $\Delta \sin ^2\theta _{13}$ does not agree with the analytical expectations and instead is always very small, showing that like in Case 1), $\sin ^2\theta _{13}$ is typically adjusted to its experimental best-fit value (since it also drives the fit for the present case).

We confirm numerically that the deviations of $\sin ^2 \theta _{ij}$ do not depend on the choice of the parameter v and we have thus fixed $v=3$. As regards the dependence of $\Delta \sin ^2\theta _{ij}$ on the parameter u, we have also checked that the aforementioned different choices of u all lead to the same result.

For the relative deviations of the CP phases $\delta$ and $\beta$, $\Delta \sin \delta$ and $\Delta \sin \beta$, we present our findings in Fig. 5. Since these deviations are also independent of the choice of v, we choose $v=3$ for concreteness. The plot for $\Delta \sin \alpha$ looks very similar to the corresponding one of Case 1), see left plot in Fig. 3. The sign and size of the deviations are in accordance with the analytical expectations, see Eqs. (82, 83). We note that both Majorana phases $\alpha$ and $\beta$ experience slightly larger effects from the non-unitarity of the lepton mixing matrix (i.e., the presence of the heavy sterile states) than the Dirac phase $\delta$. The effects of the non-unitarity of ${\widetilde{U}}_\nu$ on the behaviour of $\sin \alpha$ with respect to v/n, shown in the left plot of Fig. 6, are very similar to those encountered when studying $\sin \alpha$ with respect to s/n for Case 1), see the right plot in Fig. 3. Again, we emphasise that the periodicity of $\sin \alpha$ in v/n is not altered by the effects of the non-unitarity of the PMNS mixing matrix.

Fig. 6

Case 2). Left plot: $\sin \alpha$ with respect to v/n (fixing $n=14$ and continuously varying $0 \le v < 3n = 42$). The different (coloured) curves refer to three different masses $M_0$ like in Fig. 3, also setting $y_0=1$. The black (dashed) curve displays the result for $\sin \alpha$, obtained in the model-independent scenario, see Eq. (59). Right plot: Validity check of the exact sum rule in Eq. (57) for option 1 of the (3, 3) ISS framework with respect to the mass $M_0$ (in GeV). Line and colour code as in Fig. 5. We have chosen $n=14$ and $u=1$ so that $\tan \phi _u \approx 0.23$

Next, we detail our numerical results for the relative deviations of the two (approximate) sum rules found for Case 2), see Sect. 3.2. We have checked that for the sum rule which is common for Case 1) and Case 2) (see first approximate equality in Eq. (50)), the results do coincide with those shown in the left plot in Fig. 4. Concerning the exact sum rule, shown in Eq. (57), the numerical results are given in the right plot in Fig. 6. We see that the size and sign of the relative deviation agree with the analytical estimate shown in Eq. (90). We have also checked numerically that the results do not depend on the choice of u and v; while the plot presented relies on $u=1$ and $v=3$, similar results have been found for the other mentioned choices of u and the admitted values of v.

We comment on the choice $u=0$ that predicts maximal atmospheric mixing and maximal Dirac phase $\delta$, $\sin \beta =0$ and the exact equality in Eq. (59): the relative deviations $\Delta \sin ^2\theta _{23}$ and $\Delta \sin \delta$ are of the same sign and size, and exhibit the same dependence on the Yukawa coupling $y_0$ and on the mass scale $M_0$ as occurs for the choice $u=1$. Furthermore, the fact that the Majorana phase $\beta$ is trivial is not altered by the effects of non-unitarity of ${\widetilde{U}}_\nu$, as expected from the analytical estimates, see Sect. 4. The results for the Majorana phase $\alpha$ look very similar to those displayed in Fig. 3 (left plot) and Fig. 6 (left plot). Moreover, we confirm that whenever the choice $v=0$ is permitted, the Majorana phase $\alpha$ vanishes independently of the deviations of ${\widetilde{U}}_\nu$ from unitarity.

Finally, we notice that we have performed a numerical check to confirm that the symmetry transformations in the parameters u and $\theta$, see Eq. (60) under point (i) in Sect. 3.2, are still valid.

5.4 Case 3 a)

As representative values for n and m, we take

$$\begin{aligned} n=17 \;\; \text{ and } \;\; m=1, \end{aligned}$$

(101)

since these can satisfactorily accommodate the experimental data on the reactor and the atmospheric mixing angles, $\sin ^2 \theta _{13} = 0.02221^{+0.00068} _{-0.00062}$ and $\sin ^2 \theta _{23} = 0.570^{+0.018} _{-0.024}$ for light neutrinos with NO and $\sin ^2 \theta _{13} = 0.02240^{+0.00062} _{-0.00062}$ and $\sin ^2 \theta _{23} = 0.575^{+0.017} _{-0.021}$ for light neutrinos with IO [76], according to the expectations from the model-independent scenario, see Sect. 3.3.1 and, especially, Eq. (64). We consider all possible values of the parameter s. In addition to $m=1$, we also study the results on lepton mixing for the choice $m=16$. The rather large value of the index n of the flavour symmetry is needed in order to achieve a sufficiently small value of m/n (or $1-m/n$).

Fig. 7

Case 3 a). Contour plots for $\sin ^2\theta _{12}$ in the $(s/n-\theta _S)$ plane, obtained for option 1 of the (3, 3) ISS framework. The left plot is for $M_0=1000$ GeV and the right one for $M_0=5000$ GeV. We fix $y_0=0.5$ in order to amplify the differences between the two plots. Here the grey-shaded areas denote values of $\sin ^2\theta _{12}$ which are experimentally favoured at the $3 \, \sigma$ level [76]

Since fixing n and m determines completely the value of the reactor and the atmospheric mixing angles, we only consider, like for Case 1) and Case 2), the relative deviations $\Delta \sin ^2\theta _{13}$ and $\Delta \sin ^2\theta _{23}$. We note that their size and sign do agree with the analytical expectations, see Eq. (83). Furthermore, we confirm numerically that there is no dependence of these results on the parameter s and the free angle $\theta _S$. Since in Case 3 a) $\theta _{12}$ is the only lepton mixing angle that depends on the free angle $\theta _S$, $\sin ^2\theta _{12}$ naturally drives the fit, and thus the relative deviation $\Delta \sin ^2\theta _{12}$ is always very small. Given that $\sin ^2\theta _{12}$ further depends on the parameter s, we present in Fig. 7 plots for $\sin ^2\theta _{12}$ in the $(s/n-\theta _S)$ plane for two different values of the mass scale $M_0=1000$ GeV (left plot) and $M_0=5000$ GeV (right plot). We fix the Yukawa coupling to $y_0=0.5$ in order to better perceive the differences in the plots for the two different values of $M_0$, although such a large value of $y_0$ does require $M_0 \gtrsim 2400$ GeV in order to comply with the experimental constraints on $\eta _{\alpha \beta }$, see Sect. 5.1. As one observes in Fig. 7, the visible differences are still very small. We stress that here the grey-shaded areas indicate the values of $\sin ^2\theta _{12}$ that are experimentally favoured at the $3 \, \sigma$ level [76]. As can be clearly seen from Fig. 7, for most values of s a successful accommodation of the experimental data can be obtained for two different values of the free angle $\theta _S$. One of these values is close to $\theta _S \approx 0$ or $\theta _S \approx \pi$. These plots can be compared with a very similar one shown in the original analysis of the different mixing patterns, see [33].

Fig. 8

Case 3 a). Contour plots for the sines of the CP phases in the $(s/n-\theta _S)$ plane, obtained for option 1 of the (3, 3) ISS framework. From top to bottom (first to third row), $\sin \delta$, $\sin \alpha$ and $\sin \beta$. On the left column plots, $M_0=1000$ GeV while on the right $M_0=5000$ GeV. We again fix $y_0=0.5$ (see Fig. 7). The colour scheme denotes the values of the sines, from $-1$ (dark blue) to $+1$ (light yellow), as indicated by the colour bar on the right of each plot. The white/grey-shaded areas correspond here to those of Fig. 7, and indicate the values of the solar mixing angle that are experimentally preferred at the $3 \, \sigma$ level

The results for the relative deviations of the CP phases, $\Delta \sin \delta$, $\Delta \sin \alpha$ and $\Delta \sin \beta$, look similar to those obtained for the already presented cases, Case 1) and Case 2). For this reason, we prefer to show contour plots for the sines of all three CP phases in the $(s/n-\theta _S)$ plane. These can be found in Fig. 8, where we display $\sin \delta$, $\sin \alpha$ and $\sin \beta$, for two different values of $M_0$, $M_0=1000$ GeV (left plots) and $M_0=5000$ GeV (right plots). The colour scheme denotes the values of the sines (indicated by the colour bar on the right of each plot). We again take $y_0=0.5$ in order to enhance the visibility of differences in the plots. The white/grey-shaded areas indicate the values of the solar mixing angle that are experimentally preferred at the $3 \, \sigma$ level. It turns out that visible differences between the plots for $M_0=1000$ GeV and $M_0=5000$ GeV are (mainly) found in regions of the $(s/n-\theta _S)$ plane that are not compatible with the experimental value of $\sin ^2\theta _{12}$ at the $3 \, \sigma$ level. Nevertheless, the results presented in these plots are interesting, since the validity of the approximate formulae for the sines of the CP phases (found in Eqs. (68, 69) under point e) for the model-independent scenario) as well as the fact that the absolute value of $\sin \beta$ is bounded to be smaller than $\sim 0.9$, can be checked. Furthermore, they can be directly compared with the results for the model-independent scenario presented in [33]. Again, we confirm numerically that the effects of non-unitarity of the PMNS mixing matrix do not affect the vanishing of $\sin \delta$, $\sin \alpha$ and/or $\sin \beta$ (occurring for certain choices of group theory parameters). The approximate sum rule, quoted in Eq. (65), is valid with a plus sign for the choice $n=17$ and $m=1$. Studying its behaviour depending on the Yukawa coupling $y_0$ and on the mass scale $M_0$ thus leads to results very similar to those obtained for the second approximate sum rule (see second approximate equality in Eq. (50)), for values of the free angle $\theta _S <\pi /2$, as shown in the right plot of Fig. 4.

In the end, we note that we have numerically confirmed that the symmetry transformations, given under point (i) in Sect. 3.3.1, are valid.

5.5 Case 3 b.1)

For the last case, we focus on

$$\begin{aligned} n=20 \;\; \text{ and } \;\; m=11. \end{aligned}$$

(102)

All viable values of the parameter s are studied. We choose the index n of the flavour symmetry to be rather large¹⁶ in order to allow studying different values of m, while achieving good agreement with experimental data on the solar mixing angle. In addition to the value $m=11$ we also perform a numerical analysis for $m=9$ and $m=10$.

For Case 3 b.1) all mixing angles turn out to depend on the parameter s and the free angle $\theta _S$, in addition to the two parameters n and m which we have fixed, see Eq. (72). In what follows we identify the areas in the $(s/n-\theta _S)$ plane in which the three mixing angles (individually and simultaneously) are in agreement with the experimental data at the $3 \, \sigma$ level [76]. This is shown in the contour plots in Fig. 9, for $\sin ^2\theta _{12}$ (blue), $\sin ^2\theta _{23}$ (green) and $\sin ^2\theta _{13}$ (orange) and their combination (black), for two different values of the mass scale $M_0$, $M_0=1000$ GeV (left plot) and $M_0=5000$ GeV (right plot). We note that we have again chosen $y_0=0.5$ for better visibility of the differences in the plots, although in this case $M_0=1000$ GeV leads to conflict with the experimental constraints on the quantities $\eta _{\alpha \beta }$, see Sect. 5.1. We see that the areas of agreement with experimental data at the $3 \,\sigma$ level slightly differ between $M_0=1000$ GeV and $M_0=5000$ GeV. However, their overlap (shown in black in the two plots) is not visibly affected, and thus the parameter space in the $(s/n-\theta _S)$ plane compatible with the experimental data on lepton mixing angles hardly depends on the mass scale $M_0$. Indeed, comparing these two plots to a similar one, presented in the original analysis of the mixing pattern Case 3 b.1) for the model-independent scenario [33], we confirm that all agree very well. We note that the by far strongest constraint on the allowed parameter space in the $(s/n-\theta _S)$ plane is imposed by the reactor mixing angle $\sin ^2\theta _{13}$. The results shown in the plots in Fig. 9 also confirm that all values of the parameter s lead to a successful accommodation of the experimental data of the mixing angles for $n=20$ and $m=11$. The values of the free angle $\theta _S$ are then close to $\pi /2$. Regarding the size and sign of the relative deviations $\Delta \sin ^2\theta _{12}$ and $\Delta \sin ^2\theta _{23}$, we note that these are consistent with the analytical estimates, see Eq. (83), whereas for $\Delta \sin ^2\theta _{13}$ we always find it to be very small due to the pull in the fit that drives the adjustment of the free angle $\theta _S$ to match the best-fit value of the reactor mixing angle. This is analogous to what has been observed for Case 1) and Case 2).

Fig. 9

Case 3 b.1). Contour plots for $\sin ^2\theta _{ij}$, obtained for option 1 of the (3, 3) ISS framework, in the $(s/n-\theta _S)$ plane. Blue, green and orange respectively correspond to $\sin ^2\theta _{12}$, $\sin ^2\theta _{23}$ and $\sin ^2\theta _{13}$. Dotted lines indicate the experimental best-fit (BF) value for each $\sin ^2\theta _{ij}$, while the coloured surfaces correspond to a $3 \sigma$ interval: dashed (solid) lines respectively define the BF${\mp } 3 \sigma$ boundaries. Their overlap is highlighted in black. On the left, $M_0=1000$ GeV, while on the right $M_0=5000$ GeV. We again fix $y_0=0.5$ in order to amplify the differences between the two plots

Fig. 10

Case 3 b.1). Contour plots for the sines of the CP phases, obtained for option 1 of the (3, 3) ISS framework, in the $(s/n-\theta _S)$ plane. From top to bottom (first to third row), $\sin \delta$, $\sin \alpha$ and $\sin \beta$. The white/grey-shaded areas correspond to the black regions in the plots in Fig. 9, and indicate the regions in which all three mixing angles agree with experimental data at the $3 \, \sigma$ level. Input parameters ($M_0$ and $y_0$) and colour coding as in Fig. 8

In what concerns the CP phases, we proceed in the same way as for the three mixing angles, and show in Fig. 10 several contour plots in the $(s/n-\theta _S)$ plane. We choose the same values of $M_0$ and $y_0$ as for the analogous study done for Case 3 a); conventions and colour-coding are identical to Fig. 8. Like in Case 3 a), the visible differences for the different values of $M_0$ are mostly found in regions of the $(s/n-\theta _S)$ plane that disagree with experimental data on the three mixing angles by more than $3\, \sigma$. We can observe that the absolute value of $\sin \delta$ has an upper bound $\sim 0.8$ for the choice $n=20$ and $m=11$, whereas the sines of both Majorana phases are a priori not constrained. Comparing the relative deviations of the sines of the CP phases, $\Delta \sin \alpha$, $\Delta \sin \beta$ and $\Delta \sin \delta$, with the analytical estimates, see Eq. (83), we find agreement in the size; notice however that the sign of the relative deviations $\Delta \sin \beta$ and $\Delta \sin \delta$ is positive.

Fig. 11

Case 3 b.1). Validity check of approximate sum rules in Eq. (74) for option 1 of the (3, 3) ISS framework with respect to the mass $M_0$ (in GeV). In addition to $n=20$ and $m=10$, we fix $s=4$ ($\cos 3 \, \phi _s \approx -0.31$) and $\theta _S \approx 1.83$ for the evaluation of both approximate sum rules. Otherwise, same conventions and colour-coding as in Fig. 2

As shown in the model-independent scenario, several simplifications of the formulae in Eqs. (72, 73) can be made for $m=\frac{n}{2}$ (corresponding to $m=10$ for the present case). In particular, two approximate sum rules are found, see Eq. (74). In the following, we investigate how these are affected by the presence of the ISS heavy sterile states. We proceed in an analogous way as done for the (approximate) sum rules found for the other cases. Our results are displayed in Fig. 11 for two different values of the Yukawa coupling, $y_0=0.1$ and $y_0=0.5$, and can be compared to the analytical estimates for the relative deviations $\Delta \Sigma _4$ and $\Delta \Sigma _5$, see Eqs. (91, 92, 93) in Sect. 4. We note that the results have been obtained for the choice $s=4$ ($\cos 3 \, \phi _s \approx -0.31$). This choice has been made since it leads to a value of the atmospheric mixing angle which agrees best with current experimental data [76]. Furthermore, we remark that we have replaced $\theta _0$ by $\theta _S$ in the second approximate sum rule in Eq. (74) which, however, turns out to be very close to $\theta _0\approx 1.83$. As can be seen in Fig. 11, for $y_0 \sim 0.5$ and $M_0 \sim 500$ GeV we find a deviation of about $-3 \%$ with respect to the results obtained in the model-independent approach. We thus confirm the analytical expectation (see Eq. (93)), which was obtained for $y_0 = 1$ and $M_0 = 1000$ GeV (leading to the same value of $\eta _0$, cf. Eq. (78)). For large $M_0$ the displayed ratios may not lead to exactly one, since the two sum rules only hold approximately.

For the choice of $m = \frac{n}{2} = 10$, we can also check the (approximate) validity of the statements made for the sines of the CP phases and for the lower bound on the absolute value of the sine of the CP phase $\delta$, as observed in the model-independent scenario (compare to point (e) in Sect. 3.3.2). Indeed, these hold, up to the expected deviations due to the effects of non-unitarity of ${\widetilde{U}}_\nu$; moreover, the equality of the sines of the two Majorana phases $\alpha$ and $\beta$ still holds exactly (see first equality in Eq. (75)).

For the choice $m=\frac{n}{2}=10$ and additionally $s=\frac{n}{2}=10$, one expects from the model-independent scenario that the atmospheric mixing angle and the Dirac phase are maximal, while both Majorana phases are trivial. This also holds to a very good degree for option 1 of the (3, 3) ISS framework, for values of $M_0 \gtrsim 500$ GeV and $y_0 \lesssim 1$. In general, in all occasions in which a trivial CP phase is expected in the model-independent scenario, the same is obtained for option 1 of the (3, 3) ISS framework.

For Case 3 b.1) we numerically confirm that the symmetry transformations, given in Eq. (70) under point (i) for Case 3 a), also hold.

In summary, we find that the effects of non-unitarity (of the PMNS mixing matrix, ${\widetilde{U}}_\nu$) on the lepton mixing parameters and on the (approximate) sum rules relating them, turn out to be below the $1\%$ level, once experimental limits on the quantities $\eta _{\alpha \beta }$ are taken into account, see Sect. 5.1. Consequently, the results obtained for option 1 of the (3, 3) ISS framework are very similar to those obtained in the model-independent scenario [33]. In particular, the dependence of the CP phases on the group theory parameters (especially those determining the CP transformation X) and the vanishing of a CP phase for certain choices of group theory parameters, are not affected.

6 Results for neutrinoless double beta decay

In the following, we briefly comment on $0\nu \beta \beta$ decay prospects for option 1 of the (3, 3) ISS framework. First, we recall that in the presence of light neutrinos and of heavy sterile states, the effective mass $m_{ee}$, accessible in $0\nu \beta \beta$ decay experiments, is given by [79]

$$\begin{aligned} m_{ee}\simeq & {} \sum _{i=1}^{3+n_s} \, {\mathcal {U}}_{e i}^2 \, p^2 \, \frac{m_{ i}}{p^2-m_{i}^2} \simeq \sum _{i=1}^3 \, {\mathcal {U}}^2_{e i} \, m_i \nonumber \\&+ \sum _{k=4}^{3+n_s} {\mathcal {U}}_{e k}^2 \, p^2 \, \frac{m_{k}}{p^2-m_{k}^2}, \end{aligned}$$

(103)

where $n_s$ denotes the number of heavy sterile states, in our case $n_s=6$, and the virtual momentum $p^2$ is estimated as $p^2 \simeq -(100 \, \mathrm {MeV})^2$. For $i=1,2,3$, the mixing matrix elements ${\mathcal {U}}_{e i}$ coincide with the elements of the first row of the matrix ${\widetilde{U}}_\nu$ (and hence ${\widetilde{U}}_{\mathrm {PMNS}}$); for $k=4,\ldots , 3+n_s=4, \ldots , 9$, in our case ${\mathcal {U}}_{e k}$ are approximately given by

$$\begin{aligned} {\mathcal {U}}_{e k}\approx & {} -i \, \left( \frac{y_0 \, v}{2 \, M_0} \right) \, (U_S)_{1 \, k-3} \;\; \text{ for } \;\; k=4,\ldots , 6 \;\; \text{ and } \;\; \nonumber \\ {\mathcal {U}}_{e k}\approx & {} \left( \frac{y_0 \, v}{2 \, M_0} \right) \, (U_S)_{1 \, k-6} \;\; \text{ for } \;\; k=7,\ldots , 9 \end{aligned}$$

(104)

according to the expression for S presented in Eq. (41). For $i=1,2,3$ $m_{i}$ correspond to the light neutrino masses; we recall that, according to Eq. (42) for option 1 of the (3, 3) ISS framework, the masses of the heavy sterile states $m_{k}$ (with $k=4,\ldots , 3+n_s=4, \ldots , 9$) are approximately degenerate

$$\begin{aligned} m_{k} \approx M_0. \end{aligned}$$

(105)

Thus, we have

$$\begin{aligned} m_{ee}\simeq & {} \sum _{i=1}^3 \, {\mathcal {U}}^2_{e i} \, m_i + \left( \frac{p^2 \, M_0}{p^2-M_0^2} \right) \, \left( \frac{y_0^2 \, v^2}{4 \, M_0^2} \right) \, \left( - \sum _{k=1}^3 (U_S)_{1 k}^2 \right. \nonumber \\&\left. + \sum _{k=1}^3 (U_S)_{1 k}^2 \right) = \sum _{i=1}^3 \, {\mathcal {U}}^2_{e i} \, m_i, \end{aligned}$$

(106)

implying that the contribution of the heavy sterile states to $m_{ee}$ is very suppressed due to their pseudo-Dirac nature. Consequently, we expect that the results for $m_{ee}$ are very similar to those obtained in the model-independent scenario, as studied for example in [80].

Fig. 12

Results for ${0}{{\nu }}{{\beta }}{{\beta }}$ decay for Case 3 b.1). Effective mass $m_{ee}$ as a function of the lightest neutrino mass $m_0$ (both in eV) obtained for option 1 of the (3, 3) ISS framework. Leading to the results in the left plot, we take $n=20$, $m=10$ and $s=9$ (blue curves) and $s=10$ (orange curves), while the right plot is for $n=20$, $m=11$ and $s=9$ (blue curves) and $s=10$ (orange curves). In both plots we fix $M_0=1000$ GeV and $y_0=0.1$. Solid (dashed) curves correspond to a NO (IO) light neutrino mass spectrum. The purple (green) shaded area arises upon variation of the lepton mixing parameters and mass squared differences within the experimentally preferred $3 \, \sigma$ ranges for NO (IO) [76]. Values of $m_0$ disfavoured by the cosmological bound on the sum of the light neutrino masses [81] are identified by a vertical grey band, while regimes of $m_{ee}$ already disfavoured by searches for $0\nu \beta \beta$ decay [82] are indicated by a horizontal grey band

For completeness, we show two plots for $m_{ee}$ in Fig. 12, where we have set $y_0=0.1$ and $M_0=1000$ GeV. These plots were obtained for Case 3 b.1), and we have chosen $n=20$ like in Sect. 5.5. In the left plot of Fig. 12 we fix $m=10$, while in the right one $m=11$. In both plots we display results for two values of s, $s=9$ (blue) and $s=10$ (orange). Solid (dashed) curves correspond to a NO (IO) light neutrino mass spectrum. We remind that for $s=10$ both Majorana phases turn out to be trivial, thus allowing for the strong cancellation observed in association with the orange solid curve in the left plot. The thickness of the curves is determined by the variation of the mass squared differences in their experimentally preferred $3 \, \sigma$ ranges [76], see Eqs. (156, 157) in Appendix C. The purple (green) shaded area arises upon variation of the lepton mixing parameters and of the mass squared differences within the experimentally preferred $3 \, \sigma$ ranges for NO (IO). The upper bound on the lightest neutrino mass $m_0$ arises from the cosmological bound on the sum of the light neutrino masses [81], see Eq. (158) in Appendix C. In Fig. 12 we have depicted the experimental limit on $m_{ee}$ obtained by the KamLAND-Zen Collaboration (using the isotope $^{136} \mathrm {Xe}$) [82],

$$\begin{aligned} m_{ee} < (61 \div 165) \, \mathrm {meV}, \end{aligned}$$

(107)

with the above range resulting from different theoretical estimates of the nuclear matrix elements. Similar limits have been obtained by other collaborations, for distinct choices of isotopes: $m_{ee} < (78 \div 239) \, \mathrm {meV}$ also for $^{136} \mathrm {Xe}$, by EXO-200 [83]; $m_{ee} < (79 \div 180) \, \mathrm {meV}$ for $^{76}\mathrm {Ge}$, as derived by GERDA [84]; $m_{ee} < (200 \div 433) \, \mathrm {meV}$ also for $^{76} \mathrm {Ge}$ by the Majorana Demonstrator [85]; $m_{ee} < (75 \div 350) \, \mathrm {meV}$ for $^{130} \mathrm {Te}$, obtained by CUORE [86]. With further improvement of the experimental limits, certain combinations of group theory parameters in the different cases could be disfavoured, at least, if the light neutrino mass spectrum is assumed to follow IO.

7 Impact for charged lepton flavour violation

We now proceed to discuss the impact of endowing the (3,3) ISS realisation with flavour and CP symmetries concerning cLFV observables, such as radiative and three-body lepton decays, and neutrinoless $\mu -e$ conversion in matter.

Before addressing the cLFV rates, it is important to recall that in a regime of sufficiently small $\mu _i$, the heavy Majorana states are approximately mass-degenerate in pairs, and have opposite CP-parity, thus effectively leading to the formation of pseudo-Dirac pairs, whose phases are closely related by (see Eq. (41))

$$\begin{aligned} {\mathcal {U}}_{\alpha j }= & {} {\mathcal {U}}_{\alpha j+3}\, e^{i (\varphi _{\alpha j} - \varphi _{\alpha j+3})}\,, \quad \text {with} \quad \varphi _{\alpha j} - \varphi _{\alpha j+3} = -\pi /2,\nonumber \\ \end{aligned}$$

(108)

in which ${\mathcal {U}}_{\alpha j }$ are elements of the unitary nine-by-nine matrix (cf. Eq. (20)), with $j = 4, 5, 6$ and $\alpha = e, \mu , \tau$. For option 1, not only is the mass splitting extremely small, typically ${\mathcal {O}}(1-100)$ eV but, as can be seen from Eq. (42), the pseudo-Dirac pairs are themselves degenerate in mass up to a very good approximation. In view of the above, the loop functions entering the distinct observables (see Appendix E) can be taken universal for the heavy states, $f(x_i) = f(x_0), \, \forall i = 4,5, \ldots , 9$ with $x_0 \simeq M_0^2/M_W^2$, where $M_W$ is the mass of the W-boson.

The full expressions for the cLFV rates arising in SM extensions via $n_s$ heavy sterile states for the radiative and three-body decays are given by [87]

$$\begin{aligned}&\mathrm {BR}(\ell _\beta \rightarrow \ell _\alpha \gamma ) = \frac{\alpha _w^3\, s_w^2}{256\,\pi ^2}\,\frac{m_{\beta }^4}{M_W^4}\, \frac{m_{\beta }}{\Gamma _{\beta }}\, \left| G_\gamma ^{\beta \alpha } \right| ^2\,, \end{aligned}$$

(109)

$$\begin{aligned}&\mathrm {BR}(\ell _\beta \rightarrow 3\ell _\alpha ) \nonumber \\&\quad = \frac{\alpha _w^4}{24576\,\pi ^3}\,\frac{m_{\beta }^4}{M_W^4}\, \frac{m_{\beta }}{\Gamma _{\beta }}\times \left\{ 2\left| \frac{1}{2}F_\text {box}^{\beta 3\alpha } +F_Z^{\beta \alpha } \right. \right. \nonumber \\&\qquad \left. \left. - 2 s_w^2\,(F_Z^{\beta \alpha } - F_\gamma ^{\beta \alpha })\right| ^2 \right. \nonumber \\&\qquad + 4 s_w^4\, |F_Z^{\beta \alpha } - F_\gamma ^{\beta \alpha }|^2\nonumber \\&\qquad \left. + 16 s_w^2\,\mathrm {Re}\left[ (F_Z^{\beta \alpha } - \frac{1}{2}F_\text {box}^{\beta 3\alpha })\,G_\gamma ^{\beta \alpha \star }\right] \right. \nonumber \\&\qquad - 48 s_w^4\,\mathrm {Re}\left[ (F_Z^{\beta \alpha } - F_\gamma ^{\beta \alpha })\,G_\gamma ^{\beta \alpha \star }\right] \nonumber \\&\qquad \left. + 32 s_w^4\,|G_\gamma ^{\beta \alpha }|^2\left[ \log \frac{m_{\beta }^2}{m_{\alpha }^2} - \frac{11}{4}\right] \right\} , \end{aligned}$$

(110)

in which $m_{\alpha }$ ($\Gamma _\alpha$) denotes the mass (total width) of a charged lepton of flavour $\alpha$, $\alpha _w = g_w^2/4\pi$ the weak coupling, and $s_w$ the sine of the weak mixing angle. Concerning the conversion rate in nuclei, one has [87]

$$\begin{aligned} \mathrm {CR}(\mu - e,\,\mathrm {N})= & {} \frac{2 G_F^2\,\alpha _w^2\, m_\mu ^5}{(4\pi )^2\,\Gamma _\text {capt.}}\left| 4 V^{(p)}\left( 2 {{\widetilde{F}}}_u^{\mu e} + {{\widetilde{F}}}_d^{\mu e}\right) \right. \nonumber \\&\left. + 4 V^{(n)}\left( {{\widetilde{F}}}_u^{\mu e} + 2{{\widetilde{F}}}_d^{\mu e}\right) + s_w^2 \frac{G_\gamma ^{\mu e}D}{2 e}\right| ^2,\nonumber \\ \end{aligned}$$

(111)

in which D, $V^{(p)}$ and $V^{(n)}$ are nuclear form factors whose values can be found in Ref. [88] and e is the unit electric charge. For a given nucleus N, $\Gamma _\text {capt.}$ denotes the capture rate. The form factors present in the above equations are given by [87, 89]

$$\begin{aligned} G_\gamma ^{\beta \alpha }= & {} \sum _{i =1}^{3 + n_s} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta i}^\star \, G_\gamma (x_i)\,, \end{aligned}$$

(112)

$$\begin{aligned} F_\gamma ^{\beta \alpha }= & {} \sum _{i =1}^{3 + n_s} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta i}^\star \,F_\gamma (x_i)\,,\end{aligned}$$

(113)

$$\begin{aligned} F_Z^{\beta \alpha }= & {} \sum _{i,j =1}^{3 + n_s} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta j}^\star \left[ \delta _{ij} \,F_Z(x_j) + C_{ij}\, G_Z(x_i, x_j) \right. \nonumber \\&\left. + C_{ij}^\star \,H_Z(x_i, x_j)\right] \,, \end{aligned}$$

(114)

$$\begin{aligned} F_\text {box}^{\beta 3 \alpha }= & {} \sum _{i,j = 1}^{3+n_s} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta j}^\star \left[ {\mathcal {U}}_{\alpha i} \,{\mathcal {U}}_{\alpha j}^\star \, G_\text {box}(x_i, x_j) \right. \nonumber \\&\left. - 2 \,{\mathcal {U}}_{\alpha i}^\star \,{\mathcal {U}}_{\alpha j}\, F_\text {Xbox}(x_i, x_j) \right] \,,\end{aligned}$$

(115)

$$\begin{aligned} F_\text {box}^{\mu e uu}= & {} \sum _{i = 1}^{3 + n_s}\sum _{q_d = d, s, b} {\mathcal {U}}_{e i}\,{\mathcal {U}}_{\mu i}^\star \, V_{u q_d}\,V_{u q_d}^\star \,F_\text {box}(x_i, x_{q_d}), \nonumber \\\end{aligned}$$

(116)

$$\begin{aligned} F_\text {box}^{\mu e dd}= & {} \sum _{i = 1}^{3 + n_s}\sum _{q_u = u, c, t} {\mathcal {U}}_{e i}\,{\mathcal {U}}_{\mu i}^\star \, V_{q_u d}\,V_{q_u d}^\star \,F_\text {Xbox}(x_i, x_{q_u}), \nonumber \\\end{aligned}$$

(117)

$$\begin{aligned} {{\widetilde{F}}}^{\mu e}_d= & {} -\frac{1}{3}s_w^2 F_\gamma ^{\mu e} - F_Z^{\mu e}\left( \frac{1}{4} - \frac{1}{3}s_w^2 \right) + \frac{1}{4}F^{\mu e dd}_\text {box} \nonumber \\\end{aligned}$$

(118)

$$\begin{aligned} {{\widetilde{F}}}^{\mu e}_u= & {} \frac{2}{3}s_w^2 F_\gamma ^{\mu e} + F_Z^{\mu e}\left( \frac{1}{4} - \frac{2}{3}s_w^2 \right) + \frac{1}{4}F^{\mu e uu}_\text {box}\nonumber \\\end{aligned}$$

(119)

$$\begin{aligned} \text {with } \quad x_i= & {} \frac{m_{i}^2}{M_W^2}\,,\quad x_{q} = \frac{m_{q}^2}{M_W^2}\,,\quad C_{ij} = \sum _{\rho = 1}^3 {\mathcal {U}}_{i\rho }^\dagger \,{\mathcal {U}}_{\rho j}\,.\nonumber \\ \end{aligned}$$

(120)

In the above, $i,j=1,\ldots ,9$ denote the neutral lepton mass eigenstates, $\alpha ,\beta$ the leptonic flavours, and V the Cabibbo–Kobayashi–Maskawa quark mixing matrix.

While the radiative decays ($\ell _\beta \rightarrow \ell _\alpha \gamma$) only call upon $G_\gamma ^{\beta \alpha }$, three-body decays ($\ell _\beta \rightarrow 3\ell _\alpha$) depend¹⁷ on $F_\text {box}^{\beta 3 \alpha }$, $G_\gamma ^{\beta \alpha }$, $F_\gamma ^{\beta \alpha }$ and $F_Z^{\beta \alpha }$. Finally, $\mu -e$ conversion in nuclei involves the latter three form factors (for $\alpha , \beta = e, \mu$), as well as additional ones corresponding to box diagrams with an internal quark line, $F_\text {box}^{\mu e qq}$.

It is worth noticing that the combination $\sum _{i=4}^9 {\mathcal {U}}_{\alpha i} \,{\mathcal {U}}_{\beta i}^\star$ can be recast in terms of the unitarity violation of the PMNS mixing matrix, $\widetilde{U}_\nu$. As usually done [90], one can write

$$\begin{aligned} {\widetilde{U}}_\nu \, = \, A\,U_0, \end{aligned}$$

(121)

in which $U_0$ is a unitary three-by-three matrix (see Eq. (23)) and A is a triangular matrix,

$$\begin{aligned} A\, =\, \begin{pmatrix} \alpha _{11} &{} 0&{} 0\\ \alpha _{21} &{} \alpha _{22} &{} 0\\ \alpha _{31} &{} \alpha _{32} &{} \alpha _{33} \end{pmatrix}, \end{aligned}$$

(122)

and we define

$$\begin{aligned} {\mathcal {A}} \equiv \, A \,A^\dagger = {\widetilde{U}}_\nu \, {\widetilde{U}}_\nu ^\dagger . \end{aligned}$$

(123)

Recalling the definition of the quantity $\eta$ (see Eqs. (23, 24)), it is manifest that one has

$$\begin{aligned} {\mathcal {A}}_{\alpha \beta } \, =\, \delta _{\alpha \beta } - 2 \eta _{\alpha \beta }. \end{aligned}$$

(124)

Unitarity of the full nine-by-nine matrix ${\mathcal {U}}$ implies that

$$\begin{aligned} \sum _{i=4}^9 {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta i}^\star= & {} \delta _{\alpha \beta } - \sum _{i=1}^3 {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta i}^\star \nonumber \\= & {} \delta _{\alpha \beta } - ({\widetilde{U}}_\nu \, {\widetilde{U}}_\nu ^\dagger )_{\alpha \beta } =\delta _{\alpha \beta } - {\mathcal {A}}_{\alpha \beta } = 2\, \eta _{\alpha \beta }.\nonumber \\ \end{aligned}$$

(125)

For option 1 one has $\eta _{\alpha \beta } =0$, $\forall \alpha \ne \beta$, so that $\eta$ and thus ${\mathcal {A}}$ (and also A) are diagonal. This is of paramount importance for the cLFV observables, since – and as discussed below – any contribution proportional to $\eta _{\alpha \beta }$ will vanish (for $\alpha \ne \beta$).

7.1 Dipole terms – radiative decays ${\ell _\beta \rightarrow \ell _\alpha \gamma }$

Since the contribution of the light (mostly active) neutrinos to the dipole form factor can be neglected (the relevant limits of the loop functions can be found in Appendix E) one has

$$\begin{aligned} G_\gamma ^{\beta \alpha } \, \simeq \, G_\gamma (x_0) \sum _{i=4}^9 {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta i}^\star , \end{aligned}$$

(126)

or, and in view of the above discussion,

$$\begin{aligned} G_\gamma ^{\beta \alpha } \, \simeq \, G_\gamma (x_0)\, \left( \delta _{\alpha \beta } - {\mathcal {A}}_{\alpha \beta } \right) \,=\, 2\, G_\gamma (x_0)\, \eta _{\alpha \beta }. \end{aligned}$$

(127)

As an illustrative example, for radiative cLFV muon decays one has $G_\gamma ^{\mu e} = -G_\gamma (x_0)\, \alpha _{11}\,\alpha _{21}^\star$. For the present scenario, in which ${\mathcal {A}}$ and $\eta$ are diagonal, one thus finds $G_\gamma ^{\beta \alpha } \, \simeq \,0\,$.

In line with the analytical discussion on lepton mixing carried in Sect. 2.2, let us also emphasise that similar results can be obtained relying on the approximate analytical expression for S, which for option 1 is given at leading order in $\mu _S/M_{NS}$ by $S \,\simeq \,\frac{y_0\, v}{2\,M_0}\,(- i U_S,\, U_S)$ (see Eq. (41)). The form factor can be recast as

$$\begin{aligned} G_\gamma ^{\mu e}&\simeq G_\gamma (x_0)\left\{ \sum _{i = 4}^6 {\mathcal {U}}_{ei} \,{\mathcal {U}}_{\mu i}^\star + \sum _{i = 7}^9 {\mathcal {U}}_{ei}\, {\mathcal {U}}_{\mu i}^\star \right\} \nonumber \\&\propto G_\gamma (x_0)\left\{ \sum _{i = 1}^3 (i U_S)_{ei} \,(iU_S)_{\mu i}^\star + \sum _{i = 1}^3 (U_S)_{ei}\, (U_S)_{\mu i}^\star \right\} \,\nonumber \\&= \,2 G_\gamma (x_0) \left\{ \sum _{i = 1}^3 (U_S)_{ei} \,(U_S)_{\mu i}^\star \right\} = 0\,, \end{aligned}$$

(128)

due to the orthogonality of the ($\mu - e$) rows of $U_S$ (which we recall to be a unitary matrix, determined by the group theory parameters and the free angle $\theta _S$).

7.2 Photon and Z penguin form factors

Relevant for both ${\ell _\beta \rightarrow 3 \ell _\alpha }$ and $\mu -e$ conversion, these include several contributions (reflecting the fact that two neutral fermions can propagate in the loop).

A reasoning analogous to the one conducted for $G_\gamma ^{\beta \alpha }$ leads to $F_\gamma ^{\beta \alpha } \, \simeq \, F_\gamma (x_0)\, \left( \delta _{\alpha \beta } -{\mathcal {A}}_{\alpha \beta } \right) \,=\, 2\,F_\gamma (x_0)\, \eta _{\alpha \beta }$, which thus vanishes for the flavour violating decays. Likewise, the first term on the right-hand side of Eq. (114) leads to the same result,

$$\begin{aligned} \sum _{i,j =1}^{9} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta j}^\star \left[ \delta _{ij} F_Z(x_i) \right]\simeq & {} F_Z(x_0)\left( \delta _{\alpha \beta } -{\mathcal {A}}_{\alpha \beta } \right) \,\nonumber \\= & {} 2F_Z(x_0)\, \eta _{\alpha \beta }\,=\, 0. \end{aligned}$$

(129)

Both terms associated with $G_Z(x,y)$ and $H_Z(x,y)$ correspond to two neutral leptons propagating in the loop. Although the loop functions do tend to zero for the case of very light internal fermions, the same does not occur for $G_Z$ if at least one of the states is heavy, i.e. $G_Z (0,x_i)$ (see Appendix E). Introducing the following limits for the loop functions,

$$\begin{aligned} {{\bar{G}}}_Z(x) \,= & {} \, \lim _{x_i \gg 1} G_Z (0,x_i), \quad \quad \nonumber \\ {\bar{{\bar{G}}}}_Z(x) \,= & {} \, \lim _{x_i \approx x_j \gg 1} G_Z (x_i,x_j), \end{aligned}$$

(130)

respectively corresponding to “heavy-light” and “heavy-heavy” (combinations of) fermion propagators, one thus has

$$\begin{aligned}&\sum _{i,j = 1}^{9} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta j}^\star \,C_{ij}\, G_Z(x_i, x_j)\simeq {{\bar{G}}_Z(x_0)} \left[ 2 {\mathcal {A}} \,(\mathbb {1}-{\mathcal {A}})\right] _{\alpha \beta }\nonumber \\&\quad + {\bar{{\bar{G}}}}_Z(x_0) \left[ (\mathbb {1}-{\mathcal {A}})^2\right] _{\alpha \beta }, \end{aligned}$$

(131)

or in terms of $\eta$, ${{\bar{G}}_Z(x_0)} \left[ 4(\mathbb {1}-2\eta )\,\eta \right] _{\alpha \beta } + 4 \,{\bar{{\bar{G}}}}_Z(x_0) \, [\eta ^2]_{\alpha \beta }\,=\,0$, as previously argued.

For the $H_Z(x,y)$-associated terms, only the “heavy-heavy” case (two heavy sterile states in the loop) can potentially contribute in a non-negligible way. However, the corresponding contribution also vanishes, as a consequence of the nature of the (degenerate) heavy states, which as mentioned form pseudo-Dirac pairs. Defining (see Appendix E)

$$\begin{aligned} {\bar{{\bar{H}}}}_Z(x) \, =\, \lim _{x_i \approx x_j \gg 1} H_Z (x_i,x_j), \end{aligned}$$

(132)

one then finds (taking into account Eq. (108))

$$\begin{aligned}&\sum _{i,j =1}^{9} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\beta j}^\star \,C_{ij}^\star \, H_Z(x_i, x_j) \nonumber \\&\quad \simeq {\bar{{\bar{H}}}}_Z(x_0) \sum _{i,j =4}^{9} \sum _{\rho =1}^{3} {\mathcal {U}}_{\alpha i}\,{\mathcal {U}}_{\rho i}\, {\mathcal {U}}_{\beta j}^\star \,{\mathcal {U}}_{\rho j}^\star \nonumber \\&\quad \simeq {\bar{{\bar{H}}}}_Z(x_0) \sum _{\rho =1}^{3} \left\{ \left[ \sum _{i =4}^{6} \left( {\mathcal {U}}_{\alpha i}\, {\mathcal {U}}_{\rho i} + e^{i \pi /2} {\mathcal {U}}_{\alpha i}\, e^{i \pi /2} {\mathcal {U}}_{\rho i} \right) \right] \right. \nonumber \\&\qquad \times \left. \left[ \sum _{j =4}^{6} \left( {\mathcal {U}}_{\beta j}^\star \, {\mathcal {U}}_{\rho j}^\star + e^{-i \pi /2}{\mathcal {U}}_{\beta j}^\star \, e^{- i \pi /2}{\mathcal {U}}_{\rho j}^\star \right) \right] \right\} \,=\, 0, \end{aligned}$$

(133)

which is a direct consequence of the pseudo-Dirac nature of the heavy states.

7.3 Box diagrams

Several form factors contribute to both the three-body decays $\ell _\beta \rightarrow 3 \ell _\alpha$, and neutrinoless $\mu -e$ conversion. The first ($F^{\beta 3\alpha }_\text {box}$) can be decomposed in two terms, “box” and cross-box “Xbox”, respectively associated with the loop functions $G_\text {box}$ and $F_\text {Xbox}$. Similar contributions (single internal neutral lepton) are present for the latter ($F^{\mu e qq}_\text {box}$).

Only diagrams with two heavy neutrinos are at the source of non-vanishing contributions to $G_\text {box}$; however, and analogously to what occurred for the previously discussed $H_Z(x,y)$-associated terms, the contributions vanish, due to having the heavy states forming, to an excellent approximation, pseudo-Dirac pairs.

A priori, one can have contributions to the $F_\text {Xbox}$ form factors from “light-light” and “heavy-heavy” fermion propagators in the box. However, both turn out to be proportional to ${\mathcal {A}}_{\alpha \beta }$ and thus to $\eta _{\alpha \beta }$, and are hence vanishing.

The additional form factors relevant for ${\mu -e}$ conversion, $F^{\mu eqq}_\text {box}$ lead to contributions again proportional to $\eta _{e \mu }$, thus also vanishing in the present scenario.

7.4 cLFV for option 1 of the (3, 3) ISS with flavour and CP symmetry

In the present scenario, no new contributions to the different cLFV observables due to the exchange of heavy states are expected.¹⁸ Such a “stealth” realisation of the ISS – which in general can account for significant contributions to the observables, well within experimental sensitivity – is due to two peculiar features of option 1. First and most importantly, recall that here $\mu _S$ is the unique source of flavour violation in the sector of neutral states; this is in contrast with other ISS realisations in which the Dirac neutrino Yukawa couplings (and possibly $M_{NS}$) are non-trivial in flavour space. Moreover, notice that for option 1 of the flavour symmetry-endowed ISS the heavy mass spectrum is composed of three degenerate pseudo-Dirac pairs (to an excellent approximation), which further suppresses any new contribution.

Thus, cLFV processes will not offer any additional source of insight in what concerns the underlying discrete flavour symmetries nor the mass scale of the heavy states; however the observation of at least one cLFV transition would strongly disfavour the flavour symmetry-endowed ISS in its option 1, with strictly diagonal and universal $M_{NS}$ and $m_D$ in flavour space.

8 Summary and outlook

We have considered an inverse seesaw mechanism with $3+3$ heavy sterile states, endowed with a flavour symmetry $G_f=\Delta (3 \, n^2)$ or $G_f=\Delta (6 \, n^2)$ and a CP symmetry. The peculiar breaking of the flavour and CP symmetry to different residual symmetries $G_\ell$ in the charged lepton sector and $G_\nu$ in the sector of the neutral states, is the key to rendering this scenario predictive (and possibly testable). In the inverse seesaw mechanism, several terms in the Lagrangian determine the mass spectrum of the neutral states, in association with three matrices, $m_D$, $M_{NS}$ and $\mu _S$. Several realisations of the residual symmetry $G_\nu$ are possible, and here we have focused on one of the three minimal options, which we have called “option 1”. In this option only the Majorana mass matrix $\mu _S$ breaks $G_f$ and CP to $G_\nu$, while $m_D$ and $M_{NS}$ preserve $G_f$ and CP. In the sector of the neutral states, lepton number and lepton flavour violation are thus both encoded in $\mu _S$. Left-handed lepton doublets and the $3+3$ heavy sterile states are assigned to the same triplet $\mathbf{3}$ of $G_f$, whereas right-handed charged leptons are in singlets.

In [33] mixing patterns arising from the breaking of $G_f$ and CP to $G_\ell$ and $G_\nu$ have been analysed and four of them have been identified as particularly interesting for leptons. We have studied examples of lepton mixing for each of the different mixing patterns, Case 1) through Case 3 b.1), both analytically and numerically. For option 1, a significant consequence of the presence of the heavy sterile states is that for certain regimes there is a sizeable deviation from unitarity of the PMNS mixing matrix, and thus potential conflict with the associated experimental bounds. This leads to stringent constraints on the Yukawa coupling $y_0$ and on the mass scale $M_0$, so that regimes of large $y_0$ and small $M_0$ are disfavoured. In the viable regimes, the impact of the heavy sterile states on lepton mixing turns out to be small: deviations typically below $1\%$ are found upon comparison of the results of the (3, 3) ISS framework to those derived in the model-independent scenario. We have also discussed the potential impact of this ISS framework for several observables. An interesting implication of option 1 here discussed is that the heavy sterile states are degenerate to a very good approximation, and combine to form three pseudo-Dirac pairs. As a consequence, the results for neutrinoless double beta decay are hardly modified, compared to results obtained in the model-independent scenario. We have also addressed in detail charged lepton flavour violating processes: in sharp contrast to what generally occurs for inverse seesaw models (see, e.g. [48, 51]), the cLFV rates are highly suppressed, similar to what occurs in the Standard Model with three light (Dirac) neutrinos. This is a consequence of having strictly flavour-diagonal and flavour-universal deviations from unitarity of the PMNS mixing matrix (and also due to a very high degree of degeneracy in the heavy mass spectrum).

Throughout this work we have assumed that the desired breaking of the flavour and CP symmetries can be realised, and that the appropriate residual symmetries are preserved by the different mass matrices. As has been shown in the literature, it is possible to achieve the breaking of flavour (and CP) in different ways, e.g. spontaneously, if flavour (and CP) symmetry breaking fields acquire non-vanishing vacuum expectation values, in supersymmetric theories (see for instance [91]), or explicitly via boundary conditions in a model with an extra dimension (see e.g. [92, 93]). The predictive power of concrete models is usually higher than the one of the model-independent approach: for example, by choosing a certain set of flavour (and CP) symmetry breaking fields, the ordering of the light neutrino mass spectrum can be predicted, and by extending the flavour (and CP) symmetry to the flavour sector of the new particles, as for instance supersymmetric particles or Kaluza–Klein states, many flavour observables can be constrained and correlated. It is thus interesting to consider the construction of such models.

It is well-known that in concrete models corrections to the desired breaking of flavour (and CP) can arise. This can for instance be the case if flavour (and CP) symmetry breaking fields, whose vacuum expectation values preserve the residual symmetry $G_\ell$, couple at a higher order to the neutral states as well. We have not discussed such corrections in our analysis, but we can briefly comment on their expected impact on lepton mixing as well as predictions for branching ratios of different charged lepton flavour violating processes. Considering, for example, that corrections invariant under $G_\ell$ contribute to the mass matrices $m_D$ and $M_{NS}$, we expect that lepton mixing can still be correctly explained for corrections not larger than a few percent¹⁹ and possibly by re-fitting the value of the free angle $\theta _S$. At the same time, the branching ratios of charged lepton flavour violating processes would still remain strongly suppressed, beyond the reach of current and future experiments.²⁰

As mentioned, here we have focused on one of the three minimal options to realise the residual symmetry $G_\nu$ in the sector of the neutral states. It could be interesting to analyse lepton mixing, as well as neutrinoless double beta decay, effects of non-unitarity of the PMNS mixing matrix ${\widetilde{U}}_\nu$, and charged lepton flavour violating processes for the other two options, called option 2 and option 3. Both these options could potentially lead to larger effects in charged lepton flavour violating processes. For option 2 the non-trivial flavour structure is encoded in the Dirac neutrino mass matrix $m_D$ and thus strongly resembles ISS constructions typically associated with sizeable predictions to numerous leptonic observables. Furthermore, a non-trivial flavour structure in $M_{NS}$ for option 3 also leads to off-diagonal terms in $\eta$, thus potentially having a strong impact on cLFV processes. Should this be the case, a study of possible correlations among the lepton mixing parameters and the different charged lepton flavour violating processes for the distinct cases (Case 1) through Case 3 b.1)) could be valuable and may even help testing the hypotheses of $G_f$, CP and the residual symmetries $G_\ell$ and $G_\nu$. Going beyond the three minimal options, we can also consider options, in which at least two of the three mass matrices $m_D$, $M_{NS}$ and $\mu _S$ carry non-trivial flavour information.

Further variants could be also envisaged. These could include versions of the inverse seesaw mechanism, for instance with two right-handed neutrinos $N_i$ and two (three) neutral states $S_j$ [46], or even a minimal radiative inverse seesaw mechanism [94]. The latter generates light neutrino masses at the one-loop level and could offer a dark matter candidate. This might be an appealing playground for a scenario with $G_f$ and CP since the same symmetries could govern the phenomenology of both the visible and the dark sector.

Appendices

Generators of $\Delta (3 \, n^2)$ and $\Delta (6 \, n^2)$ and CP transformations X

This appendix contains necessary information concerning the flavour groups $\Delta (3 \, n^2)$ and $\Delta (6 \, n^2)$. Since the former is a subgroup of the latter, we focus on $\Delta (6 \, n^2)$. The groups $\Delta (3 \, n^2)$ and $\Delta (6 \, n^2)$ are series of discrete symmetries for integer n. For $n \ge 2$, $\Delta (3 \, n^2)$ is non-abelian, while all groups $\Delta (6 \, n^2)$ have this property. The groups $\Delta (3 \, n^2)$ are isomorphic to the semi-direct product $(Z_n \times Z_n) \rtimes Z_3$ and can be described in terms of three generators a, c and d that fulfil the following relations

$$\begin{aligned}&a^3=e, \;\; c^n=e, \;\; d^n=e, \;\; a \, c \, a^{-1} = c^{-1} \, d^{-1}, \;\; \nonumber \\&a \, d \, a^{-1}= c, \;\; c \, d= d \, c \end{aligned}$$

(134)

with e being the neutral element of the group. For the groups $\Delta (6 \, n^2)$ that are isomorphic to $(Z_n \times Z_n) \rtimes S_3$, one adds the fourth generator b to the set $\{ a, c, d\}$ which fulfils the relations

$$\begin{aligned} b^2=e, \;\; (a \, b)^2=e, \;\; b \, c \, b^{-1} = d^{-1} \;\; \text{ and } \;\; b \, d \, b^{-1} = c^{-1}.\nonumber \\ \end{aligned}$$

(135)

We note that all elements of the groups can be written in terms of these generators as

$$\begin{aligned} g= & {} a^\alpha \, c^\gamma \, d^\delta \;\; \text{ and } \;\; g= a^\alpha \, b^\beta \, c^\gamma \, d^\delta \;\; \text{ with } \;\; \nonumber \\ \alpha= & {} 0,1,2, \; \beta =0, 1, \; 0 \le \gamma , \delta \le n-1, \end{aligned}$$

(136)

respectively. For the analysis of lepton mixing we are interested in the generators in the irreducible faithful (complex) three-dimensional representation $\mathbf{3}$ and in the (trivial) singlet $\mathbf{1}$. For $\mathbf{3}$ we have

$$\begin{aligned}&a (\mathbf{3}) = \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad \omega &{}\quad 0\\ 0 &{}\quad 0 &{}\quad \omega ^2 \end{array} \right) , \;\; b (\mathbf{3}) = \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad 0 &{}\quad \omega ^2 \\ 0 &{}\quad \omega &{}\quad 0 \end{array} \right) , \nonumber \\&c (\mathbf{3}) =\frac{1}{3} \, \left( \begin{array}{ccc} 1 + 2 \, \cos \phi _n &{}\quad 1- \cos \phi _n-\sqrt{3} \, \sin \phi _n &{}\quad 1- \cos \phi _n+\sqrt{3} \, \sin \phi _n \\ 1- \cos \phi _n+\sqrt{3} \, \sin \phi _n &{}\quad 1 + 2 \, \cos \phi _n &{}\quad 1- \cos \phi _n-\sqrt{3} \, \sin \phi _n\\ 1- \cos \phi _n-\sqrt{3} \, \sin \phi _n &{}\quad 1- \cos \phi _n+\sqrt{3} \, \sin \phi _n &{}\quad 1 + 2 \, \cos \phi _n \end{array} \right) \end{aligned}$$

(137)

with $\omega = e^{\frac{2 \, \pi \, i}{3}}$ and $\phi _n= \frac{2 \, \pi }{n}$, while for 1 we have

$$\begin{aligned} a (\mathbf{1}) = b (\mathbf{1}) = c (\mathbf{1}) =1. \end{aligned}$$

(138)

We note that the generator d can be obtained from the generators a and c, since we find $d= a^2 \, c \, a$ from Eq. (134).

For completeness, we list the set of used CP symmetries. CP symmetries are associated with the automorphisms of the flavour group $G_f$. In particular, the automorphism

$$\begin{aligned} a \; \rightarrow \; a, \;\; c \; \rightarrow \; c^{-1}, \;\; d \; \rightarrow \; d^{-1} \;\; \text{ and } \;\; b \; \rightarrow \; b \end{aligned}$$

(139)

for $G_f =\Delta (3 \, n^2)$ and $G_f=\Delta (6 \, n^2)$, respectively, corresponds to the CP transformation $X_0$ that is of the following form in the representations $\mathbf{1}$ and $\mathbf{3}$

$$\begin{aligned} X_0 (\mathbf{1}) = 1 \;\; \text{ and } \;\; X_0 (\mathbf{3}) = \left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad 0 &{}\quad 1\\ 0 &{} \quad 1 &{}\quad 0 \end{array} \right) . \end{aligned}$$

(140)

All other CP transformations X of interest correspond to the composition of the automorphism in Eq. (139) and a group transformation g. The CP transformation $X ({\mathbf{r}})$ in the representation ${\mathbf{r}}$ is of the form

$$\begin{aligned} X ({\mathbf{r}})= & {} g ({\mathbf{r}}) \, X_0 ({\mathbf{r}}) \;\; \text{ with } \;\; g ({\mathbf{r}}) = a ({\mathbf{r}})^\alpha \, c({\mathbf{r}}) ^\gamma \, d({\mathbf{r}}) ^\delta \;\; \text{ and } \;\; \nonumber \\ g ({\mathbf{r}})= & {} a({\mathbf{r}})^\alpha \, b({\mathbf{r}})^\beta \, c({\mathbf{r}})^\gamma \, d({\mathbf{r}})^\delta \end{aligned}$$

(141)

for $G_f=\Delta (3 \, n^2)$ and $G_f=\Delta (6 \, n^2)$, respectively, as long as $X ({\mathbf{r}})$ represents a symmetric matrix in flavour space, see Eq. (1). The CP symmetries and transformations relevant for the different cases, Case 1) through Case 3 b.1), were given in Sect. 3.

Conventions of mixing angles, CP invariants and neutrino masses

We follow the conventions of the PDG in the parametrisation of a unitary mixing matrix (W) in terms of the lepton mixing angles and the Dirac phase $\delta$ [95]

$$\begin{aligned} W= \begin{pmatrix} c_{12} c_{13} &{} s_{12} c_{13} &{} s_{13} e^{- i \delta } \\ -s_{12} c_{23} - c_{12} s_{23} s_{13} e^{i \delta } &{} c_{12} c_{23} - s_{12} s_{23} s_{13} e^{i \delta } &{} s_{23} c_{13} \\ s_{12} s_{23} - c_{12} c_{23} s_{13} e^{i \delta } &{} -c_{12} s_{23} - s_{12} c_{23} s_{13} e^{i \delta } &{} c_{23} c_{13} \end{pmatrix}\nonumber \\ \end{aligned}$$

(142)

with $s_{ij}=\sin \theta _{ij}$ and $c_{ij}=\cos \theta _{ij}$, while we define the Majorana phases $\alpha$ and $\beta$ through

$$\begin{aligned} P=\left( \begin{array}{ccc} 1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad e^{i \alpha /2} &{}\quad 0\\ 0 &{}\quad 0 &{}\quad e^{i (\beta /2 + \delta )} \end{array} \right) \end{aligned}$$

(143)

so that

$$\begin{aligned} U_{\mathrm {PMNS}} = \left( \begin{array}{ccc} U_{e1} &{}\quad U_{e2} &{}\quad U_{e3}\\ U_{\mu 1} &{}\quad U_{\mu 2} &{}\quad U_{\mu 3}\\ U_{\tau 1} &{}\quad U_{\tau 2} &{}\quad U_{\tau 3} \end{array} \right) = W \, P \end{aligned}$$

(144)

with $0 \le \theta _{ij} \le \pi /2$ and $0 \le \alpha , \beta , \delta \le 2 \, \pi$. We extract the sine squares of the lepton mixing angles as follows

$$\begin{aligned} \sin ^2 \theta _{13}= & {} |U_{e3}|^2, \;\; \sin ^2 \theta _{12} = \frac{|U_{e2}|^2}{1-|U_{e3}|^2}, \;\; \nonumber \\ \sin ^2 \theta _{23}= & {} \frac{|U_{\mu 3}|^2}{1-|U_{e3}|^2}. \end{aligned}$$

(145)

The CP phases are most conveniently extracted with the help of the CP invariants $J_{\mathrm {CP}}$ [96], $I_1$ and $I_2$ [97] ²¹

$$\begin{aligned} J_{\mathrm {CP}}= & {} \mathrm{Im} \left[ U_{e1} U_{e3}^\star U_{\tau 1}^\star U_{\tau 3} \right] \nonumber \\= & {} \frac{1}{8} \, \sin 2 \theta _{12} \, \sin 2 \theta _{23} \, \sin 2 \theta _{13} \, c_{13} \, \sin \delta \end{aligned}$$

(146)

and

$$\begin{aligned} I_1= & {} \mathrm{Im} [U_{e2}^2 (U_{e1}^\star )^2] = s^2_{12} \, c^2 _{12} \, c^4_{13} \, \sin \alpha , \;\; \nonumber \\ I_2= & {} \mathrm{Im} [U_{e3}^2 (U_{e1}^\star )^2] = s^2 _{13} \, c^2 _{12} \, c^2_{13} \, \sin \beta . \end{aligned}$$

(147)

From these, $\sin \delta$, $\sin \alpha$ and $\sin \beta$ can be computed.

The light neutrino masses $m_i$, $i=1,2,3$, can be expressed in terms of the lightest neutrino mass $m_0$ and the two measured mass squared differences $\Delta m_{\mathrm {sol}}^2$ and $\Delta m_{\mathrm {atm}}^2$, which are defined as

$$\begin{aligned} \Delta m_{\mathrm {sol}}^2=m_2^2 -m_1^2 \;\;\; \text{ and } \;\;\; \Delta m_{\mathrm {atm}}^2 = \left\{ \begin{array}{cc} m_3^2-m_1^2 &{}\quad \text{ for } \text{ NO }\\ m_3^2-m_2^2 &{}\quad \text{ for } \text{ IO } \end{array} \right. ,\nonumber \\ \end{aligned}$$

(148)

depending on the light neutrino mass ordering, NO or IO. For NO, the light neutrino masses $m_i$ read

$$\begin{aligned} m_1= & {} m_0, \;\;\; m_2= \sqrt{m_0^2 + \Delta m_{\mathrm {sol}}^2}, \;\;\;\nonumber \\ m_3= & {} \sqrt{m_0^2 + \Delta m_{\mathrm {atm}}^2}, \end{aligned}$$

(149)

while for IO we have

$$\begin{aligned} m_1= & {} \sqrt{m_0^2 + |\Delta m_{\mathrm {atm}}^2| - \Delta m_{\mathrm {sol}}^2 }, \;\; \nonumber \\ m_2= & {} \sqrt{m_0^2 + |\Delta m_{\mathrm {atm}}^2| }, \;\; m_3= m_0. \end{aligned}$$

(150)

Furthermore, we define the sum of the light neutrino masses

$$\begin{aligned} \Sigma _\nu = m_1 + m_2 + m_3, \end{aligned}$$

(151)

which is constrained by cosmological measurements.

Data on lepton mixing parameters and neutrino masses

We use the latest global fit results from the NuFIT Collaboration, NuFIT 5.0 (July 2020) [76] (without SK atmospheric data). For the lepton mixing angles we have

for NO

$$\begin{aligned}&\sin ^2 \theta _{13} = 0.02221^{+0.00068} _{-0.00062} \;\;\;\;\;\;\;\;\,\, \nonumber \\&\quad \text{ and } \;\;\; 0.02034 \le \sin ^2 \theta _{13} \le 0.02430, \nonumber \\&\sin ^2 \theta _{12} = 0.304^{+0.013} _{-0.012} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\,\nonumber \\&\quad \text{ and } \;\;\; 0.269 \le \sin ^2 \theta _{12} \le 0.343, \nonumber \\&\sin ^2 \theta _{23} = 0.570^{+0.018} _{-0.024} \;\;\;\,\;\;\;\;\;\;\;\;\;\;\;\; \nonumber \\&\quad \text{ and } \;\;\; 0.407 \le \sin ^2 \theta _{23} \le 0.618, \end{aligned}$$

(152)

and for IO

$$\begin{aligned}&\sin ^2 \theta _{13} = 0.02240^{+0.00062} _{-0.00062} \;\;\;\;\;\;\;\;\,\, \;\;\; \nonumber \\&\quad \text{ and }\, \,0.02053 \le \sin ^2 \theta _{13} \le 0.02436, \nonumber \\&\sin ^2 \theta _{12} = 0.304^{+0.013} _{-0.012} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\, \;\;\; \nonumber \\&\quad \text{ and }\,\,0.269 \le \sin ^2 \theta _{12} \le 0.343, \nonumber \\&\sin ^2 \theta _{23} = 0.575^{+0.017} _{-0.021} \;\;\;\,\;\;\;\;\;\;\;\;\;\;\;\; \;\;\; \nonumber \\&\quad \text{ and } \,\,0.411 \le \sin ^2 \theta _{23} \le 0.621. \end{aligned}$$

(153)

The experimental constraint on the Dirac phase $\delta$ reads

for NO

$$\begin{aligned} \delta =\left( 195^{+51}_{-25}\right) ^\circ \;\;\; \text{ and } \;\;\; 107^\circ \le \delta \le 403^\circ , \end{aligned}$$

(154)

and for IO

$$\begin{aligned} \delta =\left( 286^{+27}_{-32}\right) ^\circ \;\;\; \text{ and } \;\;\; 192^\circ \le \delta \le 360^\circ \;\;\; \text{ at } \text{ the } \;\; 3\,\sigma \;\; \text{ level } . \nonumber \\ \end{aligned}$$

(155)

For NO the experimental results for the mass squared differences are

$$\begin{aligned}&\Delta m_{\mathrm {sol}}^2=\left( 7.42^{+0.21} _{-0.20} \right) \times 10^{-5} \, \mathrm {eV}^2 \;\;\;\;\;\;\;\;\;\nonumber \\&\quad \text{ and } \;\;\; 6.82 \le \frac{\Delta m_{\mathrm {sol}}^2}{10^{-5} \, \mathrm {eV}^2} \le 8.04, \nonumber \\&\Delta m_{\mathrm {atm}}^2=\left( 2.514^{+0.028} _{-0.027} \right) \times 10^{-3} \, \mathrm {eV}^2 \;\;\;\;\, \nonumber \\&\quad \text{ and } \;\;\; 2.431 \le \frac{\Delta m_{\mathrm {atm}}^2}{10^{-3} \, \mathrm {eV}^2} \le 2.598, \end{aligned}$$

(156)

and for IO

$$\begin{aligned}&\Delta m_{\mathrm {sol}}^2=\left( 7.42^{+0.21} _{-0.20} \right) \times 10^{-5} \, \mathrm {eV}^2 \;\;\;\;\;\;\;\;\;\;\,\, \nonumber \\&\quad \text{ and } \;\;\; 6.82 \le \frac{\Delta m_{\mathrm {sol}}^2}{10^{-5} \, \mathrm {eV}^2} \le 8.04, \nonumber \\&\Delta m_{\mathrm {atm}}^2=\left( -2.497^{+0.028} _{-0.028} \right) \times 10^{-3} \, \mathrm {eV}^2 \;\;\;\,\, \nonumber \\&\quad \text{ and } \;\;\; -2.583 \le \frac{\Delta m_{\mathrm {atm}}^2}{10^{-3} \, \mathrm {eV}^2} \le -2.412. \end{aligned}$$

(157)

(Note that NO is currently slightly preferred over IO by experimental data with $\Delta \chi ^2=\chi ^2_{\mathrm {IO}}-\chi ^2_{\mathrm {NO}}=2.7$.)

As limit on the sum of the light neutrino masses, we use the one given by the Planck Collaboration in [81],

$$\begin{aligned} \Sigma _\nu \le 0.12 \; \mathrm {eV} \;\; \text{ corresponding } \text{ to } \;\; m_0 \lesssim 0.04 \, \mathrm {eV} \end{aligned}$$

(158)

for the lightest neutrino mass. The experimental limits from KATRIN [99] and from the searches for $0\nu \beta \beta$ decay [82,83,84,85,86] do not pose stronger constraints on the lightest neutrino mass $m_0$ than the ones obtained from cosmology.

Numerical treatment and fit procedure

Predicting the CP phases for the different choices of the CP symmetry requires identifying the values of the free angle $\theta _S$ which lead to a set of lepton mixing parameters in agreement with experimental data. The free angle $\theta _S$ is fit by maximising the joint likelihood of the predictions for $\sin ^2\theta _{ij}$. As mentioned in Appendix C, we refer to the combination of experimental data (latest update on global fits) provided by NuFIT 5.0 [76]. In order to fit $\theta _S$ with high accuracy, and especially to take into account the “double well” structure in $\sin ^2\theta _{23}$, we interpolate the numerical $\Delta \chi ^2$ values (available at [76]), and linearly extrapolate beyond the provided ranges to ensure a smooth behaviour for arbitrary input values. The interpolated $\chi ^2$ functions are then transformed into probability distributions so that a global joint likelihood of all relevant parameters ($\sin ^2\theta _{ij}$ and, in the case of the (3,3) ISS framework, also $\Delta m_{ij}^2$) can be constructed. To ensure that the cosmological bound on the sum of light neutrino masses is respected, a half-normal distribution (as a gaussian upper limit) is further included. We first fit predictions for $U_S$ and therefore $\theta _S$ on the data for $\sin ^2\theta _{ij}$, from which we proceed to consider the model-dependent, i.e. (3,3) ISS framework. This is done by maximising the joint likelihood function using the migrad algorithm of the iminuit library [100]. Local maximum likelihood estimators lying outside of the global $3\,\sigma$ region around the (experimental) best-fit point are rejected.

Due to the peculiar structure and almost degenerate heavy states in the ISS mass matrix, a large numerical precision ($\sim 100$ digits) is needed for a reliable matrix diagonalisation. This is achieved using the mpmath python library [101] and algorithms within. To study the effects of the heavy sterile states on the predictions for lepton mixing parameters, we use “effective” mixing angles (and phases) which we define as in Appendix B (Eqs. (145, 146, 147), with $U_\text {PMNS}$ replaced by ${\widetilde{U}}_\nu$). The free angle $\theta _S$ then needs to be re-fitted, using the results obtained within the model-independent approach as starting values for the fit, thus allowing to study deviations from those predictions.

Keeping the lightest neutrino mass $m_0$ fixed – and thus the lightest Majorana mass – ($\mu _1$ or $\mu _3$ depending on the ordering of the light neutrino mass spectrum), the remaining two Majorana masses $\mu _i$ are treated as free parameters to be determined by a fit to $\sin ^2\theta _{ij}$ and $\Delta m_{ij}^2$ data. The starting values for $\mu _i$ are determined by inverting the leading order expression given in Eq. (17) and a modified Casas–Ibarra parametrisation [102]

$$\begin{aligned} \mu _S \,\simeq \,M_{NS}^T \,m_D^{-1} \,U^\star _S \, m^\text {diag}_\nu \,U^\dagger _S \,(m_D^T)^{-1} \,M_{NS}, \end{aligned}$$

(159)

where $U_S$, the matrix which diagonalises $\mu _S$ and at leading order also the light neutrino mass matrix, is determined by the flavour symmetry $G_f$ and CP and the residual symmetry $G_\nu$. For choices of $y_0$ and $M_0$ in conflict with bounds on the unitarity of ${{\widetilde{U}}}_\nu$, sizeable departures from the model-independent results are observed, as described in the main body of the paper.

Loop functions

The loop functions and their relevant limits²² are taken from Refs. [87, 89]. The photon dipole and anapole functions, as well as some relevant limits, are given by

$$\begin{aligned}&F_\gamma (x) = \frac{7 x^3 - x^2 - 12x}{12(1-x)^3} - \frac{x^4 - 10x^3 + 12x^2}{6(1-x)^4}\log x,\nonumber \\&F_\gamma (x) \xrightarrow [x\gg 1]{} -\frac{7}{12} - \frac{1}{6}\log x,\nonumber \\&F_\gamma (0) = 0, \end{aligned}$$

(160)

$$\begin{aligned}&G_\gamma (x) = -\frac{x(2x^2 + 5x - 1)}{4(1-x)^3} - \frac{3x^3}{2(1-x)^4}\log x,\nonumber \\&G_\gamma (x) \xrightarrow [x\gg 1]{} \frac{1}{2},\nonumber \\&G_\gamma (0) = 0. \end{aligned}$$

(161)

The functions associated with the Z penguins are given by a two-point function

$$\begin{aligned}&F_Z(x) = -\frac{5 x}{2(1 - x)} - \frac{5x^2}{2(1-x)^2}\log x,\nonumber \\&F_Z(x) \xrightarrow [x\gg 1]{} \frac{5}{2} - \frac{5}{2}\log x,\nonumber \\&F_Z(0) = 0, \end{aligned}$$

(162)

and two three-point functions which are symmetric under interchange of the arguments.

$$\begin{aligned}&G_Z(x,y) = -\frac{1}{2(x-y)}\left[ \frac{x^2(1-y)}{1-x}\log x - \frac{y^2(1-x)}{1-y}\log y \right] ,\nonumber \\&G_Z(x, x) = -\frac{x}{2} - \frac{x\log x}{1-x},\nonumber \\&G_Z(0,x) = -\frac{x\log x}{2(1-x)},\nonumber \\&G_Z(0,x)\xrightarrow [x\gg 1]{} \frac{1}{2}\log x,\nonumber \\&G_Z(0,0) = 0, \end{aligned}$$

(163)

$$\begin{aligned}&H_Z(x,y) = \frac{\sqrt{xy}}{4(x-y)}\left[ \frac{x^2 - 4x}{1 - x}\log x - \frac{y^2 - 4y}{1 - y}\log y\ \right] ,\nonumber \\&H_Z(x,x) = \frac{(3 - x)(1-x) - 3}{4(1-x)} - \frac{x^3 - 2x^2 + 4x}{4(1-x)^2}\log x,\nonumber \\&H_Z(0,x) = 0. \end{aligned}$$

(164)

The (symmetric) box-loop-functions and their limits are given by

$$\begin{aligned}&F_\text {box}(x,y) = \frac{1}{x-y}\left\{ \left( 4 + \frac{xy}{4}\right) \left[ \frac{1}{1-x} + \frac{x^2}{(1-x)^2} \log x\right. \right. \nonumber \\&\qquad \left. - \frac{1}{1-y} - \frac{y^2}{(1-y)^2}\log y\right] \nonumber \\&\qquad -2xy\left[ \frac{1}{1-x} + \frac{x}{(1-x)^2} \log x \right. \nonumber \\&\qquad \left. \left. - \frac{1}{1-y} - \frac{y}{(1-y)^2}\log y \right] \right\} ,\nonumber \\&F_\text {box}(x,x) = -\frac{1}{4(1-x)^3}\left[ x^4 - 16x^3 + 31x^2\right. \nonumber \\&\qquad \left. - 16 + 2x\left( 3x^2 + 4x - 16\right) \log x\right] ,\nonumber \\&F_\text {box}(0,x) = \frac{4}{1 - x} + \frac{4x}{(1-x)^2}\log x,\nonumber \\&F_\text {box}(0,x)\xrightarrow [x\gg 1]{} 0,\nonumber \\&F_\text {box}(0,0) = 4, \end{aligned}$$

(165)

$$\begin{aligned}&F_\text {Xbox}(x,y) = -\frac{1}{x-y}\left\{ \left( 1 + \frac{xy}{4} \right) \left[ \frac{1}{1-x} + \frac{x^2}{(1-x)^2} \log x \right. \right. \nonumber \\&\qquad \left. - \frac{1}{1-y} - \frac{y^2}{(1-y)^2}\log y\right] \nonumber \\&\qquad -2xy\left[ \frac{1}{1-x} + \frac{x}{(1-x)^2} \log x \right. \nonumber \\&\qquad \left. \left. - \frac{1}{1-y} - \frac{y}{(1-y)^2}\log y \right] \right\} ,\nonumber \\&F_\text {Xbox}(x,x) = \frac{x^4 - 16x^3 + 19x^2 - 4}{4(1-x)^3} + \frac{3x^3 + 4x^2 - 4x}{2(1-x)^3}\log x,\nonumber \\&F_\text {Xbox}(0,x)= -\frac{1}{1-x} - \frac{x}{(1 - x)^2}\log x,\nonumber \\&F_\text {Xbox}(0,x)\xrightarrow [x\gg 1]{} 0,\nonumber \\&F_\text {Xbox}(0,0) = -1, \end{aligned}$$

(166)

$$\begin{aligned}&G_\text {box}(x,y) = -\frac{\sqrt{xy}}{x-y}\left\{ (4 + xy)\left[ \frac{1}{1-x} + \frac{x}{(1-x)^2} \log x \right. \right. \nonumber \\&\qquad \left. - \frac{1}{1-y} - \frac{y}{(1-y)^2}\log y\right] \nonumber \\&\qquad \left. -2\left[ \frac{1}{1-x} + \frac{x^2}{(1-x)^2} \log x - \frac{1}{1-y} - \frac{y^2}{(1-y)^2}\log y \right] \right\} ,\nonumber \\&G_\text {box}(x,x) = \frac{2x^4 - 4x^3 + 8x^2 - 6x}{(1-x)^3} - \frac{x^4 + x^3 + 4x}{(1-x)^3}\log x,\nonumber \\&G_\text {box}(0,x) = 0. \end{aligned}$$

(167)