Introduction

Since the discovery of time-reversal invariant (Z2) topological insulators1,2,3,4 a decade ago, there has been a major increase in interest in topological condensed matter systems, due to both scientific interest and potential applications.4,5,6,7,8,9,10 Despite these efforts, it remains challenging to find robust materials with realizations of many phases, especially those with broken time reversal symmetry (TRS). For example, Chern insulators,11 which are two-dimensional insulators that display the quantum anomalous Hall effect (AHC), were first demonstrated by Haldane in 1988,12 but the first material realizations were not created until recently.13 Current quantum anomalous Hall materials, based on magnetically doped topological insulators,13,15,15,16,17 are limited to very low temperatures (~1 K),13 but there is no intrinsic reason for this limit, and there remains significant interest19,19,20,21,22,23 in finding Chern insulators with larger band gaps and higher magnetic ordering temperatures.24,24,25,26,27,28

There are many additional related topological phases with broken TRS that are also difficult to realize experimentally. For example, Weyl semimetals (WSM)29,30,31 are materials with topologically protected linearly dispersing band crossings, known as Weyl points, near the Fermi level. WSM require the breaking of at least one among inversion symmetry and TRS.33,33,34,35,36,37 Broken inversion WSM have been studied experimentally,39,39,40,41,42,43,44,45,46 while TRS-broken WSMs have been more difficult to verify.36,48,48,49,50 TRS-broken WSM have different and in many cases simpler properties than broken-inversion WSM, such as the potential to have the minimum two Weyl points in the Brillioun zone, as well as nonzero anomalous Hall conductivity (see discussion in SM).33,34 Furthermore, the interplay between TRS invariant topological insulators and broken TRS materials can lead to a variety of interesting behaviors related to the quantized topological contribution of the magneto-electric effect: \(S_\theta = \frac{\theta }{{2\pi }}\frac{{e^2}}{h}{\int} {{\mathrm {d}}^3x{\mathrm {d}}t\overrightarrow E \overrightarrow B }\),2,18,51,52 where θ is a dimensionless phase that ranges from 0 to 2π. θ = 0 in trivial insulators with TRS, but θ = π in Z2 topological insulators with TRS, making Z2 materials potentially useful magnetoelectrics.53,54,55 However, to observe this effect, it is necessary to remove the topologically protected metallic surface state of the Z2 material by breaking TRS on the surface.56,57,58 Axion insulators, including antiferromagnetic (AFM) topological insulators,59 are materials with intrinsically broken TRS, but with θ = π protected by crystal symmetry instead of TRS.18,60

In this work, we investigate the electronic properties of Bi2MSe4 (BMS, M = Pb, Mn) using density functional theory (DFT) and Wannier-based tight-binding models, and we predict a series of topologically non-trivial phases. Recently, we have reported the growth of a thin film of magnetic systems with stoichiometric composition Bi2MnSe4 by molecular beam epitaxy (MBE).61 We demonstrated that the introduction of an elemental beam of Mn during the molecular beam epitaxial growth of Bi2Se3 results in the formation of layers of Bi2MnSe4 that intersperse layers of pure Bi2Se3. Here, we concentrate on the topological properties of this crystal structure. First, we look at Bi2PbSe4, which we find to be a Z2 topological insulator due to spin–orbit-induced band inversion at the Z point. In the rest of this letter, we focus on combining this band inversion, which is common in this crystal structure, with the magnetic properties present in Bi2MnSe4. We find that depending on the symmetry of the magnetic ordering and the sample thickness, Bi2MnSe4 can access many different topological phases with broken TRS: a nodal line system, magnetic Weyl semimetal, AFM topological insulator, or Chern insulator, in addition to displaying the half-integer quantum AHC. The band inversion of this system is the fundamental driving force linking all these non-trivial topological phases. However, the magnetic ordering and dimensionality controls the symmetry of the bands near the Fermi level, which changes whether and how the inverted bands form a semimetal or insulating phase, as well as determining which topological invariants and topological surface and edge features are relevant. We expect that it should be possible to observe these closely related phases experimentally by manipulating the sample thickness, as well as the spin order by using perturbations like an external magnetic field or temperature. In this work, we consider three different materials, two of which (Bi3Se4 and Bi2MnSe4) have already been synthesized. Bi2PbSe4 has not yet been synthesized. All these materials are stoichiometric and have significant band gaps, which should help make these topological phases robust and possible to observe at increased temperature.

Results

Bi2Se3 and Bi2PbSe4

Bi2MSe4, like Bi2Se3, is a layered material, with \(R\bar 3m\) space group (Fig. 1). The Bi2MnSe4 septuple layer (SL) shown in Fig. S1a, is a 2D layer consisting of seven atomic layers with stacking sequence of Se–Bi–Se–M–Se–Bi–Se along the c-axis (Fig. 1). The individual 2D layers are weakly bound to each other by van der Waals forces. In this work, we considered bismuth (Bi), lead (Pb), or manganese (Mn) as the metal atom (M), which is the inversion center. The optimized lattice parameters of BMS have been tabulated in Table S1.

Fig. 1
figure 1

Crystal structure of Bi2MSe4 (M = Bi, Pb, Mn). Here M is the inversion center of the crystal structure, whereas yellow, green, and blue sphere are the Se, Bi, and M atom, respectively

We begin our discussion of the Bi2MSe4 structure by considering Bi3Se4.62 Each septuplet building block of Bi3Se4 is comprised of seven atomic layers (Se–Bi1–Se–Bi2–Se–Bi1–Se) along the c-axis and separated by a van der Waals gap. The optimized lattice constants (aBMS = 4.28 Å and cBMS = 40.9 Å) are in good agreement with experimental results.62 Relaxed structural parameters, such as bond length and intralayer distance, have been tabulated in Table S1 (Supplementary). As expected by electron counting, our band structure calculations reveal that Bi3Se4 is a metallic system,63 as shown in Fig. S1. However, we expect that substituting a (+2) ion for one Bi will exactly fill the Se states, possibly opening a gap.

Accordingly, we replace the Bi2 atom by with Pb, creating Bi2PbSe4 (BPS). The calculated bulk band structure, shown in Fig. 2a, demonstrates that BPS has a bulk band gap of 0.15 eV. We calculate the topological invariants of BPS, and we find that, like the closely related Bi2Se3, BPS is a strong (Z2) topological insulator, although the topological band inversion happens at the Z point instead of the Γ point in Bi2Se3. This is confirmed by the surface local density of states (LDOS) (Fig. 2b), calculated using our Wannier tight-binding parameters, which clearly shows a single Dirac cone at the Г point in the surface band structure.

Fig. 2
figure 2

a The bulk band structure of Bi2PbSe4 including spin–orbit. b Energy and momentum dependence of local density of states (LDOS) of the (001) surface of Bi2PbSe4. Here, the red region shows the bulk energy bands and the blue region shows the energy gap

Bi2MnSe4

Encouraged by these results, we next investigate the magnetic material Bi2MnSe4 (BMS), which we will concentrate on for the rest of the paper. We will first consider bulk phases of BMS with different magnetic orderings, and then consider thin film geometries. Consistent with Otrokov et al.64, who studied Bi2Se3/Bi2MnSe4 interfaces, we find that the Mn-spins prefer to orient ferromagnetically in the case of a single BMS layer. We focus on the topological properties of two spin orderings, ferromagnetic and AFM, with alternating layers along the (001) direction (Fig. S3). We find that AFM BMS is 1.5 meV per formula unit lower in energy than the ferromagnetic phase. In addition, we find that spins prefer to orient along the Z direction, with in-plane spins higher in energy by 0.2 meV/Mn. In order to calculate the magnetic ordering temperature, we fit the spin–spin interactions in this system to an Ising model, which we use to calculate an upper bound Neel temperature of 75 K (see supplementary materials section SV for details). Given the small energy differences in the system, we expect that it may be possible to control the spin direction and potentially even the magnetic ordering of the system with an external magnetic field or with other small perturbations like doping or strain. Therefore, we study both ferromagnetic spin ordering, with spins along the Z and X directions, as well as the ground state AFM spin ordering. We summarize the different phases we consider in Table 1.

Table 1 Table summarizing the various magnetic orderings and layer thicknesses we consider in this work that have non-trivial topology

We first investigate the ferromagnetic band structure with spins oriented in the (001) direction, as shown in Fig. 3a. The red color in Fig. 3a, shows projection onto Bi states. We can see that there is band in inversion at Г, with the Γ4+ band, which has Bi character, located well below the Fermi level. We also note that there is spin splitting on the order of 0.1 eV, even though the states near the Fermi level all originate from Bi and Se. At the Fermi level, there are two linearly dispersing band crossing points, at ±0.021 Ang.−1 along the line from Г to Z, shown more clearly in Fig. 3b. By calculating the Chern number on a surface surrounding each point, we show that the crossings are topologically protected Weyl points due to band inversion at Г of bands with Г4− and Г5− symmetry. Along the line from Г to Z, the bands that cross at the Weyl points are characterized by Г4 and Г5 symmetry; however, we emphasize that the Weyl points are topologically protected. Breaking symmetries can move the Weyl crossings off the high symmetry line, but it cannot gap them. Unlike WSM with TRS but broken inversion symmetry, which must have at least four Weyl points, Bi2MnSe4 has the minimum two Weyl points, and they are both exactly at Fermi level due to inversion.33 As shown in Fig. S2, we confirm the topological character of the bulk by calculating the surface band structure using our Wannier tight-binding model, finding the expected surface Fermi arcs connecting the two Weyl points. We present a two-band tight-binding model of a magnetic Weyl semimetal and discuss additional properties in section SIV of the supplementary materials.

Fig. 3
figure 3

a The bulk band structure of ferromagnetic Bi2MnSe4 with spin aligned along \(\hat z\) direction. The colorbar shows projection onto Bi-centered Wannier functions. b and c Energy spectrum of ferromagnetic Bi2MnSe4 near Г in the kykz plane with energy along the z-axis. Spins along z in b, spins along x in c

Next, we again consider the ferromagnetic spin ordering, but with spins in-plane along the (100) direction. The band structure is very similar to the previous case, except near the band crossing, where as shown in Fig. 3c, instead of two Weyl points, we find an elliptical nodal line around the Г point in the ky, kz plane. This nodal line appears because aligning spins in the (100) direction does not break mirror symmetry in this plane. Even after considering the effects of spin–orbit coupling, the bands in this plane can be classified by their mirror eigenvalues, which results in a symmetry-protected Weyl nodal line in the presence of band inversion, rather than isolated Weyl points. The sensitivity of the Fermi surface of this material to the direction of the spins, which may be possible to control with an external field, makes it a rich playground for topological behavior, all of which can be traced back to the fundamental band inversion at Г.

Now, we discuss the ground state antiferromagnetic (AFM) phase of Bi2MnSe4, the band structure of which is shown in Fig. 4a. Unlike the ferromagnetic phases, AFM Bi2MnSe4 has a symmetry operation that combines TRS with a partial translation along the z direction, and therefore allows for a classification by a Z2 invariant for layered antiferromagnetic topological insulators.59 We use Wannier charge centers65,66 to calculate the topological invariant of this phase (see Fig S8). Again due to band inversion at Г, we find that the AFM phase is a non-trivial antiferromagnetic topological insulator. However, we note that, unlike a typical non-magnetic topological insulator, the (001) surface of AFM BMS will break time-reversal, due to the loss of translation symmetry along z. This change leads to surface properties that differ from a typical Z2 material, as the Dirac cone feature that is characteristic of topological insulators, including Bi2PbSe4 as shown in Fig. 2b, is normally protected by TRS, and TRS is absent in this material. AFM topological insulators are part of the broader class of materials known as axion insulators.18,60 Axion insulators have broken TRS, but still have a non-trivial axion phase angle θ = π, with a topological classification protected by inversion symmetry. Axion insulators are ideal systems to observe the more exotic magneto-electric behaviors of topological insulators with broken TRS in a single-phase material, without needing to engineer an interface between a topological insulator and a magnetic insulator. The most dramatic feature of AFM topological insulators is the half-integer quantum anomalous Hall effect, where both the top and bottom surfaces will contribute ± (e2/2h) to the Hall conductivity, depending on the direction of spins at each surface.18,59 If there are an even number of layers, the total AHC of the slab will be zero, but each surface will separately contribute |e2/2h|, and if there are an odd number of layers, the slab will have a non-zero Chern number and AHC of |e2/h|.

Fig. 4
figure 4

a The bulk band structure of antiferromagnetic Bi2MnSe4 with spin aligned along \(\hat z\) direction. Colors as in Fig. 3. b AHC of 31 layer (solid blue) and 32 layer (dashed red) tight-binding slab of antiferromagnetic Bi2MnSe4, separated into contributions of each layer pair, in units of e2/h. c Cumulative AHC, summing over layers from b

We demonstrate the half integer QAH effect in BMS in Fig. 4b, where we use our Wannier tight-binding model of bulk AFM BMS to calculate the geometric contribution to the AHC of 31 and 32 layer thick slabs.67 As shown in the supplementary materials section SVI, individual spin-up and spin-down layers have strongly alternating contributions to the AHC, but by averaging over pairs of layers, we can clearly see the net contribution to the AHC of each surface. As shown in Fig. 4c, the cumulative contribution to the AHC of the bottom surface in both the 31 and 32 layer slabs is +0.5e2/h. In the 31 layer case, the top surface also contributes +0.5e2/h, for a net AHC of +1.0e2/h, while in the 32 layer case, the top layer contributes −0.5e2/h, for a net AHC of zero. This sensitivity to thickness and magnetic ordering at the surface will result in chiral conducting channels at either step edges or AFM domain walls, which are characteristic of axion insulators.18,59,60

Both WSM and AFM topological insulators, which are bulk three-dimensional phases, have topological contributions to the AHC and are closely related to two-dimensional quantum anomalous Hall insulators (a.k.a. Chern insulators). To investigate possible Chern insulating phases in detail, we perform first principles slab calculations with a variety of thicknesses and magnetic orders, with (001) surfaces exposed. In the case of a ferromagnetic slab with spins along the z direction, we find that one and two layer thick slabs are topologically trivial, due to confinement increasing the band gap and preventing band inversion. For three layer and thicker slabs, a band inversion occurs at Г, and the material becomes a Chern insulator with C = −1. In Fig. 5a, we show the band structure of the three layers case, which has a band gap of 29 meV. We show other band structures with different thickness of FM and AFM systems in the supplementary materials (Fig. S3, S4). In Fig. 5b, we calculate the edge state of the slab, finding a single spin-polarized edge channel, as expected for a Chern insulator. We also find a Chern insulator in the four layers case, with a gap of 58 meV. As the material converges to bulk-like behavior, the gap should eventually close and the AHC will be proportional to the distance between the Weyl points, as discussed in section SIV of the supplementary materials.68,69

Fig. 5
figure 5

a Band structure of 4SL-Bi2MnSe4 with FM spins. Colors as in Fig. 3. b Topologically protected edge states of same system, on left and right edges. c Band structure of 5SL-Bi2MnSe4 with AFM spins

For AFM slabs, we find that all slabs from one to five layers are topologically trivial, but with decreasing band gaps from 240 meV for two layers down to 22 meV for five layers, as shown in Fig. 5c. To match the bulk calculations, this trend must continue, and thicker slabs will show a band inversion and a Chern insulating phase for odd numbers of layers, as discussed above. Using our Wannier model, we confirm that any perturbation that closes the band gap of the five-layer slab, for instance lowering the Bi on-site energies, results in a Chern insulator.

Discussion

We study the topological behavior of Bi2MnSe4. We find that like many materials with structures related to Bi2Se3, including Bi2PbSe4, Bi2MnSe4 displays spin–orbit-induced band inversion, leading to topological phases. However, this band inversion interacts with the broken TRS due to the Mn, leading to a variety of broken TRS topological phases that have either rarely or never been observed in stoichiometric single-phase materials. The symmetry of the magnetic ordering and the dimensionality of the system control which topological phases and invariants are possible, but the phases are all fundamentally driven by the spin–orbit-induced band inversion. Bulk FM spin ordering leads to symmetry-protected band crossings at the Fermi level, resulting in either Weyl points or Weyl nodes, depending of the direction of the spins. The ground state AFM structure is an AFM topological insulator, with axion phase angle θ = π. In thin film form, either magnetic ordering will result in two-dimensional Chern insulators in sufficiently thick slabs. Due to the relatively small energy differences between the various phases and the layered structure of Bi2MnSe4, we expect that it should be possible to observe most of these phases experimentally using magnetic fields or other perturbations, as well as careful crystal growth or exfoliation. We hope that due to its strong magnetic interactions and significant band gaps, Bi2MnSe4 will prove a fruitful material for studying TRS-broken topological phases in a stoichiometric compound and at higher temperatures.

The purpose of identifying the computer software in this article is to specify the computational procedure. Such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology.

After submitting this manuscript for publication, several independent works on Bi2MnTe4 showing related topological phases to this work have been submitted.70,71,72

Method

Calculations were carried out using density-functional theory (DFT)73,74 as implemented in QUANTUM ESPRESSO code.75 We used the PBEsol generalized gradient approximation as exchange and correlation potential.76 We have used fully relativistic norm-conserving pseudopotentials.77,78 We use a plane-wave cutoff of 70 Ry, a 10 × 10 × 6 Monkhorst–Pack79 k-point mesh for bulk geometries and a 8 × 8 × 1 k-point grid for slab geometries. For calculations with Mn, we use DFT+U80,81 with U = 3 eV, although we find that our results are robust to changes in the value of U, as the states near the Fermi level are not predominantly Mn states. In addition, we use hybrid functional calculations to assess the influence of exact exchange on our band structures, which we present in section SVIII of the supplementary materials. All the geometric structures are fully relaxed until the force on each atom is <0.002 eV/Å, and the energy-convergence criterion is 1 × 10−6 eV. Results from our DFT calculations are then used as input to construct maximally localized Wannier functions using WANNIER90.82,83 Topological numbers and band gaps are calculated using Wannier interpolation of the band structure using the Wannier-TOOLS package.84

In addition to the DFT+U calculations presented in the main work, we use hybrid functional HSEsol calculations performed with VASP85,86 to assess the influence of exact exchange (x) on both Bi2Se3 and the FM phase of Bi2MnSe4. Consistent with previous works,87 we find the best agreement between hybrid functional and experimental Bi2Se3 bands structures occurs for x~0.05. We find that the Bi2MnSe4 is in the Weyl semimetal phase for x < 0.115, which is within the range preferred by Bi2Se3, justifying our use of semilocal functionals. See Section SVIII of the supplementary materials for full details.