Introduction

Although childhood and adolescence are considered comparatively healthy periods of life, especially in industrialised countries, researchers point to an epidemiological shift that places greater emphasis on non-communicable diseases (NCDs) such as mental health problems [1]. Indeed, an analysis of the Global Burden of Disease (GBD) 2015 study revealed that mental disorders were the second leading cause of disability-adjusted life years (DALY) among children aged 5–14 years in America and Europe [2]. A meta-analysis of 41 studies from 27 countries estimated a global pooled prevalence of mental disorders in children and adolescents of 13.4%, with the highest prevalence rates found for anxiety and disruptive disorders [3]. Summarising studies using the General Health Questionnaire (GHQ-12), a recent review found a global prevalence of common mental disorders of 25 to 31% depending on the cut-off value used [4]. However, the mental health of young people has deteriorated since the COVID-19 pandemic. In their recent rapid review, Schlack et al. [5] found evidence of significant increases in the proportion of children and adolescents with lower quality of life, higher levels of stress or more symptoms of specific mental disorders, particularly in the early phase of the pandemic. Results from the German longitudinal COPSY study indicate an increase in overall mental health problems from 18% (pre-pandemic) to 29% (mid-2020) to 31% (early 2021), which remained at a high level in the last wave (end 2021, [6]).

Based on these epidemiological findings, there is a strong need for interventions beginning in early childhood before the onset of mental health problems. Schools have long been identified as a key setting for (mental) health promotion and prevention for several reasons. First, it is argued that because of compulsory schooling, a large proportion of children and adolescents, regardless of their social, economic and cultural backgrounds, can be easily reached by school-based interventions. Second, schools are not only places that provide access to young people; they can also influence pupils’ mental health through environmental factors. Although the scientific evidence is heterogeneous, studies suggest that negative classroom learning environments (e.g. classrooms with fewer material resources and whose teachers receive less respect from colleagues) and low school climate (teacher ratings) are associated with lower mental health of pupils [7, 8]. Third, cross-sectional and longitudinal studies have found associations between key educational outcomes and mental health indicators, suggesting that mental health is an important educational resource. Analyses of large Norwegian longitudinal data show that externalising and internalising problems were associated with reductions in upper secondary school completion ranging from 33 to 54% and that this association remained significant even after controlling for family background and school characteristics [9].

In light of these arguments, a number of comprehensive intervention approaches have been developed that focus not only on individual behaviour change but also on organisational change by strengthening the wider physical and social environment and by targeting not only pupils but also teachers and parents [10,11,12]. However, the evidence base for these whole-school approaches is still limited and inconsistent. In their Cochrane systematic review, Langford and colleagues [13] found small intervention effects for Health Promoting School interventions on a wide range of outcomes (e.g. BMI, physical activity, fruit and vegetable consumption), but not on mental health. However, in terms of age, this refers to older children, not primary school children. In contrast, Weare and Nind’s [14] review of reviews identified small low-to-moderate effects of school-based interventions on mental health and social, emotional and academic outcomes, with greater effects for interventions with longer duration, a balance of universal and targeted approaches, use of a multimodal/whole-school approach and high implementation fidelity. In addition, findings from a meta-analysis of 45 studies showed small improvements from whole-school interventions on pupils’ social and emotional skills, externalising and internalising symptoms. The heterogeneity of these results may be explained, among other things, by the differences in methodological criteria or the range of interventions included. While the Cochrane review by Langford et al. [13] focused exclusively on cluster-randomised controlled trials examining interventions based on the Health Promoting School approach, all the other reviews also included studies using a quasi-experimental design with a comparison group or other designs. Moreover, the review by Weare and Nind [14] included a wide range of universal and targeted interventions. In terms of school type, there is also a scarcity of evidence for whole-school interventions such as MindMatters, particularly for primary schools. Existing evaluation studies for primary schools focus primarily on behavioural prevention programmes rather than on holistic interventions [15].

In summary, uncertainties remain in the evidence for whole-school approaches to mental health promotion, partly due to weak study designs. MindMatters is a comprehensive German school-based intervention that extends from primary to secondary schools and includes curriculum materials and elements aimed at developing the school ethos and environment, taking into account the wider school environment. Previous German evaluation studies have focused on secondary schools and used a questionnaire‐based pre‐post design. While teachers reported improvements in school quality, pupils were found to have lower levels of psychovegetative complaints and psychological distress after programme implementation [16, 17]. The following paper presents the study protocol for the evaluation of the primary school module of MindMatters.

Aims and hypothesis

The study aims to replicate the intervention effects of MindMatters previously identified for primary schools [16, 17] and to examine the additional effects. Based on a confirmatory multicentre trial, a more rigorous study design will be applied, including a comparable control group. At the pupil level, the focus will be on social-emotional skills, mental health and behavioural problems. In addition to replicating the results at the pupil level, the evaluation aims to examine the changes at the organisational and classroom level, e.g. in terms of improved learning and social climate as perceived by teachers. The study will also examine whether pupils’ academic performance can be improved by strengthening their mental health.

The following hypotheses will be tested:

  1. 1.

    Pupils in intervention schools show higher levels of emotion-related knowledge than pupils in control schools.

  2. 2.

    Pupils in intervention schools have higher levels of social-emotional skills and lower levels of mental health and behavioural problems at post-test compared to pupils in control schools.

  3. 3.

    Teachers in intervention schools report improved diagnostic skills, improved learning and social climate and reduced classroom disruption at post-test compared to teachers in control classes.

  4. 4.

    Parents and teachers of pupils in intervention schools report improved academic performance and learning behaviour compared to parents and class teachers in control schools.

  5. 5.

    Intervention schools with higher implementation quality show greater effectiveness in terms of pupils’ social-emotional skills and mental health, and classroom outcomes such as bullying, classroom climate and disruption, compared to schools with lower implementation quality.

Methods and analyses

The MindMatters intervention

Originally developed in Australia in the late 1990s, MindMatters is a universal mental health promotion intervention for primary and secondary schools that takes a holistic approach to school health promotion. The German adaptation (www.mindmatters-schule.de) took place from 2002 to 2005, with 32 schools implementing the intervention modules over an 18-month period. It is based on the concept of the ‘Good and Healthy School’, which proposes a close link between health and education. This approach starts from the educational quality of the school and tries to promote this quality through health interventions. Health is thus seen as a driving force or an essential determinant of successful educational processes [18]. In line with the Health Promoting School approach, the Good and Healthy School is a holistic or whole-school approach that addresses different target groups (pupils, teachers, parents) and, in addition to individual behaviour, the wider school environment. In addition to the general promotion of mental health and the prevention of mental disorders in pupils, MindMatters aims to:

  • Develop a school culture where all school members feel safe, valued and included

  • Support schools in building or developing a caring school culture and networking between schools and the school environment

  • Contribute to improvements in teaching and learning, thereby enhancing the quality of education in schools

For secondary schools, the intervention consists of a total of nine modules. Three of these modules are related to school development and address the development of school structures, policies and community partnerships to promote mental health. In addition, there are five subject-specific teaching modules dealing with topics such as preventing bullying, making friends, dealing with stress, mental disorders and experiences of loss and grief (Additional file 1: Overview of the German-language MindMatters intervention).

Since 2012, a separate intervention module for primary schools called ‘Learning Together with Emotions’ has been launched [19]. It supports primary schools in promoting social-emotional learning (SEL) in the classroom and is based on the five broad and interrelated domains of competence developed by the Collaborative for Academic, Social and Emotional Learning [20]. These include the following:

  1. 1.

    Self-awareness, i.e. the ability to understand one’s own emotions, thoughts and values

  2. 2.

    Self-management, i.e. the ability to manage one’s emotions, thoughts and behaviour effectively in different situations

  3. 3.

    Responsible decision-making, i.e. the ability to make caring and constructive choices about personal behaviour and social interactions in different situations

  4. 4.

    Relationship skills, i.e. the ability to establish and maintain healthy and supportive relationships and to navigate an environment with diverse individuals and groups

  5. 5.

    Social awareness, i.e. the ability to understand the perspectives of and empathise with others

Each competence area is covered by a teaching unit with five exercises and a final activity (e.g. a game, a song), which can be used by the teacher as a flexible resource at the curricular level (i.e. 30 activities in total). There are different variations for each exercise, which allows the intervention to be adapted to the individual needs of the pupils (different learning levels, e.g. exercises for children with and without literacy skills). Each exercise takes between 15 and 30 min. to complete, depending on the variation chosen. In addition, there are seven exercises that can be used to complement or introduce the units (e.g. setting class rules, finding each other in groups, pupils as researchers). As shown in Additional file 1, the teaching module is complemented by the SchoolMatters school development module. Designed as a basic module, SchoolMatters aims to support schools in creating structures and processes to maintain and promote mental health (e.g. conducting a needs assessment, developing a project plan, establishing a MindMatters school team, building partnerships with community stakeholders).

The intervention materials are primarily considered as a flexible resource to be used by the teacher according to the needs of the school. Although a sequence of exercises is provided within the teaching module, there is a low degree of standardisation in the implementation of the intervention. The main target group for implementation are teachers, school principals and school teams who can voluntarily attend a MindMatters training. A website (www.mindmatters-schule.de) provides training dates and access to a toolbox of further classroom exercises.

Study design

This study is an ongoing two-arm cluster-randomised controlled superiority trial (CRCT) with baseline and post-measurement. The study protocol follows the standard protocol for clinical trials according to the SPIRIT 2013 statement (Fig. 1; Additional file 2: SPIRIT checklist) [21].

Fig. 1
figure 1

SPIRIT figure

Full size image

Primary schools are eligible to participate if (1) they are from one of the three federal states (North Rine-Westphalia, Rhineland-Palatinate, Schleswig–Holstein), (2) they have not implemented the MindMatters intervention in the past and (3) they agree to be assigned to either the intervention or control group. After enrolment, eligible schools will be randomly assigned 1:1 to either the intervention or control group using the Microsoft Excel random number function, with randomisation stratified by subgroups: federal state, school size, social gradient and level of activity in school health promotion. Within the subgroups, each school will be assigned a random number, with the proviso that, for example, schools with large random numbers will be assigned to the intervention group and schools with small random numbers will be assigned to the control group. Randomisation and communication with schools will be carried out by the investigators. Schools will be informed individually by e-mail of the outcome of the group allocation. Data will be collected at baseline and again 18 months after the start of the intervention, including the perspectives of pupils and teachers (both baseline and post-test) as well as parents (post-test only). While the MindMatters intervention is implemented in the intervention schools according to defined criteria, no intervention is implemented in the control schools. The trial flow of the evaluation study is shown in Fig. 2. The trial has been approved by the Ethical Review Board of Fulda University of Applied Sciences (Nr. 3.1.9.2), the Ministries of Education in Rhineland-Palatinate and Schleswig–Holstein, and registered with the German Clinical Trials Register (DRKS00023762).

Fig. 2
figure 2

Trial flow chart

Full size image

Sample size calculation

In this study, schools represent the intervention clusters in which classes and pupils are nested. The power analysis relates to the individual pupil level, taking into account the cluster effect by including the intraclass correlation (ICC). There is no ICC estimate for mental disorders in the target age group, but it is estimated to be rather low compared to substance use or risky behaviour and is therefore set at p = 0.01 for the power analysis [22]. While the medium effects can be expected at the level of knowledge for primary school pupils (d = 0.5), small effects (d = 0.1 to 0.3) are assumed at the level of competence due to the short study period. Given the most recent findings on the prevalence, it is expected that the prevalence of mental health problems in the intervention group will be at most four percentage points lower than in the control group at the post-test.

Assuming an alpha level of 5% and a statistical power of 0.80, the estimated sample size is 2066 (with equal allocation to both groups, i.e. 1033 pupils in the intervention group and 1033 in the control group, https://www.sealedenvelope.com/power). Taking into account an intra-class correlation of 0.01 and a number of approximately 15 pupils per class with parental consent and actually present on the day of the survey, the required total sample size increases as follows: (1 + (15 − 1) × 0.01) × 2066 = 2355. Based on previous studies in primary school education, an attrition rate of about 10% is calculated for the pre-post survey at the pupil level across the different measurement points, so that a total sample of 2615 pupils from 174 classes is required for the baseline test. It is planned to implement this with a total of 40 participating primary schools (20 in the intervention group and 20 in the control group).

Recruitment and strategies for achieving adequate participant enrolment

As MindMatters is a complex intervention that addresses several levels, a period of 4 months is estimated for the schools to be recruited. Recruitment takes place in a total of three federal states (Rhineland-Palatinate, North Rhine-Westphalia and Schleswig–Holstein), with all primary schools being invited to participate. As a first step, all school principals will be invited to participate in the study by email. After a reminder by post, all schools will be contacted by telephone by the study team. Schools with a positive response will be contacted personally and will be informed about the process of the study. After agreeing to participate, the principal receives a short online questionnaire to assess the school’s general conditions (e.g. current implementation of school health promotion).

At the pupil level, data response is ensured by having trained academic staff collecting the data in the classroom. The success rate depends largely on the number of pupils with parental consent. To increase the response rate of parental consent forms, parents receive a detailed information letter. To increase the response rate among parents and teachers, the survey is offered both online and in paper–pencil form. The return of data is continuously monitored and reported back to schools at regular intervals. The feedback explicitly states that the significance of participating in the study is highly dependent on the level of data response. To encourage retention and completion of follow-up, schools are promised individual feedback on results at the end of the study. In addition, intervention and control schools will receive regular updates on the status of the study and motivational messages (e.g. greeting cards).

Measures

The intervention model includes a set of outcomes that unfold gradually in the short-, medium- and long-term levels and can be observed at the school and individual levels (mainly pupils and teachers, Additional file 3: MindMatters intervention model for primary schools) [23]. The primary outcome of this study includes several indicators of pupils’ mental health (e.g. emotional symptoms, conduct problems, prosocial behaviour). According to Nutbeam’s outcome model, these effects are preceded by direct effects at the individual level, such as knowledge or skills, but also at the classroom and organisational levels, such as classroom climate, classroom disruption, integration of health promotion into the school mission or organisational commitment. In line with the basic assumption of the Good Healthy School approach, it is also assumed that improvements in direct outcomes and mental health will have a positive long-term effect on pupils’ educational outcomes (e.g. academic performance, learning behaviour). Finally, the intervention model includes a number of factors that have been shown in previous studies to be relevant to intervention effectiveness. In addition to ‘traditional’ measures of implementation fidelity, such as adherence, dosage or acceptability [24], we included further measures such as health-promoting leadership or readiness for change [25]. Table 1 provides an overview of the measurements.

Table 1 Overview of study measurements
Full size table

Mental health outcomes

The change in mental and behavioural problems from baseline to post-assessment is measured using the German version of the Strengths and Difficulties Questionnaire (SDQ). It contains five subscales (hyperactivity, emotional symptoms, conduct problems, peer problems and prosocial scale) with five items for each dimension. In order to increase the comprehensibility and applicability for primary school children, the SDQ items have been revised, and each item has been visualised with a picture. The pupil survey is administered using an innovative tablet computer-based procedure and was pre-tested in June 2021 with 908 pupils in 10 primary schools and 43 classes in North Rhine-Westphalia.

Direct outcomes

Direct outcomes at the pupil level are emotion-related knowledge and social-emotional skills. Social-emotional skills were operationalised with a random selection of 14 items from the German FEESS questionnaire. The instrument consists of seven subscales, each assessed by two items (social integration, classroom climate, self-concept, attitude to school, willingness to make an effort, enjoyment of learning, feeling of acceptance). In addition, teachers’ diagnostic ability to identify conflicts and problems among pupils is measured by four items.

Classroom and organisational outcomes

At the classroom level, classroom disruption, e.g. due to disturbing noises, insufficient preparation by pupils or constant need for reminders, is operationalised with seven items. In addition, bullying and classroom climate (e.g. conflicts among pupils, good relationships in class) are measured with six and four items, respectively. At the organisational level, the extent to which health promotion is embedded in the school mission is assessed with eight items, while organisational commitment is operationalised with a five-item scale.

Educational outcomes

In terms of pupils’ educational outcomes, we assess teachers’ and parents’ reports of pupils’ academic achievement in mathematics, reading and spelling, as well as their learning behaviour, which could be rated on a 5-point scale (very good to poor).

Implementation

In order to assess the quality of implementation of the intervention, each MindMatters lesson or exercise delivered is documented. The documentation includes information on dose (i.e. number of exercises delivered), variation used, fidelity, deviations and preparedness. As a growing body of research suggests that school mental health interventions that expose the whole class to the same content may also have adverse effects [42, 43], teachers are asked to provide open feedback on each lesson. In this way, unintended events (e.g. harmful effects) or deviations are assessed in a non-systematic approach.

In addition to the activities implemented at the curriculum level, teachers from intervention schools are asked to document their school development activities to maintain and promote mental health as defined in the SchoolMatters intervention module (degree of engagement with the SchoolMatters materials, establishment of a school development team, conducting a needs assessment, creating a project plan and a plan for crisis situations, publicity, raising awareness of the intervention within and outside the school) and to assess satisfaction with support and other feedback.

Covariates, mediators and moderators

For both intervention and control schools, it will be recorded whether other mental health interventions were implemented during the study period. Furthermore, both groups will provide information on health promotion activities, school development, readiness to change and health-promoting leadership. Sociodemographic characteristics (sex, age), socioeconomic status and migration background of participants will also be recorded.

Data collection, management and statistical analysis

IFT Nord is responsible for the coordination and monitoring of the study. Two study centres (IFT-Nord, Fulda University of Applied Sciences) are responsible for the collection, processing and use of the data. In both study centres, all data will be collected using the same procedures, standards and elaborated instruments of proven psychometric quality. Three different sources of data are used: (1) online survey data from school principals, teachers and parents; (2) pupil survey data collected at classroom level via tablet computers; and (3) paper–pencil questionnaire data from principals, teachers and parents. All data will be stored on a secure IT infrastructure. Data from paper-based questionnaires are stored in a lockable cabinet and are only removed temporarily for data entry. A data safety concept is in place that has been reviewed and monitored by the Ministries of Education of the participating study regions. Data entry is carried out by trained scientific staff and is randomly checked by a second independent researcher. Once collected and entered, all data will be checked for plausibility, consistency and completeness. Data transfer between the two study centres is ensured via a secure connection (e.g. virtual private network (VPN)). The status of the trial will be documented in regular progress reports and by the Federal Ministry of Education and Research. To ensure quality, regular site visits will be organised at the study centres.

Statistical analyses will be performed using the latest version of STATA (Stata Corp., College Station, TX, USA). The main analysis will consist of a multilevel regression in which the primary outcomes are predicted at the pupil level using group status (intervention versus control group), controlled for baseline condition. Group status will be included in the model as a fixed effect and a random intercept will be provided for the cluster variables (study region, school, class). Furthermore, all variables with differences between the intervention and control group at baseline will be included in the model. Further mediation analyses will investigate which mediating mechanisms can be identified. It is assumed that the change in mental health outcomes is caused by improvements in proximal (i.e. direct and class/organisational) outcomes. This requires that the group status (i.e. the intervention) is correlated with the proximal outcomes and that the proximal outcomes are significantly related to the distal outcomes. Mediation effects will be examined using the Baron and Kenny partial correlation analysis approach and structural equation models [44, 45]. Moderator analyses will also examine which school-level factors (e.g. size, social gradient) are associated with or influence the impact of MindMatters. Moderator effects are tested using interaction terms in the regression model. The analysis will first be conducted according to the intention-to-treat principle and additionally compared with a per-protocol analysis. Analysis sample 1 includes all those for whom data are available at both measurement points (complete case analysis, as randomised). Analysis sample 2 includes all those reached at baseline (ITT as randomised). Analysis sample 3 includes all those with data available at both measurement points, and schools without intervention are included in the control group. Multiple imputation is used for missing values. To ensure the methodological quality of all analyses, statistical advice is provided by an independent institute.

Discussion

This study protocol documents the two-arm cluster randomised controlled trial with baseline and post-assessment for MindMatters primary schools, a holistic school-based intervention to promote mental health in school children. While the overall intervention targets children and adolescents in primary and secondary schools, this study focuses on the primary school module, for which there is no previous evidence. Reductions in mental and behavioural problems in children will be the primary outcomes, preceded by improvements in knowledge and social-emotional skills, as well as organisational improvements such as classroom climate, classroom disruption and organisational commitment. In addition, we expect these intervention effects to have a positive long-term impact on pupils’ educational outcomes.

While the effectiveness of highly standardised behavioural modification interventions is well documented, there is still an evidence gap for complex intervention programmes that take into account environmental and organisational determinants. This problem has also been labelled as ‘inverse evidence law’ [23], and the results of this study aim to contribute to strengthen the evidence base for holistic interventions on school (mental) health promotion. Its strengths can be seen in the study design (a two-arm cluster randomised controlled trial), which has a comparatively high internal validity. By interviewing younger children, the study focuses on a target group that has received little attention so far; to date, parents or teachers have mostly been interviewed, but not the children themselves [46]. Younger children’s own perspectives can provide valuable information for assessing mental health as part of a multi-informant approach [47]. In addition, innovative methods are used: pupils are surveyed using tablet computers that guide them through the questionnaire in the classroom in a structured and visually appealing way. The inclusion of pictorial representations of the questions may improve the reliability of self-reports in younger children [48]. It should also be emphasised that the study links several indicators of implementation fidelity with intervention effectiveness and will also help to shed more light on the relationship between health and educational attainment. Evidence from the Wales-wide School Health Research Network suggests that school health policies and curriculum practices are associated with educational attainment at age 14, particularly in disadvantaged schools [49]. However, as emphasised by Langford et al. [13] in their Cochrane review, the available evidence is scarce and does not allow conclusions to be drawn about the effectiveness of interventions on school health promotion. Evidence of such effects would also confirm the conceptual approach of the Good Healthy School [18]. As health is not a mandatory part of the school curriculum or cross-curricular activities in many countries, evidence would provide important arguments for establishing (mental) health in everyday school life. Finally, it should be noted that the results of the trial also aim at helping funding institutions and decision-makers in allocating resources.

The main limitation of this study is the assessment via self-reports. There is an increased risk of systematic bias, such as recall bias. Especially in younger children, challenges such as limited language and cognitive abilities or comprehension difficulties must be taken into account [50], although self-reports have previously been shown to be reliable and valid measures of mental well-being in adolescents. In addition, the power calculation was made a priori without information on the ICC estimate for mental disorders in children. Post hoc, the ICC for the SDQ could be determined from the pilot data and it was found that the ICCs ranged from p = 0.03 (hyperactivity) to p = 0.07 (behavioural problems with peers), indicating a severe underestimation of the ICCs for the SDQ. Furthermore, the SDQ might not be an adequate proxy for mental disorders, as it includes, for example, items for classroom disruption and prosocial behaviour. Although the trial might therefore not be suitably powered to detect small intervention effects, type I error inflation might be cushioned by the large number of clusters at the classroom level.

Also, as participation is voluntary, non-compliance could become an issue. Teachers are expected to teach the prepared modules and excercises independently of their teaching responsibilities and may find this an additional burden. These classes may discontinue participation in MindMatters early. Another limitation to the implementation of the intervention is the significant additional burden placed on schools by the Corona pandemic. As health promotion is not legally anchored in schools, this could lead to limited resources being invested in school health interventions [51]. As a result, a certain selection bias in school participation cannot be ruled out. Finally, this study is conducted in three German federal states, and conclusions cannot be directly transferred to other federal states due to the differences in the education systems.

Trial status

Protocol version 2, 6 October 2023. Recruitment of schools started in May 2021 and was completed in January 2022, with data collection continuing until September 2023. Due to the Corona pandemic, the data collection process had to be adapted several times.