Abstract
There is increasing interest about the fidelity with which interventions are implemented because it is theorized that better implementation fidelity by facilitators is associated with better participant outcomes. However, in the parenting program literature, there is mixed evidence on the relationship between implementation fidelity and outcomes. This paper provides a synthesis of the evidence on the relationship between facilitator delivery and outcomes in the parenting program literature. Following PRISMA guidelines, this paper synthesizes the results of a systematic review of studies on parenting programs aiming to reduce violence against children and child behavior problems. Specifically, it examines associations between observational measures of facilitator competent adherence and parent and child outcomes. A meta-analysis was not feasible due to study heterogeneity. As a result, Synthesis Without Meta-Analysis guidelines were followed. Searches in electronic databases, reference searching, forward citation tracking, and expert input identified 9653 articles. After screening using pre-specified criteria, 18 articles were included. The review found that most studies (n = 13) reported a statistically significant positive relationship with at least one parent or child outcome. However, eight studies reported inconsistent findings across outcomes, and four studies found no association with outcomes. The results suggest that better facilitator competent adherence is generally associated with positive parent and child outcomes. However, this finding is weakened by the methodological heterogeneity of included studies and due to the wide variety of ways in which studies conceptualized competent adherence-outcome relationships.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Background
In recent years, there has been increasing interest in the fidelity with which evidence-informed interventions are implemented as it is theorized that better implementation fidelity — the extent to which an intervention is implemented as intended (Dane & Schneider, 1998) — is associated with better participant outcomes. Systematic review and meta-analytic evidence from a variety of fields now empirically supports that improving implementation fidelity is an important mechanism through which to enhance participant outcomes (Carroll et al., 2007). In the field of health promotion, a well-cited systematic review of over 500 studies reported evidence of a relationship between higher implementation fidelity and improved participant health and well-being at the study level (Durlak & DuPre, 2008). In the field of educational interventions, several systematic reviews found a positive relationship between fidelity and outcomes. A review of randomized trials found that higher levels of teacher fidelity were associated with improved student achievement outcomes (Hill & Erickson, 2019). Further, a review of the implementation of 29 school-based physical activity programs found program delivery by more highly competent teachers consistently predicted better student outcomes (Naylor et al., 2015), and a meta-analysis of 221 school-based child behavior programs found that implementation fidelity was a key contributor to positive changes in student behavior (Wilson et al., 2003).
Substantial evidence demonstrates that parenting programs are an effective way to support parents to acquire the knowledge and skills to enhance their children’s health and well-being and thereby improve child outcomes (e.g., Barlow & Coren, 2018; Barlow et al., 2006; Chen & Chan, 2016; Furlong et al., 2013; Knerr et al., 2013). However, the role of implementation fidelity and its relationship to intended outcomes in these programs is unclear. Few studies report on parenting intervention fidelity (Gardner et al., 2016), and among these, there is limited evidence on the relationship between implementation fidelity and outcomes (Rojas-Andrade & Bahamondes, 2019). One exception is a recently published systematic review of 24 studies on programs aiming to reduce child externalizing behaviors (Leitão et al., 2020). It reported on the role of several facilitator factors, including facilitator adherence, on program outcomes (Leitão et al., 2020). While this review concluded that facilitator delivery mattered for outcomes, it only included interventions specifically addressing child behavior problems and did not summarize the results of each study in detail. Another example is a meta-analysis of 156 studies on nine home visiting programs aiming to reduce child maltreatment which found that several implementation fidelity components, including facilitator adherence, were positively related to reductions in child maltreatment (Casillas et al., 2016).
Among the growing number of primary studies on the relationship between fidelity and outcomes, the evidence is inconsistent with some studies finding an association and others finding no association (Breitenstein et al., 2010; Cantu et al., 2010; Durlak, 1998; Fixsen et al., 2005; Forgatch et al., 2005; Olofsson et al., 2016). Some studies have found the relationship between fidelity and outcomes to be curvilinear wherein the highest levels of fidelity were detrimental to participant outcomes (e.g., Hogue et al., 2008). The literature may be inconsistent due to a variety of challenges connected with studying the relationship between fidelity and outcomes, including publication bias; the potential influence of confounding variables (Breitenstein et al., 2010); interaction effects with other aspects of implementation (Berkel et al., 2011); inaccurate measurement due to the use of tools which have not had their reliability and validity established (Breitenstein et al., 2010); insufficient power to examine the relationships due to small sample sizes; and little variation in fidelity or intervention outcomes, making analyses lack sensitivity to discern associations.
Given the widespread dissemination of parenting programs, the relationship between fidelity and outcomes should be clarified. Knowledge about the role that fidelity plays in outcomes would lead to an enhanced understanding of whether implementation is one of the key mechanisms via which parenting programs achieve their positive results for children and families. Such an understanding could then be used to inform future program delivery, particularly as programs are translated into community settings and taken to scale (Fixsen et al., 2005; Mowbray et al., 2003). As implementation fidelity is potentially an important factor in enhancing parenting program outcomes, there is a need to take stock of the existing literature on the relationship between fidelity and outcomes.
This paper synthesizes the research on the relationship between implementation fidelity and outcomes found via a systematic review of the existing literature on parenting programs aiming to (a) reduce child maltreatment; harsh or dysfunctional parenting; and/or child conduct problems and/or (b) improve positive child behavior management strategies; parent–child bonding/attachment and relationships; and/or early childhood development outcomes. The review focused on two aspects of implementation fidelity articulated in Proctor et al.’s taxonomy (2011) — adherence (strictness with which a facilitator implements the prescribed content) and competence (skill and style with which a facilitator delivers program components). Although distinct, these aspects were selected for this review as it is commonly thought that they should be assessed simultaneously (Forgatch et al., 2005; Breitenstein et al., 2010; Martin et al., 2021). Together, competence and adherence (or “competent adherence”) refers to the quality and strictness with which facilitators deliver an intervention as intended (Carroll et al., 2007; Forgatch et al., 2005). This review included studies that reported on competence and/or adherence.
This study is the first synthesis of the evidence on the relationship between observational measures of facilitator competent adherence and parenting program outcomes and summarizes the methods used to study the relationship. It specifically focuses on observational assessments of facilitator competent adherence — completed on facilitator program delivery based on their live or video-taped delivery — as these are considered most rigorous and provide a more detailed account of program delivery (Dusenbury et al., 2003; Eames et al., 2008). Further, the study provides critical insight into whether better fidelity is associated with improved family outcomes.
Methods
This study builds on a systematic review conducted by Martin et al. (2021) that compiled the observational and non-observational measures of facilitator competent adherence found in the parenting program literature and synthesized the psychometric properties of the measures found. Using articles reporting on observational measures of facilitator competent adherence from the review, this paper synthesizes the evidence on the association, if any, between facilitator competent adherence and parent and/or child outcomes.
Systematic Review
The full details of the methods used for the systematic review are described by Martin et al. (2021). In sum, a tested search strategy was implemented in 12 electronic bibliographic databases (see Online Resource 1). To find additional studies, the database searches were supplemented by reviewing articles included in a systematic review of parenting programs in low- and middle-income countries to ensure representation from these contexts (Gardner et al., forthcoming); conducting backward citation tracking; conducting forward citation tracking using Google Scholar; and seeking input from parenting program experts. The inclusion and exclusion criteria applied are summarized in Table 1. The review tested inter-rater reliability between two coders at the title/abstract, full-text, and data extraction stages. Percentage agreements ranged from 92.8 to 94.4% and were thus sufficiently high.
The systematic review by Martin et al. found 9653 articles as of August 2021 (see Fig. 1). To be included in the review, articles must have been written in English; reported on observational measures of facilitator competent adherence; and used an observational, quasi-experimental, or experimental approach to analyze the association between facilitator competent adherence and family outcomes. Of the original articles, 18 articles reported on the relationship between observational measures of facilitator competent adherence and parent and child outcomes and were thereby included in the review.
Synthesis Without Meta-Analysis
Upon reviewing the 18 included studies, a meta-analysis was deemed methodologically unfeasible. As a result, a synthesis without meta-analysis was performed based on the Synthesis Without Meta-Analysis (SWiM) guidelines outlined by Campbell et al. (2020) (see Online Resource 2). These guidelines specify nine key categories of information that should be provided when a quantitative synthesis is not possible. The revised Meta-Analysis Reporting Standards was followed where possible (Appelbaum et al., 2018).
Reporting
Findings are reported following the PRISMA guidelines (Liberati et al., 2009) (see Online Resource 3). For studies which differentiated competent adherence-outcome relationships by subscale (e.g., adherence subscale score, competent subscale score, overall competent adherence score), overall competent adherence scores were extracted.
Results
The 18 included studies are disparate in terms of the programs studied, parent and child outcomes considered, and methods used. This heterogeneity meant that a meta-analysis could not be conducted. Further, the small number of studies limited the feasibility of meta-analyzing subgroups. The results of the 18 studies are summarized in Table 2 and Online Resource 4. Using a modified version of the Thomson and Thomas (2013) effect direction plot visual display system, results are visualized using: a sideways arrow (⇔) indicating no statistically significant association between competent adherence and parent/child outcomes; an upwards arrow (⇑) indicating a positive, statistically significant association between stronger competent adherence and better parent/child outcomes; or a downwards arrow (⇓) indicating a negative, statistically significant association between stronger competent adherence and poorer parent/child outcomes.
Programs and Outcomes
The 18 studies examined 11 different parenting programs delivered in high-income countries in Europe and the USA: Strengthening Families Program (n = 1; S1), Multi-Dimensional Family Therapy (n = 1; S8), Basic Parent Training Program (n = 1; S10), Brief Strategic Family Therapy (n = 1; S11), Early Head Start (n = 1; S12), Functional Family Therapy (n = 1; S13), Parent Child Interaction Therapy (n = 1; S16), Familias Unidas (n = 1; S17), Family Check-Up (n = 2; S2, S15), Incredible Years (n = 3; S3, S6; S14), and Parent Management Training Oregon Model (n = 5; S4, S5, S7, S9, S18). The studies reported an average of 38 facilitators and 159 families. The average level of fidelity reported in the studies was 69.46%. According to the Institute of Medicine (2009) classification system, two programs were universal prevention (S1, S9), two were indicated prevention (S5, S7), three were selected prevention (S4, S12, S17), 10 were treatments (S2, S3, S6, S8, S10, S11, S13, S14, S15, S18), and one was a combination of universal and treatment approaches (S16). Ten of the 18 studies included programs that targeted caregivers only, with the remaining eight targeting both caregivers and children (S1, S2, S8, S11, S13, S15–S17).
Studies assessed the association of competent adherence with five outcomes — child development (n = 1; S12), parenting stress (n = 2; S9, S18), family functioning (n = 2; S11, S17), parenting behaviors and skills (n = 9; S1, S3–S6, S9, S12, S15, S16), and child behavior (n = 13; S2, S6–S11, S13–S18). These outcomes were measured using numerous scales. For instance, in the 10 studies reporting on parenting behaviors and skills, the outcomes were measured using 10 different scales — the Intervention Targeted Parenting Attitude and Behavior Scale, the Dyadic Parent–Child Interaction Coding System, the Family and Peer Process Code, the Parenting Sense of Competence Scale, Parenting Behavior Questionnaire, the Home Observation Measure of the Environment, Relationships Process Code, Coders Impression Inventory, Alabama Parenting Questionnaire, and Caregiver Wish List. In the 13 studies reporting on child behavior, the outcomes were measured using eight different scales — the Child Behavior Checklist (which was used most often; CBCL), Eyberg Child Behavior Inventory (ECBI), Swanson Nolan and Pelham-IV Questionnaire (SNAP-IV), Strengths and Difficulties Questionnaire, Parent Account of Child Symptoms Interview, Parent Daily Report, and two different researcher-created self-reported drug use measures. The variability in the scales used to measure parent and child outcomes contributed to the methodological heterogeneity of the studies.
Designs and Analysis Methods
The studies employed a wide range of methods. All studies examined facilitator delivery using data from the intervention arms of randomized trials. As expected, fidelity to outcome associations were observed as they occurred rather than experimentally manipulated. Associations between competent adherence and outcomes were analyzed using correlations (n = 1; S18), regression and one-way ANOVA (n = 1; S3), SEM/path analysis and correlation (n = 2; S5, S15), latent growth curve modeling (n = 3; S2, S8, S11), SEM/path analysis (n = 3; S7, S17, S4), and regression (n = 8; S1, S6, S9, S10, S12, S13, S14, S16). Competent adherence variables were modeled as categorical (n = 1, three categories: “no exposure,” “low exposure,” and “high exposure”; S3) or continuous (n = 17; all but S3) (Durlak & DuPre, 2008). Five studies conducted associations between fidelity captured at more than one timepoint, yet not all results were reported (S7, S8, S9, S16, S17).
Control Variables
Of the 18 studies, 17 reported controlling for potential confounding variables to estimate the relationship between competent adherence and outcomes (all but S18). The controls varied considerably with only two studies conducted by the same researcher reporting the same combination of variables. The control variables included therapeutic alliance, program site, baseline participant outcomes, facilitator characteristics, organizational characteristics, and participant characteristics (e.g., child age, child gender). For one study, control variables were only partially used as facilitator, organizational, family, and attendance variables were included in other parts of the structural equation model (S17). The studies did not discuss the rationale for selection of control variables.
Clustering and Multiple Comparison
Six of the 18 studies reported that they accounted for clustering of observations due to the nested design of delivering group-based parenting programs, often by facilitators working in pairs (S1, S4, S6, S8, S11, S14). Two of the five studies accounted for the same combination of clustering variables. The clustering variables accounted for included program level (n = 1; S1), within-couple dependence (n = 1; S4), family level (n = 1; S11), parenting group (n = 1; S6), and unspecified (n = 2, S8, S14). In one of the 18 studies, it was unclear whether clustering was used because multi-level modeling accounted for repeated measures and multiple respondents per family, but not for multiple families per facilitator (S9). Further, when associations with several outcomes are investigated, it is best practice to account for multiple comparisons. None of the studies reported accounting for multiple comparisons.
Associations of Competent Adherence with Outcomes
Most studies found that facilitator competent adherence was positively associated with one or more parent and/or child outcomes. Of the 18 studies, six found statistically significant positive associations (⇑) between competent adherence and all parent and child outcomes examined (S2–S4, S11, S12, S14). A further eight studies found mixed evidence (S5, S7–10, S13, S17, S18) wherein at least one outcome was positively associated with competent adherence and one outcome was not (⇔). Of these eight, all found that while some outcomes had a statistically significant positive association with competent adherence others had no significant association. None of the studies had a negative association with competent adherence. Of the 18 studies, four found no significant association between facilitator competent adherence and any of the outcomes studied (S1, S6, S15, S16). In conducting these analyses, most studies reported on the average level of fidelity achieved (all but S3 and S5).
Parenting Behaviors and Skills
Of the nine studies examining competent adherence and parenting behaviors and skills, three found a positive association (S3, S4, S12), two found mixed associations based on the two types of analyses performed (S5) or in the types of parenting behaviors measured (S9), and four found no associations (S1, S6, S15, S16).
Parenting Stress and Family Functioning
Of the two studies examining the relationship between competent adherence and parenting stress, one found a positive association (S9), and the other found mixed associations (S18). Similar findings were observed in the two studies examining competent adherence and family functioning wherein one found a positive association (S11) and one found no association (S17).
Child Behavior and Development
Of the 13 studies examining the relationship between competent adherence and child behavior outcomes, three found a positive association (S2, S11, S14); five found a mix of positive and no association based on who reported outcomes (e.g., parent- versus teacher-report) or based on varying dimensions of competent adherence examined (S8, S9, S10, S13, S17); and five found no associations (S6, S7, S15, S16, S18). Finally, in the one study examining child developmental outcomes, a positive association was observed between competent adherence and child academic attainment (S12).
Dimensions of Competent Adherence
Two studies examined competence and adherence separately and found differences in their association with outcomes (S8, S17). Utilizing linear models, Hogue et al. (2008; S8) found that adherence was related to greater reductions in child externalizing behavior while competence was not. Furthermore, neither adherence nor competence were associated with child internalizing behavior. In exploring curvilinear relationships, this study found some evidence of a curvilinear relationship between adherence and internalizing behavior where medium levels of adherence were positively associated with outcomes. However, the study did not report the results of an analysis of curvilinear relationships between adherence and externalizing behavior issues and did not report on competence and either internalizing or externalizing behavior issues. St. George et al. (2016; S17) found that competence was related to decreased substance use, but not related to improvements in family functioning and that adherence was not related to either reduced substance use or improved family functioning.
Four studies examined whether specific dimensions of competent adherence were associated with outcomes (S5, S10, S11, S18). For example, in one of two models tested, Rendu (2004; S10) found that one dimension of competent adherence — a group facilitation technique — was related to reductions in child behavior issues but another facilitation technique was not so related in either model tested. As another example, Robbins et al. (2011; S11) found that a facilitation approach called “joining” was associated with improved family functioning and reduced adolescent drug use, but three other facilitator approaches were not.
Several studies found differences in relationships between facilitator competent adherence and participant outcomes when reported using different measures (S7, S8, S13). To illustrate, in the paper by Hukkelberg and Ogden (2013; S7), competent adherence was associated with reductions in child behavior problems based on parent-reports, but not based on teacher-reports. As another example, the paper by Satterfield (2013; S13), competent adherence was associated with reductions in child behavior issues based on parent-reports, but no association was found based on youth-reports.
Discussion of Clinical Implications
Overall Findings
The synthesis considered 18 studies reporting on the relationship between observational measures of facilitator competent adherence and parent/child outcomes, with most interventions having a treatment focus. Studies focused variously on selective or indicated prevention, with most evaluating treatment programs. Treatment studies nevertheless have considerable implications for prevention, as treatment for child behavior problems is intended also to serve as prevention of their long-term adverse outcomes, including offending and poor mental health, education, and employment prospects. Of the 18 included studies, studies reported on child behavior, nine on parenting skills and behaviors, two on parenting stress, two on family functioning, and one on child development. The studies were highly heterogeneous in their design and analysis methods. Five studies conducted analyses on associations between fidelity captured at more than one timepoint.
Most studies found that facilitator delivery is associated with at least one parent or child outcome — eight of the 13 studies examining child behavior, five of the nine studies examining parenting skills and behaviors, both studies examining parenting stress, one of the two studies examining family functioning, and the one study examining child development. These findings generally suggest better competent adherence is associated with better parent and child outcomes. There was no discernible difference in associations between competent adherence and outcomes based on the aspects of competent adherence measured. Still, there was a sizeable number of studies with mixed findings where some outcomes are associated with competent adherence and others are not. Lack of detected associations between competent adherence and outcomes has several potential explanations, such as that fidelity does not matter for outcomes (low fidelity has no negative impact), that our efforts regarding fidelity are not worthwhile (high fidelity has no positive impact), issues due to poor measurement, and/or lack of statistical power to detect associations due to small sample sizes. Fidelity may also be indirectly associated with outcomes. Indirect associations were explored in several studies — such as Smith et al. (2013; S15) who found that although fidelity was not directly associated with parenting or child behavior, it was indirectly associated with some outcomes. Finally, other implementation variables, such as participant responsiveness or engagement during program sessions, participant attendance, and facilitator-participant working alliance, may mediate or moderate the relationship between fidelity and outcomes (Berkel et al., 2011; Carroll et al., 2007). For example, if the effect of fidelity on child outcomes is mediated by participant engagement, statistical models that adjust for this variable may reduce the observed relationship between fidelity and child outcomes. Further research on competent adherence-outcome relationships would benefit from a more systematic theoretical understanding of the variable relationships.
Conceptualizing the Role of Facilitator Competent Adherence
The finding that better facilitator competent adherence is generally associated with better outcomes is limited by the diverse conceptualizations of the relationship between competent adherence and outcomes in the studies. This diversity is exemplified by the range of controls used in the models tested, including facilitator (e.g., therapeutic alliance), organizational (e.g., amount of coaching support provided to facilitators), and participant characteristics (e.g., child age, baseline outcomes), with only two studies using the same combination of controls. These differences reveal considerable variation in how researchers theorize about the potential mechanisms impacting, and dissensus on how they hypothesize, the relationship between facilitator competent adherence and outcomes. As few papers articulated a clear rationale or delineated a conceptual framework for their choice of controls, such as through directed acyclic graphs (DAGs) (Pearl et al., 2016), research in the field may be at risk of including unnecessary controls and overcontrol bias (incorporating inappropriate variables leading to spurious results or underpowered models) (Achen, 2005; Rohrer, 2018). As a result, future research would benefit from greater consideration and documentation of the theory underpinning the research — the mechanism(s) linking facilitator, organizational, and participant characteristics with competent adherence and its association with outcomes.
Methodological Issues and Study Quality
There are limitations in the studies reviewed. In particular, issues concerning the robustness of the analyses, reliability and validity, and quality of reporting will be discussed using ROBINS-I (Risk of Bias In Non-randomized Studies of Interventions) (Sterne et al., 2016).
Robustness of Analyses
The sample size in the studies was generally small, with data collected from an average of 38 facilitators and 159 families. Further, none of the studies performed power calculations to determine the number of observations necessary to examine the relationship between competent adherence and outcomes or accounted for multiple comparisons. Studies differed considerably as to whether and which type of clustering was accounted for in the analyses. Clustering may occur at the parent group level (parents often receive an intervention in a group format) and the facilitator level (programs are often delivered by more than one facilitator and facilitators typically deliver a program to more than one parent) leading to non-independent observations. However, only six studies accounted for clustering. A wide range of statistical approaches (such as latent growth curve modeling, regression, and SEM/path analysis) were used, demonstrating variation in researcher thinking about how an analysis of competent adherence-outcome relationships should be examined. This variation was one factor which prevented a meta-analysis. If future studies report bivariate correlations and unstandardized regression coefficients, these studies will produce results that are easier to standardize and incorporate in meta-analyses.
Reliability and Validity
Little is known about the accuracy and reliability of observational measures of competent adherence meaning conclusions drawn from this synthesis should be made with caution (Breitenstein et al., 2010). Eleven of the 18 studies synthesized herein provide some information on the reliability and/or validity of the measures of competent adherence used — with ten of these reporting on inter-rater reliability, nine reporting on internal consistency, and five reporting on construct validity (Martin et al., 2021) (see Online Resource 4).
Reporting
In future research, study reporting could be more detailed. Five of 18 papers did not provide information on facilitator sample size. Additionally, several studies made claims about competent adherence-outcome relationships yet did not provide the numerical results. Other studies did not provide information about key outcomes mentioned in their methods.
Strengths and Limitations
Although this paper makes an important contribution to implementation science as it relates to parenting programs, it has limitations. This review did not include studies reporting on non-observational measures of facilitator competent adherence (e.g., self-report measures). A synthesis of such measures would be worthwhile to conduct when greater study homogeneity permits meta-analyses. In addition, this review focused on parenting programs aiming to (a) reduce child maltreatment; harsh or dysfunctional parenting; and/or child conduct problems and/or (b) improve positive child behavior management strategies; parent–child bonding/attachment and relationships; and/or early childhood development outcomes. However, as studies reported on associations between facilitator competent adherence and several secondary outcomes (e.g., parenting stress), we synthesized information on these analyses as well. Thus, the associations reported herein between facilitator competent adherence and these secondary outcomes are likely not inclusive of all literature reporting on these outcomes. Further, this review was unable to explore selective reporting bias or publication bias. Aggregate associations and publication bias could not be explored as the substantial methodological heterogeneity prevented a meta-analysis from being conducted. While there are limitations, the review is the first to synthesize the data from studies examining the relationship between observationally measured facilitator competent adherence and parenting program participant outcomes to clarify the mixed evidence found in the literature.
Suggestions for Future Research
Future studies, and the parenting intervention field at large, would benefit from investigations of competent adherence-outcome relationships that: clearly conceptualize the mechanisms hypothesized to influence competent adherence-outcome relationships; utilize larger sample sizes; account for clustering variables at the parent- and facilitator-level; incorporate carefully chosen control variables; follow best practices in open science, including pre-registration to minimize the risk of selective reporting; and report bivariate correlations and unstandardized regressions. As a result, this literature may benefit from reporting guidelines.
The field would also benefit from an examination of understudied and novel aspects of the relationship between competent adherence and family outcomes. For instance, there was some evidence from one study that the relationship between fidelity and outcomes was not linear and at high levels fidelity could be associated with worse program outcomes (Hogue et al., 2008). Exploration of curvilinear relationships would provide information on whether there is a tipping point at which further attention to fidelity is unnecessary or even unhelpful. Other topics to examine in future studies analyzing competent adherence-outcome relationships include examining the test–retest reliability of measures to see how competent adherence fluctuates over time as this review only identified two studies that examined associations between competent adherence and outcomes at more than one timepoint; determining how to weigh fidelity with adaptation; exploring whether competent adherence-outcome relationships are significant in the long-term; and testing whether fidelity is indirectly associated with parent/child outcomes or whether other implementation variables such as engagement or working alliance mediate the relationship between fidelity and outcomes. Further, it may be valuable to study competent adherence in different contexts, since all of studies found were nested within randomized trials, which may not always be fully representative of routine delivery contexts.
Conclusion
This review aimed to provide clarity on the evidence regarding the role facilitator competent adherence plays in achieving parent and child outcomes. While this paper finds that the evidence is inconsistent, the synthesis also finds a general trend indicating that higher levels of facilitator competent adherence are related to improved parent and child outcomes. The latter finding is limited by a high number of the studies having found mixed evidence and no associations with outcomes as well as the diverse methodological approaches employed by the studies. The review highlights the need for further research on whether there is an association between facilitator competent adherence and outcomes and recommends how researchers and practitioners can advance the field.
Availability of Data and Materials
In line with TOP Guidelines, the plans for the systematic review and meta-analysis were pre-registered on PROSPERO (registration number: CRD42020167872). The paper analyzes a subset of the data used in a paper published in Clinical Child and Family Psychology Review (details provided in Online Resource 1). Two reporting guidelines were followed: Synthesis Without Meta-Analysis (Online Resource 2) and PRISMA (Online Resource 3). All study data used in the analyses have been made available at the Open Science Framework: https://osf.io/md2p7/?view_only=02b3b74f7ecb48668ee44b08e3c96007 and Online Resource 4.
References
Achen, C. H. (2005). Let’s put garbage-can regressions and garbage-can probits where they belong. Conflict Management and Peace Science, 22(4), 327–339.
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 3.
Barlow, J., & Coren, E. (2018). The effectiveness of parenting programs: A review of Campbell reviews. Research on Social Work Practice, 28(1), 99–102.
Barlow, J., Johnston, I., Kendrick, D., Polnay, L., & Stewart‐Brown, S. (2006). Individual and group‐based parenting programmes for the treatment of physical child abuse and neglect. Cochrane Database of Systematic Reviews (3).
Berkel, C., Maruicio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12(1), 23–33.
Breitenstein, S. M., Gross, D., Garvey, C. A., Hill, C., Fogg, L., & Resnick, B. (2010). Implementation fidelity in community-based interventions. Research in Nursing & Health, 33(2), 164–173.
Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., & Thomas, J. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ, 368.
Cantu, A. M., Hill, L. G., & Becker, L. G. (2010). Implementation quality of a family-focused preventive intervention in a community-based dissemination. Journal of Children’s Services, 5(4), 18.
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40.
Casillas, K. L., Fauchier, A., Derkash, B. T., & Garrido, E. F. (2016). Implementation of evidence-based home visiting programs aimed at reducing child maltreatment: A meta-analytic review. Child Abuse & Neglect, 53, 64–80.
Chen, M., & Chan, K. (2016). Effects of parenting programs on child maltreatment prevention: A meta-analysis. Trauma, Violence, & Abuse, 17(1), 88–104.
Chiapa, A., Smith, J. D., Kim, H., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2015). The trajectory of fidelity in a multiyear trial of the family check-up predicts change in child problem behavior. Journal of Consulting and Clinical Psychology, 83(5), 1006.
Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45.
Durlak, J. A. (1998). Why program implementation is important. Journal of Prevention & Intervention in the Community, 17(2), 5–18.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350.
Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237–256.
Eames, C., Daley, D., Hutchings, J., Hughes, J., Jones, K., Martin, P., & Bywater, T. (2008). The leader observation tool: A process skills treatment fidelity measure for the Incredible Years parenting programme. Child: Care, Health and Development, 34(3), 391–400.
Eames, C., Daley, D., Hutchings, J., Whitaker, C. J., Bywater, T., Jones, K., & Hughes, J. C. (2010). The impact of group leaders’ behavior on parents acquisition of key parenting skills during parent training. Behavior Research and Therapy, 48(12), 1221–1226.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., Wallace, F., Burns, B., Carter, W., Paulson, R., Schoenwald, S., & Barwick, M. (2005). Implementation research: A synthesis of the literature. https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-MonographFull-01-2005.pdf
Forgatch, M. S., & DeGarmo, D. S. (2011). Sustaining fidelity following the nationwide PMTO™ implementation in Norway. Prevention Science, 12(3), 235–246.
Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy, 36(1), 3–13.
Furlong, M., McGilloway, S., Bywater, T., Hutchings, J., Smith, S. M., & Donnelly, M. (2013). Cochrane review: Behavioral and cognitive-behavioral group-based parenting programmes for early-onset conduct problems in children aged 3 to 12 years. Evidence-Based Child Health: A Cochrane Review Journal, 8(2), 318–692.
Gardner, F., Montgomery, P., & Knerr, W. (2016). Transporting evidence-based parenting programs for child problem behavior (age 3–10) between countries: Systematic review and meta-analysis. Journal of Clinical Child & Adolescent Psychology, 45(6), 749–762.
Giannotta, F., Özdemir, M., & Stattin, H. (2019). The implementation integrity of parenting programs: Which aspects are most important? Child & Youth Care Forum, 48(6), 917–933.
Hill, H. C., & Erickson, A. (2019). Using implementation fidelity to aid in interpreting program impacts: A brief review. Educational Researcher, 48(9), 590–598.
Hogue, A., Henderson, C., Dauber, S., Barajas, P., Fried, A., & Liddle, H. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76(4), 544.
Hukkelberg, S. S., & Ogden, T. (2013). Working alliance and treatment fidelity as predictors of externalizing problem behaviors in parent management training. Journal of Consulting and Clinical Psychology, 81(6), 1010.
Institute of Medicine. (2009). Defining the scope of prevention. In: Preventing mental, emotional, and behavioral disorders among young people. Washington (DC): National Academics Press (US).
Knerr, W., Gardner, F., & Cluver, L. (2013). Improving positive parenting skills and reducing harsh and abusive parenting in low-and middle-income countries: A systematic review. Prevention Science, 14(4), 352–363.
Leitão, S. M., Seabra-Santos, M. J., & Gaspar, M. F. (2020). Therapist factors matter: A systematic review of parent interventions directed at children’s behavior problems. Family Process, 60(1), 84–101.
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: Explanation and elaboration. BMJ, 339, b2700.
Maaskant, A. M., van Rooij, F. B., Overbeek, G. J., Oort, F. J., & Hermanns, J. M. (2016). Parent training in foster families with children with behavior problems: Follow-up results from a randomized controlled trial. Children and Youth Services Review, 70, 84–94.
Martin, M., Steele, B., Lachman, J. M., & Gardner, F. (2021). Measures of facilitator competent adherence used in parenting programs and their psychometric properties: A systematic review. Clinical Child and Family Psychology Review, 24(4), 834–853.
Mowbray, C., Holger, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. The American Journal of Evaluation, 24(3), 325–340.
Naylor, P.-J., Nettlefold, L., Race, D., Hoy, C., Ashe, M. C., Wharf Higgins, J., & McKay, H. A. (2015). Implementation of school based physical activity interventions: A systematic review. Preventive Medicine, 72, 95–115.
Olofsson, V., Skoog, T., & Tillfors, M. (2016). Implementing group based parenting programs: A narrative review. Children and Youth Services Review, 69, 67–81.
Pearl, J., Glymour, M., & Jewell, N. (2016). Causal inference in statistics: A primer. John Wiley & Sons.
Proctor, E., Silmere, H., Raghaven, R., Hovmand, P., Aarons, G., Bunger, A., Griffery, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 32(2), 65–76.
Rendu, A. (2004). Treatment adherence in a behaviorally based parenting program. University of London.
Robbins, M. S., Feaster, D. J., Horigian, V. E., Puccinelli, M. J., Henderson, C., & Szapocznik, J. (2011). Therapist adherence in brief strategic family therapy for adolescent drug abusers. Journal of Consulting and Clinical Psychology, 79(1), 43.
Roggman, L. A., Cook, G. A., Innocenti, M. S., Jump Norman, V., Boyce, L. K., Christiansen, K., & Peterson, C. A. (2016). Home visit quality variations in two early head start programs in relation to parenting and child vocabulary outcomes. Infant Mental Health Journal, 37(3), 193–207.
Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1(1), 27–42.
Rojas-Andrade, R., & Bahamondes, L. L. (2019). Is implementation fidelity important? A systematic review on school-based mental health programs. Contemporary School Psychology, 23(4), 339–350.
Satterfield, L. W. (2013). The effectiveness of functional family therapy in an Irish context: An examination of international implementation. Doctoral dissertation, Indiana University.
Scott, S., Carby, A., & Rendu, A. (2008). Impact of therapists’ skill on effectiveness of parenting groups for child antisocial behavior. Institute of Psychiatry, Kings College London.
Smith, J. D., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2013). Indirect effects of fidelity to the family check-up on changes in parenting and early childhood problem behaviors. Journal of Consulting and Clinical Psychology, 81(6), 962.
Snider, M. D. (2019). Examining the impact of treatment fidelity on client outcomes in a statewide implementation of parent-child interaction therapy. Western Virginia University.
St. George, S. M., Huang, S., Vidot, D. C., Smith, J. D., Brown, C. H., & Prado, G. (2016). Factors associated with the implementation of the Familias Unidas intervention in a type 3 translational trial. Translational Behavioral Medicine, 6(1), 105–114.
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., & Boutron, I. (2016). ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 355.
Thijssen, J., Albrecht, G., Muris, P., & de Ruiter, C. (2017). Treatment fidelity during therapist initial training is related to subsequent effectiveness of parent management training—Oregon model. Journal of Child and Family Studies, 26(7), 1991–1999.
Thomson, H. J., & Thomas, S. (2013). The effect direction plot: Visual display of non-standardised effects across multiple outcome domains. Research Synthesis Methods, 4(1), 95–101.
Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71(1), 136.
Funding
MM and BS are supported with doctoral funding from the Social Sciences and Humanities Research Council of Canada. MM is also supported by doctoral funding from the Rhodes Trust. YS is supported by the Centre for Development, Evaluation, Complexity and Implementation in Public Health Improvement (DECIPHer) and the Wolfson Centre for Young People’s Mental Health. DECIPHer is funded by the Welsh Government through Health and Care Research Wales. The Wolfson Centre for Young People’s Mental Health has been established with support from the Wolfson Foundation.
Author information
Authors and Affiliations
Contributions
MM, BS, TS, and YS worked on designing the paper. MM led the drafting of the manuscript. BS, TS, YS, JML, and FG contributed sections and revisions to the manuscript. MM did the data extraction and analysis. BS did the double coding. All authors read and approved the submitted manuscript.
Corresponding author
Ethics declarations
Ethics Approval
Not applicable.
Consent for Publication
Not applicable.
Conflict of Interest
TS has been a paid consultant for Catholic Relief Services (CRS), done contracted training for the Department of Work and Pensions, and has received honoraria for teaching statistical programming in R as well as being an Associate Editor for the Journal of Child and Mental Health. MM, YS, FG, and JML have been involved in parenting program evaluations. JML is the CEO of Parenting for Lifelong Health, a charity based in the UK. FG is an associate editor of Prevention Science; however, the peer review of this manuscript was managed by another editor.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Martin, M., Steele, B., Spreckelsen, T.F. et al. The Association Between Facilitator Competent Adherence and Outcomes in Parenting Programs: a Systematic Review and SWiM Analysis. Prev Sci 24, 1314–1326 (2023). https://doi.org/10.1007/s11121-023-01515-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-023-01515-3