Abstract
This special section presents the psychometric properties of fidelity scales used in a national mental health services project in Norway to improve the quality of care of people with psychoses. Across Norway, 39 clinical units in six health trusts participated. The project provided education, implementation support and fidelity assessments. The papers in the section address the psychometrics of fidelity measurement for the specific evidence-based practices: illness management and recovery, family psychoeducation, physical healthcare and antipsychotic medication management. Another paper analyzes the psychometrics of a scale measuring individualization and quality improvement that may be used in conjunction with fidelity scales for specific evidence-based practices. The first paper in the section presents the development and field of fidelity scales, and the two final papers with comments add some additional perspectives and discuss fidelity scales in a wider context. The psychometrics of the five scales were good to excellent. Fidelity assessment is a necessary and effective strategy for quality improvement.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Background for the Special Section on Fidelity Scales
Evidence-based practices and clinical guidelines can improve the quality of treatment in health care, including for people with psychoses. But accurate implementation of evidence-based practices and clinical guidelines in routine healthcare remains problematic. To achieve optimal outcomes, the field needs more effective strategies to assure faithful implementations.
Fidelity scales are tools to assess the quality of implementation in a program or clinical unit and guide improvements. “The rationale for using fidelity scales to guide practice is based on the working hypothesis that programs successfully replicating the core principles and procedures of the program models rigorously evaluated in controlled studies will achieve similar outcomes as these earlier studies” (Bond et al. 2009).
Widespread use of fidelity scales requires reliability and validity. The National Evidence-Based Practices (EBP) Project in the U.S. demonstrated a standardized method for development and format of fidelity scales (Bond et al. 2009) and methods for testing the properties for such scales (McHugo et al. 2007). Nonetheless, few studies of the psychometric properties of currently available fidelity scales exist. The papers in this special section address that deficit.
The Context for Our Testing of Properties of Fidelity Scales
The testing of properties of fidelity scales in this section occurred as a part of a cluster randomized controlled trial on the implementation of four evidence-based practices for treatment of patients with psychoses in mental health services (ClinicalTrials NCT03271242). Thirty-nine sites (community mental health centers/inpatient departments) in six health trusts throughout Norway participated. The four practices—illness management and recovery, family psychoeducation, physical health care, antipsychotic medication management—are among the core evidence-based practices for treatment of people with psychosis in Norway.
Each site chose two of the four practices for implementation and agreed to receive implementation support for one of the two, determined by random assignment. The site received the following support for the practice randomized to implementation support: A workshop with experts at the start, a toolkit for implementing the practice, a visiting implementation trainer biweekly for 6 months and then monthly for 12 months, and telephone supervision for the two manualized psychosocial programs. Sites randomized to the control condition only received a written description of the practice. Two trained experts assessed fidelity of both practices at each site at baseline and after 6, 12, and 18 months using independent and consensus ratings. The papers in this special section report on psychometric properties of the fidelity scales for all practices at sites that received implementation support.
Papers in the Special Section
The overview by Bond and Drake (2019) covers the history and current status of fidelity scales for evidence-based practices in mental health. It describes the use of fidelity scales, procedures for development and validation of fidelity scales, and benefits and challenges in using fidelity scales in research and in improvement of clinical programs and practice.
The following papers address the psychometric properties for each of the four fidelity scales and for the General Organizational Index (GOI). The scales for illness management and recovery and family psychoeducation are existing scales for well known, manualized psychosocial programs. The scales for physical health care and antipsychotic medication management are new scales measuring recommended practice components. The GOI is an established scale for measuring the overall organization of services. All five scales followed the guidelines for fidelity scale development described by Bond and Drake (2019).
Egeland et al. (2019) report on the Fidelity Scale for Illness Management and Recovery (IMR), a scale developed in the National EBP Project (McHugo et al. 2007) and subsequently used in several studies. The paper by Egeland et al. is the first comprehensive assessment of the psychometric properties of the scale. Joa et al. (2020) report on the Family Psychoeducation Fidelity Scale, also developed in the National EBP Project, and this paper is the first to report its psychometric properties.
Ruud et al. (2020a) report on the content and psychometric properties of the Physical Health Care Fidelity Scale for measuring a best practice approach to physical health care for people with psychosis. This is the first published fidelity scale to assess physical health care for people with psychosis. In a separate paper, Ruud et al. (2020b) report on the content and psychometric properties of the Antipsychotic Medication Management Fidelity Scale for measuring evidence-based recommendations for antipsychotic medication management.
Heiervang et al. (2020) report on the psychometric properties of the General Organizational Index (GOI), also developed in the National EBP Project. The GOI measures aspects of implementation common to all evidence-based practices but usually not included in fidelity scales for specific practices.
In his comments van Weeghel (2020) discusses some of the challenges and limitations in the fidelity scales reported in this section, overcoming resistance to fidelity measures, fidelity assessments in mental healthcare systems, and the importance of predictive validity. He also gives an international perspective by describing some developments in Europe, including in the Netherlands.
Wiltsey Stirman (2020) comments on challenges in the assessment of providers’ competence, observer-rated quality of service delivery and calibration of fidelity ratings. She emphasizes the need to assess the process of implementation in addition to fidelity assessments. She also discusses research on adaptions of interventions and distinguishes between core functions and forms of delivery of complex interventions that may be important in further development of both programs/interventions and of fidelity scales.
Conclusions
The papers in this special section contribute to knowledge on psychometric properties of some fidelity scales for evidence-based practices in the treatment of people with psychoses. The section gives an overview and illustrates the present state of fidelity measurement and some aspects for further development and use of fidelity scales.
References
Bond, G. R., & Drake, R. E. (2019). Assessing the fidelity of evidence-based practices: History and current status of a standardized measurement methodology. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-019-00991-6.
Bond, G. R., Drake, R. E., McHugo, G. J., Rapp, C. A., & Whitley, R. (2009). Strategies for improving fidelity in the National Evidence-Based Practices Project. Research on Social Work Practice, 19, 569–581.
Egeland, K. M., Heiervang, K. S., Landers, M., Ruud, T., Drake, R. E., & Bond, G. R. (2019). Psychometric Properties of a Fidelity Scale for illness management and recovery. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-019-00992-5.
Heiervang, K. S., Egeland, K. M., Landers, M., Ruud, T., Joa, I., Drake, R. E., et al. (2020). Psychometric Properties of the General Organizational Index (GOI): A measure of individualization and quality improvement to complement program fidelity. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-020-01025-2.
Joa, I., Johannessen, J. O., Heiervang, K. S., Sviland, A. K., Nordin, M. H. A., Landers, M., et al. (2020). The Family Psychoeducation Fidelity Scale: Psychometric properties. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-020-01040-3.
McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., et al. (2007). Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services, 58(10), 1279–1284. https://doi.org/10.1176/appi.ps.58.10.1279.
Ruud, T., Hoifodt, T. S., Hendrick, D. C., Drake, R. E., Hoye, A., Landers, M., et al. (2020a). The Physical Health Care Fidelity Scale: Psychometric properties. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-020-01019-0.
Ruud, T., Drivenes, K., Drake, R. E., Haaland, V. O., Landers, M., Stensrud, B., et al. (2020b). The Antipsychotic Medication Management Fidelity Scale: Psychometric properties. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-020-01018-1.
van Weeghel, J. (2020). Fidelity Scale: From black box to Holy Grail. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-020-01057-8.
Wiltsey Stirman, S. (2020). Commentary: Challenges and opportunites in the assessment of fidelity and related constructs. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-020-01069-4
Acknowledgments
Open Access funding provided by Akershus University Hospital (Ahus).
Funding
This work was supported by Helse Sør-Øst RHF [Grant Number 2015106].
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Ruud, T., Drake, R.E. & Bond, G.R. Measuring Fidelity to Evidence-Based Practices: Psychometrics. Adm Policy Ment Health 47, 871–873 (2020). https://doi.org/10.1007/s10488-020-01074-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-020-01074-7