Abstract
Epidemiological research over the past two decades has highlighted substance use disparities that affect Native Hawaiian and Pacific Islander youth, and the lack of effective approaches to address such disparities (Okamoto et al. in Asian American Journal of Psychology 10(3):239–248, 2019). The Ho‘ouna Pono curriculum is a culturally grounded, teacher-implemented, video-enhanced substance use prevention program that has demonstrated efficacy in rural Hawaiʻi in a large-scale trial (Okamoto in Asian American Journal of Psychology 10(3):239–248, 2019). Despite its potential to ameliorate health disparities and address youth substance use, prevention programs such as Ho‘ouna Pono have been poorly disseminated and implemented across Hawaiʻi, raising the question: Why are effective prevention programs not used in communities that most need them? The present study used concept mapping to understand previously identified implementation barriers and develop implementation strategies for Ho‘ouna Pono. Seven Hawaiʻi Department of Education (HIDOE) educational leaders and administrators sorted Ho‘ouna Pono implementation barriers (e.g., “There is a lack of HIDOE funding to support prevention curricula”), named concepts, and rated barriers’ perceived impact and difficulty. Multidimensional scaling and cluster analysis yielded a five-cluster solution: (1) Kumu (Hawaiian word for teacher) Controlled, (2) School Level Buy-in, (3) Curriculum, (4) Student Attitudes + Mindsets (Family + Community), and (5) Policy. Participant ratings identified eight high-impact and low-difficulty barriers. Discussion revealed important intersections among barriers indicating the need for coordinated and cross-level implementation strategies to support Ho‘ouna Pono sustainment. Brainstormed implementation strategies using participants’ own language highlighted a need for participatory methods in school settings to bidirectionally share ways to best sustain substance use prevention programs.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Decades of research have indicated the need for substance prevention programs for Native Hawaiian and Pacific Islander (NHPI) youth, given their higher rates of drug use than non-NHPI youth (Mayeda et al., 2006; Nishimura et al., 2013; Wong et al., 2004). While few programs have been developed specifically for NHPI youth (Durand et al., 2015; Edwards et al., 2010) and for rural communities (Okamoto et al., 2014; Saka et al., 2014), even fewer are developed with community engaged and culturally grounded approaches (Okamoto et al., 2014). Culturally grounded prevention programs can reach and motivate youth by reflecting the cultural, community, and social context of the targeted population (Marsiglia & Kulis, 2009). One such program that has demonstrated efficacy is Ho‘ouna Pono, a video-enhanced, teacher-delivered prevention intervention developed and tested with youth from rural Hawaiʻi Island. The intervention significantly improved youth substance use resistance strategies (e.g., avoid, explain, leave) in a 6-month randomized pilot study (Okamoto et al., 2016). In a subsequent fully powered, large-scale randomized controlled trial, receipt of the intervention was associated with a significantly slower rate of growth in cigarette and e-cigarette use (Okamoto et al., 2019).
Despite these promising advances in building prevention interventions for youth in their schools and communities, implementation has often been hindered by various contextual barriers, including shifting school priorities, teacher perceptions of the intervention, and changing societies and communities (Eisman et al., 2022; Okamura et al., 2022). Owens and colleagues (2014) noted that schools are uniquely challenging settings for implementation. They often differ from each other in their organizational culture, include professionals from a variety of disciplines (e.g., teachers, administrators, psychologists), and the school calendar may truncate innovation implementation efforts. Eisman et al. (2022) identified acceptability, intervention-context fit, and adaptability as implementation domains that posed challenges to teachers implementing a universal health prevention intervention with fidelity. Other school-based implementation research has found that organization characteristics like leadership and climate impact both individual attitudes toward the intervention and fidelity (Locke et al., 2016; Locke et al., 2020; Shoesmith et al., 2021; Zhang et al., 2023).
In our research about factors affecting Ho‘ouna Pono implementation, teachers and other school staff identified competing priorities, lack of training, and external (i.e., school administration) pressure for other initiatives as implementation barriers (Okamoto et al., under review). Another important barrier identified was the emergence of new drug modalities and devices (e.g., electronic nicotine delivery systems, e-cigarettes, vapes) that were not addressed by the intervention. Moreover, Ho‘ouna Pono has not been sustained in the schools that participated in the efficacy trial and there have not been efforts to promote widespread, systematic implementation of the intervention. Therefore, as we revise the program to address new drug modalities and devices, we are concurrently attempting to generate implementation strategies: Such work could promote widespread implementation of an efficacious prevention intervention and shift substance use trajectories for students who engage in the intervention.
Implementation Frameworks
Implementation science employs a wide array of theories, frameworks, and measures to explore context and systematically study the process of translating science to practice (Eccles & Mittman, 2006; Wang et al., 2023). Determinant frameworks are used to identify the contextual factors that impede or facilitate implementation in a specific setting. One commonly used determinant framework is the Consolidated Framework for Implementation Research (CFIR), which was first developed by Damschroder and colleagues in 2009 and updated in 2022 based on user feedback (Damschroder et al., 2022). The current study launched in 2021 and therefore used the original version of the CFIR. Both versions of the CFIR organized implementation barriers and facilitators into five domains: (a) outer setting—the system where an organization implementing an innovation exists, (b) inner setting—the organization implementing the innovation, (c) characteristics of individuals—the people implementing the innovation, (d) characteristics of the innovation—the “thing” being implemented (e.g., Ho‘ouna Pono), and (e) process—the stages by which the innovation is implemented.
The CFIR was used in a previous Ho‘ouna Pono study to identify 50 barriers via focus groups with 24 Hawaiʻi State Department of Education (HIDOE) teachers and frontline staff (Okamoto et al., 2020). Focus group members were asked open-ended questions about Ho‘ouna Pono implementation based on CFIR domains and revealed consistent themes, including the outer setting (e.g., “Marijuana use is socially acceptable on Hawai‘i island, diminishing the need”), inner setting (e.g., “There is a lack of HIDOE funding to support prevention curricula”), and individual characteristics of the implementor (e.g., “Some HIDOE teachers are resistant to trying new curricula…because it feels like one more thing administrators want [us] to do in the classroom”).
A key benefit of using a determinant framework such as CFIR is that the identification of barriers and facilitators can be used to develop effective implementation strategies (Proctor et al., 2013). Implementation strategies are the actions that community members can take to implement an innovation in a specific setting. Powell and colleagues identified and defined 73 discrete implementation strategies that comprise the Expert Recommendations for Implementing Change (ERIC) strategy compilation (Powell et al., 2012, 2015; Waltz et al., 2015). Many websites, institutes, and technical assistance centers contain resources for using, identifying and selecting implementation strategies, which may be useful for school-based implementation efforts (e.g., Active Implementation Hub, https://implementation.fpg.unc.edu/; Penn Implementation Science Institute, https://mshp.med.upenn.edu/implementation-science-institute; University of Washington, https://impsciuw.org/). However, these resources take some expertise to navigate, which may pose a challenge for school-based staff less familiar with implementation science. Using existing implementation frameworks and taxonomies to create a shared understanding around implementation between researchers and school-based staff is a valuable first step for implementation of efficacious interventions such as Ho‘ouna Pono.
Participatory Methods
Participatory methods create processes and change by promoting democratic collaboration with those affected by an issue through meaningful engagement (Vaughn & Jacquez, 2020). For Ho‘ouna Pono, engagement with school-based staff and leadership in co-creating implementation and evaluation aligns with participatory method values (Ramanadhan et al., 2023). Concept mapping is one promising participatory approach that “integrates well-known group processes such as brainstorming and unstructured sorting with multivariate statistical methods” (Trochim & Kane, 2005, p. 187) to generate and validate community-derived meaning of constructs (Powell et al., 2017a, 2017b). The concept mapping steps include (a) preparation, (b) idea generation, (c) organization/sorting/rating, (d) visual representation, (e) interpretation, and (f) utilization. We used concept mapping to facilitate the co-creation of the implementation process and promote bidirectional learning between researchers and school-based staff (see Okamura et al., 2020 for more details). Concept mapping can also be used to identify and prioritize implementation barriers, which begins by selecting participants, defining the conceptual focus, generating a list of barriers, and gaining consensus on the most critical barriers. These steps were completed in a prior trial (Okamoto et al., under review), which obtained consensus around a list of 50 barriers to the intervention (described further in Methods).
The current study is built on this formative work and initial concept mapping steps to develop an implementation action plan for Ho‘ouna Pono. The study progressed through two phases: ranking, sorting, and reflecting on the barriers using the CFIR determinant framework (Phase One) and refining the concept map and generating implementation strategies using the ERIC taxonomy (Phase Two). The goal of concept mapping was to understand Ho‘ouna Pono implementation barriers from the perspective of educational leadership in Hawai‘i Island and HIDOE, while promoting state-, district/complex-, and school-level investment and “buy-in” for the strategies used to implement the curriculum (Waltz et al., 2019). This participatory approach aligns with other community-engaged, team-based, and equitable methods to ensure equal voice in preparing, implementing, and sustaining an intervention (Vaugh & Jacquez, 2020). The study was exploratory with the goal of generating a shared understanding of Ho‘ouna Pono implementation by eliciting concepts and strategies for further evaluation and testing.
Method
Participants
The first author and a HIDOE consultant/co-author on Hawaiʻi Island created a list of potential state (n = 13) and complex/district (n = 17) participants based on their leadership roles in promoting health education standards, health curriculum, and district- and school-level decision making. The research staff sent recruitment emails to the group and followed up via individual emails and phone calls as needed. Seven HIDOE state and complex leaders consented and participated in this study. It was important to recruit leaders who did not have experience implementing the curriculum to both provide insight into teacher and other school staff perspectives and to engage leadership to develop strategies that could be addressed with their positionality and role. The group included one Complex Area Superintendent, five State or District Education Specialists, and one School-based Behavioral Health Specialist. Participants self-identified as predominantly female (62.5%) and as Native Hawaiian (16.7%), Asian (25%), Portuguese (16.7%), White (25%), and Other Pacific Islander (8.3%). One participant did not identify their racial background. One third of participants held a master’s degree or higher.
Procedure
The objective, guiding framework, and key strategies used in each study phase are outlined in Table 1. The two study phases were: (a) in-person sorting, ranking, and discussion and (b) online naming, reorganizing, and strategy brainstorming. The research team conducted multidimensional scaling and cluster analysis (see Data Analytic Plan) between meetings. After the first meeting, two participants that could not attend the in-person sorting completed their sorting and rating asynchronously. One person opted out of the in-person ranking activity. Participants were compensated for each time point with a $50 donation to their school or complex area. Data integrity and security were maintained through standardized procedures aligned with university standards with only trained staff having data access. This study was deemed exempt by the University of Hawai‘i, Hawai‘i Pacific University, and Hawai‘i Department of Education institutional review boards.
Phase One—Sorting, Rating, and Reflecting
In the first phase, participants met for two hours (with lunch included) to complete informed consent, collect background demographic and training information, and sort and rank barriers. Consensus was attained on fifty barriers in a prior study via focus groups with 24 HIDOE teachers and frontline staff (Okamoto et al., 2020). In these focus groups, staff were asked open-ended questions about Ho‘ouna Pono implementation based on CFIR domains. Participant feedback yielded a set of consistent barriers, spanning the outer setting (e.g., “Marijuana use is socially acceptable on Hawai‘i island, diminishing the need”), inner setting (e.g., “There is a lack of HIDOE funding to support prevention curricula”), and individual characteristics of the implementor (e.g., “Some HIDOE teachers are resistant to trying new curricula…because it feels like one more thing administrators want [us] to do in the classroom”).
Participants were instructed to sort the 50 barriers based on their own version of how they are related. General guidance was given to participants that clarified that concept groupings tended to range from five to 20 concepts, avoided labels like “important” or “other,” and could be composed of a single statement if the statement did not relate to others. Participants also ranked barriers on a five-point scale regarding the extent to which they perceived a particular barrier as impacting implementation (1 = no impact, 2 = minimal impact, 3 = some impact, 4 = significant impact, and 5 = substantial impact) and the difficulty in overcoming each barrier (1 = very easy, 2 = easy, 3 = a little difficult, 4 = moderately difficult, and 5 = very difficult). A general discussion closed the first phase with participants answering open reflection questions such as “What are the most impactful barriers for you?” and “Describe implementation strategies to address these barriers.” The first author facilitated the discussion and provided summaries, prompts, and definitions for terms with the goal of producing insight and thoughtfulness for the next phase.
Phase Two—Refining of the Concept Map and Generation of Strategies
In the second phase, concepts refined by multidimensional scaling and cluster analysis were shared with participants. Participants were shown various concept mapping visualizations (i.e., cluster map, go zone) to understand and reflect on their individual understanding relative to the group’s collective understanding of Ho‘ouna Pono implementation barriers. The group reviewed individual barriers within clusters, reorganized clusters (i.e., moved individual barriers from one cluster to another), and renamed them to reflect shared understanding. A final discussion question followed each concept asking participants to reflect on “What implementation strategies could be used to address this concept/cluster of barriers?” Participants used the ERIC compilation (Powell et al., 2012, 2015; Waltz et al., 2015) as a resource to inspire potential strategies but they could also brainstorm strategies and ways of operationalizing those strategies using their own terms and language. There was no prioritization or selection of strategies beyond the brainstorming discussion. The research team told participants that these brainstormed strategies would inform the implementation action plan and future implementation research.
Data Analytic Plan
Demographic information, sort piles, and difficulty and impact ratings were entered and verified in GroupWisdom software (Concept Systems Incorporated, 2021). Multidimensional scaling was used to create a matrix of similarities between statements and HIDOE groupings from the sorting task. A two-dimensional solution served as the input for a cluster analysis using Ward’s algorithm, which uses minimal variance at each step to identify the optimal value of an objective function to define orthogonal clusters (Anderberg, 1973; Everitt, 1980). Cluster analysis adhered to the typical rule of at least five statements per cluster. Eleven solutions were examined based on point map and interpretability of the clusters (Johnsen et al., 2000; Trochim & McLinden, 2017). Difficulty and impact ratings were layered onto the cluster map to understand relative difficulty and impact via Go Zone graph. The concept mapping discussions were transcribed and reviewed to contextualize clusters and strategies.
Results
Phase One: Sorting, Rating, and Reflecting
The point map (stress valueFootnote 1 = 0.19, iterationsFootnote 2 = 18; not presented) visually represented multidimensional scaling and the extent to which each barrier was commonly grouped with other barriers. Cluster analysis revealed maps with four to 14 clusters, and bridging values ranging from zero to one were examined by item and cluster. Lower bridging values are typically considered anchoring items that represent homogeneity among items, whereas higher bridging values indicate a higher relationship across various statements on the map. Layer values aggregate bridging values for the concept, with higher values indicating more connectedness with other concepts and items within the map. After reviewing the total eleven cluster maps, the research team reviewed the four-, five-, and six-cluster solution items for cohesive themes as identified by each label. The five-cluster solution was chosen based on stress values, iterations, bridging values, layer values, and similarities to CFIR domains. Table 2 presents the initial and final cluster solutions with individual items, and Fig. 1 shows the final cluster map described in phase two of the study.
The Go Zone map (see Fig. 2) integrated difficulty and impact ratings such that barriers identified fell into one of four quadrants: low difficulty and high impact (green), high difficulty and high impact (yellow), high difficulty and low impact (red), and low difficulty and low impact (blue). For example, item 32, “The 9-lesson Ho‘ouna Pono curriculum has too much drug-specific content to fit within a semester-long health course,” had an average difficulty rating of approximately two and impact rating of approximately three (lower difficulty and higher impact) and is therefore in the green quadrant. Table 3 highlights eight items in the Go Zone map (see green quadrant in Fig. 1) with high impact and low difficulty scores. These items potentially represent ideal barriers to prioritize in the implementation plan.
The phase one discussion reflecting on Ho‘ouna Pono barriers and facilitators elucidated themes related to shared ownership and responsibility over the intervention. For example, one participant noted that characteristics of a successful implementation effort included ownership and normalization of the intervention, cutting across the “Kumu (Hawaiian word for teacher) Controlled” and “School Level Buy-In” domains. Other important “Kumu Controlled” strategies were training and creating teams (e.g., “make sure the teachers have a team”) to learn with a cohort of teachers and other staff. For example, one participant noted:
[Teachers have] taken ownership over it and created this, you know, kind of relationship. So, it prioritizes something that used to not be a normal part of – but now it's more like normalized.
Also related to the “Kumu Controlled” domain, participants brought up another example of a successful implementation strategy (peer-to-peer learning) when teachers were required to pivot to online teaching during the COVID-19 pandemic.
They [teachers] convened these peer-to-peer sessions, and it was like come after school, drop in for an hour, learn from a friend and so, teachers started to just share [their] practice-and it became this little community, and all said and done, it attracted private school, public, charter, and home school, [and] even students joined.
Phase Two: Refining of the Concept Map and Generation of Strategies
In the first part of the phase two discussion, participants were asked to review the barriers within each cluster to look for similarities and any outliers that needed to be moved. Then, cluster names were reviewed and edited within the group. Cluster names included “Kumu Controlled,” “School Level Buy-in,” “Curriculum,” “Student Attitudes + Mindsets (Family + Community),” and “Policy.” Participant discussion changed one cluster name from “School Level” to “School Level Buy-In.” Additionally, three barriers (“The Ho‘ouna Pono curriculum does not extensively cover current or recent forms of substance use, such as vaping,” “The Ho‘ouna Pono curriculum lacks a social media presence (e.g., Twitter, Instagram, Facebook),” and “After 10 years, the Ho‘ouna Pono curriculum may need updating, by changing youths’ language and jargon depicted in the videos”) were moved from the “Student Attitudes + Mindsets (Family + Community)” to “Curriculum” based on fit and proximity on the map.
In the second part of the phase two discussion, participants brainstormed potential implementation strategies within concepts while referencing the ERIC implementation strategies handout. For example, when presented the “Student Attitudes + Mindsets (Family + Community)” concept, one participant noted the potential effectiveness of local school-community councils to elevate student voice to promote Ho‘ouna Pono awareness and implementation. Related to “Policy,” one participant suggested talking with legislators to promote visibility of the curriculum.
Are our legislators/state education leaders aware of the work you all have done? I am not really sure what it [will] take…but awareness might be the first step at that level.
Another participant affirmatively stated “I’m wondering if combining a grassroots effort with high-level decision makers might help. The BOE [Board of Education], local legislators, etc. could be involved too.” During the “Kumu Controlled” discussion, one participant noted about the concept mapping process that “You’re still allowing space for voice—all of our voices so, yeah, I think it's just a matter of training.” Coordinated and cohort-based training, integration into existing curricula with lesson plans to address all students, and drug treatment counselor, school counselor and school-based behavioral health involvement were also identified as potential strategies to address the “Kumu Controlled” concept. For the “Curriculum” domain, one participant stated that using teachers to develop the curriculum might aid in wider applicability to all students and motivation to use the intervention. At the “School Level Buy-In”, more required health education courses in middle school, creating new incentives for continuing education for teachers (especially those who have reached their ceiling), and aligning educational policies with outcomes that address health and well-being were offered as potential strategies. A compilation of brainstormed strategies was given to participants as a starting point for future implementation discussions for tailored approaches.
Discussion
Implementing culturally grounded substance prevention programs for NHPI youth in schools is a public health priority. This study examined educational leaders’ perspectives on implementation barriers and used a community-engaged and participatory approach (concept mapping) to generate types of barriers and potential implementation strategies to address these concerns. The process highlighted how participatory methods can be used to engage and generate a robust list of potential strategies in a diverse group of HIDOE leadership. The concept mapping process also created bidirectional learning with HIDOE leadership and researchers and let HIDOE leadership learn from frontline teachers and staff about Ho‘ouna Pono barriers. The process and outcome of this study provides a thoughtful pathway to Ho‘ouna Pono implementation.
Concept mapping results revealed clusters that were aligned with CFIR domains as applied to schools. For example, “Kumu Controlled” barriers related to the CFIR Individual Characteristics and Implementation Process. Additionally, the “School Level Buy-in” cluster was similar to the CFIR Inner Setting and the “Curriculum” cluster was similar to the CFIR Innovation domain. There was an intersection between the “Student Attitudes + Mindset (Family + Community)” and “Policy” clusters that related to the CFIR Outer Setting domain. There is value added in having school staff members, in this case the HIDOE, name these constructs for better understanding and meaning. Indeed, emerging evidence suggests that team-based approaches to implementation science (Mathieu et al., 2019), including shared mental models (Mathieu et al., 2008), can create shared ownership and responsibility for the innovation.
Findings from this study revealed interconnected barrier concepts related to Ho‘ouna Pono, especially for “School Level Buy-In,” “Curriculum,” and “Student Attitudes + Mindset (Family + Community).” Interestingly, concept map bridging values suggested that participants perceived the “Kumu Controlled” barriers as homogenous and did not tend to group these items with other barriers associated with other concepts like “Policy.” The Go Zone map revealed eight high-impact and low-difficulty barriers from the “Kumu Controlled,” “Curriculum,” and “Policy” concepts. Interestingly, the “Policy” high-impact and low-difficulty barriers related to state and complex standards, which may have been an artifact of the elevation at which participants perceived implementation in combination with their positionality. The “Kumu Controlled” and “Curriculum” high impact and low difficulty barriers related mostly to awareness around the curriculum, its contents, and the applicability over existing substance use prevention. It is interesting that HIDOE leadership did not perceive these concepts to be highly related as evidenced by the lower bridging values in the “Kumu Controlled” concept. This intersection of individual barriers and concept proximity is consistent with previous literature in school-based implementation that suggests that people vary in their understanding of their own and others’ roles (Owens et al., 2014) and may indicate a need for co-created implementation strategies that involve state-, complex-, and school-level staff.
It is important to situate these findings in the previous studies examining Ho‘ouna Pono implementation using the CFIR with teachers and frontline HIDOE staff (Okamoto et al., 2019). Given the content in the concept mapping clusters (see Table 2), it appears that state and complex HIDOE leadership perceive Ho‘ouna Pono barriers differently. Okamoto et al. (under review) found a four-factor structure (innovation barriers, state-level barriers, teacher-level barriers, and administrator-level barriers) using exploratory factor analysis on an identical barrier survey with 204 HIDOE teachers, administrators, and staff. These empirically derived groupings are less aligned with CFIR domains than the rationally derived concepts in the current study. Inner Setting barriers that map onto “School Level Buy-In” have impact on what teachers do; however, these barriers are largely outside of their control. Similarly, Individual Characteristics barriers that map onto the “Student Attitudes + Mindsets (Family + Community)” domain influence the extent to which teachers can deliver curricula related to substance prevention and health. While outside the scope of the current study, future studies could engage teachers and staff through concept mapping to create a shared understanding of barriers among teachers, school administrators, and HIDOE leadership.
Implications for School-Based Mental Health Promotion and Substance Use Prevention Program Implementation
The Go Zone Map identified multilevel barriers that would have a high impact on successful implementation, while being less difficult to overcome (see Table 2). HIDOE leadership could prioritize addressing these barriers, which may have broader implications for school-based prevention. Three barriers (39, 28, 7) specifically target teacher perceptions (i.e., “Kumu Controlled”), which may be addressed with strategies that employ cohort-based experiential training to build teacher efficacy in speaking with youth about drug and alcohol use, integration with current substance use curricula, and involving other school staff (e.g., counselors, school-based behavioral health specialists) to deliver the intervention. An additional three barriers (21, 22, 32) related to the “Curriculum” that are currently being addressed with updated video content to reflect modern language and current forms of drug use like vaping and could incorporate more teachers in the development of the content to bridge existing curriculum. The final two barriers (37 and 14) focus on HIDOE “Policy” and outer setting related concerns which may be addressed with legislative priorities (i.e., speaking with legislators) and internal HIDOE policies to mandate substance use-related health education.
Additional strategies brainstormed by HIDOE leadership included using school community councils to elevate student voice (for the “Student Attitudes + Mindsets (Family + Community)” concept), creating policies to require more health education courses and aligning educational outcomes that address health and well-being, and creating new incentives for continuing education (for the “School Level Buy-In” concept). These strategies align with a multilevel approach to promote Ho‘ouna Pono across community-, student-, teacher-, and administrator-levels and the need for co-creation within these groups to sustain the intervention. Similar school-based prevention programs could focus on promoting teacher self-efficacy, ensuring updated intervention content, and addressing outer setting issues to support implementation (Cook et al., 2019; Kuriyan et al., 2021). Moreover, aligning implementation strategies that have been adapted for (Cook et al., 2019) and evaluated for feasibility and importance within school-based settings (Lyon et al., 2019) will be an important next step to expanding the generalizability of the current study’s findings.
Strengths and Limitations
Community-engaged and participatory approaches allow researchers to connect and partner with school staff members differently. Concept mapping is a participatory approach in which researchers and school staff members have equal input in creating shared meaning and ownership, in this case regarding implementation. The process elevates voices and redistributes power in the research process as one participant highlighted in the title of this manuscript. Too often, researchers engage with communities through advisory boards to address these concerns, but this method falls short because it creates an additional burden on school staff, and most studies do not have flexibility built in to change research designs to meet their needs. The current study’s approach also bridged school staff, teachers, and administrators with complex and state-level leadership to share and build off implementation barrier perspectives from those closest to the delivery of the intervention. Participatory methods have the potential not only to create generalizable findings that elevate voice but also to engage people from differing levels within school organizations to foster a shared understanding of an implementation problem.
The results of this study inform the co-planning and design of an implementation project to spread Ho‘ouna Pono to a novel school district in Hawai‘i. The present study provides direction for proactively addressing potential implementation barriers within the new school district by redistributing power to youth and families to plan implementation, while simultaneously supporting school-level engagement and “buy-in” for the implementation of the curriculum. Lessons learned from this study have the potential to inform how concept mapping can aid in co-creating implementation strategies and action plans (Powell et al., 2017a, b).
The current study is not without limitations. It is unclear how representative concept mapping findings are of the larger HIDOE system and other districts given the small sample size. In a review of 104 concept mapping dissertations, sample size ranged in each phase—brainstorming from one to 555, sorting from five to 152, rating from zero to 152, and interpreting from one to 112 (Donnelly, 2017). The goal of the current study was to elicit sufficient feedback on barriers and strategies to develop a Ho‘ouna Pono implementation plan. As such, participants were carefully selected through consultation with educational partners, community members, and school staff, and the 30 originally contacted represented State Education Specialist (n = 14), or District Education Specialists (n = 12), or Complex Area Superintendents (n = 5). The current sample is representative of the contacted participant pool in terms of their administrative and leadership HIDOE experience with the exception of the School-based Behavioral Health Specialist, who was included based on the suggestion of one Complex Area Superintendent. Nevertheless, we cannot conclude that the perceptions of study participants represent the other HIDOE leadership that did not participate in the study.
Moreover, the findings represent a snapshot in time that may not be representative of ever-changing priorities within the HIDOE and larger policy ecology, such as federal educational standards and increased mental health parity (Raghavan et al., 2008; Wortham et al., 2023). The implementation science field would benefit from more clearly articulating time within implementation phases and studies to elucidate the progression of implementation processes (Alley et al., 2023). Additionally, given that the CFIR was used to develop barriers, this may have limited the identified concepts. Finally, while beneficial to researchers, the ERIC implementation strategies were difficult for participants to understand. For example, one participant noted that the strategies felt too discrete and there was overlap between many of them. This is a challenge that has been identified by others, and there are ongoing efforts to further clarify the terminology used within the ERIC compilation (e.g., Yakovchenko et al., 2023). Future studies may wish to integrate tools that link strategies to barriers like the CFIR-ERIC matching tool (Waltz et al., 2019) and would ideally continue to involve youth and families in creating and testing implementation strategies in school settings. As the implementation science field continues to make strides in developing and specifying implementation strategies, a potentially vital role of researchers and implementation specialists will be to define and operationalize these strategies in local language for their given project and community. For example, the ERIC compilation has been translated into German, and Japanese and Spanish language translations are underway (Regauer et al., 2021; Van Pelt et al., 2023). This study could also have been strengthened by further contextualizing policy and outer setting recommendations with HIDOE leadership to create commitments to addressing these concepts and individual barriers to aid in future prevention implementation initiatives.
Conclusions
This study benefited from eliciting implementation barrier perspectives across different levels within the HIDOE. It also engaged both state- and complex-school staff and researchers in creating shared meaning and ownership over the implementation process. This study culminated in a set of potential implementation strategies that can be used to implement the intervention throughout schools in Hawai‘i. The main barrier concepts were related to teachers, school context, the innovation, student, family, and community attitudes, and policies, and the potential strategies generated including cohort-based training, integration of the curriculum to existing lesson plans, using non-teachers to deliver the intervention, creating new requirements, outcomes, and incentives for health education, elevating student voice through school community councils, and talking with legislators. Implementation is an ongoing effort that requires years of relationship building, planning, and evaluation. Our findings suggest that, specifically for school-based implementation, determinants interact in unique ways, highlighting the need for multilevel implementation strategies. Specifically, a combination of strategies that address teachers’ relative priority (i.e., competing demands) should take in to account federal, state, and school priorities for health education, teachers’ current continuing education credit level, and family and community culture related to substance use (including teachers’ personal experience and current attitudes). Future studies may examine time invested and the cost of those investments to inform future funding priorities.
Change history
26 July 2024
A Correction to this paper has been published: https://doi.org/10.1007/s12310-024-09693-3
Notes
The stress value is the major metric for indicating the degree to which a multidimensional scaling solution fits the original similarity matrix. The better the fit of a map to the similarity matrix, the lower the stress value. In typical projects, the stress is usually from .10 to .35 and the results are interpretable.
Iterations is the number of attempts completed by the software to get the lowest stress value that represents the aggregated sort data.
References
Alley, Z. M., Chapman, J. E., Schaper, H., & Saldana, L. (2023). The relative value of pre-Implementation stages for successful implementation of evidence-informed programs. Implementation Science: IS, 18(1), 30. https://doi.org/10.1186/s13012-023-01285-0
Anderberg, M. (1973). The broad view of cluster analysis. Elsevier.
The Concept System® groupwisdom™ (Build 2022.24.01) [Web-based Platform]. (2021). Ithaca, NY. Available from https://www.groupwisdom.tech
Cook, C. R., Lyon, A. R., Locke, J., Waltz, T., & Powell, B. J. (2019). Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science, 20, 914–935. https://doi.org/10.1007/s11121-019-01017-1
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(50), 1–15. https://doi.org/10.1186/1748-5908-4-50
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022). The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Science, 17(1), 1–16. https://doi.org/10.1186/s13012-022-01245-0
Donnelly, J. P. (2017). A systematic review of concept mapping dissertations. Evaluation and Program Planning, 60, 186–193. https://doi.org/10.1016/j.evalprogplan.2016.08.010
Durand, Z., Cook, A., Konishi, M., & Nigg, C. (2015). Alcohol and substance use prevention programs for youth in Hawaii and Pacific Islands: A literature review. Journal of Ethnicity in Substance Abuse, 15(3), 240–251. https://doi.org/10.1080/15332640.2015.1024811
Eccles, M. P., & Mittman, B. S. (2006). Welcome to implementation science. Implementation Science, 1(1), 1. https://doi.org/10.1186/1748-5908-1-1
Edwards, C., Giroux, D., & Okamoto, S. K. (2010). A review of the literature on Native Hawaiian youth and drug use: Implications for research and practice. Journal of Ethnicity in Substance Abuse, 9(3), 153–172. https://doi.org/10.1080/15332640.2010.500580
Eisman, A. B., Palinkas, L. A., Brown, S., Lundahl, L., & Kilbourne, A. M. (2022). A mixed methods investigation of implementation determinants for a school-based universal prevention intervention. Implementation Research and Practice, 3, 26334895221124962. https://doi.org/10.1177/26334895221124962
Everitt, B. (1980). Cluster analysis. Halsted.
Johnsen, J. A., Biegel, D. E., & Shafran, R. (2000). Concept mapping in mental health: Uses and adaptations. Evaluation and Program Planning, 23(1), 67–75. https://doi.org/10.1016/S0149-7189(99)00038-5
Kuriyan, A., Kinkler, G., Cidav, Z., Kang-Yi, C., Eiraldi, R., Salas, E., & Wolk, C. B. (2021). Team strategies and tools to enhance performance and patient safety (TeamSTEPPS) to improve collaboration in school mental health: Protocol for a mixed methods hybrid effectiveness-implementation study. JMIR Research Protocols, 10(2), e26567. https://doi.org/10.2196/26567
Locke, J., Beidas, R. S., Marcus, S., Stahmer, A., Aarons, G. A., Lyon, A. R., & Mandell, D. S. (2016). A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools. Implementation Science, 11(1), 1–9. https://doi.org/10.1186/s13012-016-0501-8
Locke, J., Kang-Yi, C., Frederick, L., & Mandell, D. S. (2020). Individual and organizational characteristics predicting intervention use for children with autism in schools. Autism, 24(5), 1152–1163. https://doi.org/10.1177/1362361319895923
Lyon, A. R., Cook, C. R., Locke, J., Davis, C., Powell, B. J., & Waltz, T. J. (2019). Importance and feasibility of an adapted set of implementation strategies in schools. Journal of School Psychology, 76, 66–77. https://doi.org/10.1016/j.jsp.2019.07.014
Marsiglia, F. F., & Kulis, S. (2009). Diversity, oppression, and change: Culturally grounded social work. Lyceum Books.
Mathieu, J. E., Gallagher, P. T., Domingo, M. A., & Klock, E. A. (2019). Embracing complexity: Reviewing the past decade of team effectiveness research. Annual Review of Organizational Psychology and Organizational Behavior, 6, 17–46. https://doi.org/10.1146/annurev-orgpsych-012218-015106
Mathieu, J., Maynard, M. T., Rapp, T., & Gilson, L. (2008). Team effectiveness 1997–2007: A review of recent advancements and a glimpse into the future. Journal of Management, 34(3), 410–476. https://doi.org/10.1177/0149206308316061
Mayeda, D. T., Hishinuma, E. S., Nishimura, S. T., Garcia-Santiago, O., & Mark, G. Y. (2006). Asian/Pacific Islander youth violence prevention center: Interpersonal violence and deviant behaviors among youth in Hawai‘i. Journal of Adolescent Health, 39(2), 276.e271-276.e211. https://doi.org/10.1016/j.jadohealth.2005.12.006
Nishimura, S. T., Hishinuma, E. S., & Goebert, D. (2013). Underage drinking among Asian American and Pacific Islander adolescents. Journal of Ethnicity in Substance Abuse, 12(3), 259–277. https://doi.org/10.1080/15332640.2013.805176
Okamoto, S. K., Okamura, K. H., Marshall, S. M., Chin, S. K., Carson, A. B., An, K. J., Song, S., Saladino, P., Prado, G., & Kulis, S. S. (revise and resubmit). Examining implementation determinants of a culturally grounded, school-based prevention curriculum in rural Hawai‘i: A test development and validation study. Implementation Research and Practice.
Okamoto, S. K., Helm, S., Chin, S. K., Hata, J., Hata, E., & Okamura, K. H. (2020). The implementation of a culturally grounded, school-based, drug prevention curriculum in rural Hawai‘i. Journal of Community Psychology, 48(4), 1085–1099. https://doi.org/10.1002/jcop.22222
Okamoto, S. K., Kulis, S. S., Helm, S., Chin, S. K., Hata, J., Hata, E., & Lee, A. (2019). An efficacy trial of the Ho‘ouna Pono drug prevention curriculum: An evaluation of a culturally grounded substance abuse prevention program in rural Hawai‘i. Asian American Journal of Psychology, 10(3), 239–248. https://doi.org/10.1037/aap0000164
Okamoto, S. K., Kulis, S., Helm, S., Lauricella, M., & Valdez, J. K. (2016). An evaluation of the Ho‘ouna Pono curriculum: A pilot study of culturally grounded substance abuse prevention for rural Hawaiian youth. Journal of Health Care for the Poor and Underserved, 27(2), 815–833. https://doi.org/10.1353/hpu.2016.0061
Okamoto, S. K., Kulis, S., Marsiglia, F. F., HolleranSteiker, L. K., & Dustman, P. A. (2014). A continuum of approaches toward developing culturally focused prevention interventions: From adaptation to grounding [Theoretical]. The Journal of Primary Prevention, 35(2), 103–112. https://doi.org/10.1007/s10935-013-0334-z
Okamura, K. H., Okamoto, S. K., Marshall, S. M., Chin, S. K., Garcia, P. M., Powell, B. J., & Mandell, D. S. (2022). Ho‘ouna Pono implementation: applying concept mapping to a culturally grounded substance use prevention curriculum in rural Hawai‘i schools. Implementation Science Communications, 3(1), 1–6. https://doi.org/10.1186/s43058-022-00359-2
Owens, J. S., Lyon, A. R., Brandt, N. E., Masia Warner, C., Nadeem, E., Spiel, C., & Wagner, M. (2014). Implementation science in school mental health: Key constructs in a developing research agenda. School Mental Health, 6, 99–111. https://doi.org/10.1007/s12310-013-9115-3
Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S. (2017a). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services & Research, 44(2), 177–194. https://doi.org/10.1007/s11414-015-9475-6
Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., Glass, J. E., & York, J. L. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157. https://doi.org/10.1177/1077558711430690
Powell, B. J., Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Barwick, M. A., Damschroder, L. J., Wensing, M., Wolfenden, L., & Lewis, C. C. (2017b). Toward criteria for pragmatic measurement in implementation research and practice: A stakeholder-driven approach using concept mapping. Implementation Science, 12(1), 118. https://doi.org/10.1186/s13012-017-0649-x
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10(1), 21. https://doi.org/10.1186/s13012-015-0209-1
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 1–11. https://doi.org/10.1186/1748-5908-8-139
Raghavan, R., Bright, C. L., & Shadoin, A. L. (2008). Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implementation Science, 3, 1–9. https://doi.org/10.1186/1748-5908-3-26
Ramanadhan, S., Alemán, R., Bradley, C. D., Cruz, J. L., Safaeinili, N., Simonds, V., & Aveling, E. L. (2023). Using participatory implementation science to advance health equity. Annual Review of Public Health, 45, 17.1-17.21. https://doi.org/10.1146/annurev-publhealth-060722-024251
Regauer, V., Seckler, E., Campbell, C., et al. (2021). German translation and pre-testing of consolidated framework for implementation research (CFIR) and expert recommendations for implementing change (ERIC). Implementation Science Communications, 2, 120. https://doi.org/10.1186/s43058-021-00222-w
Saka, S. M., Takeuchi, L., Fagaragan, A., & Afaga, L. (2014). Results of the 2013 Hawai‘i state and counties Youth Risk Behavior Surveys (YRBS) and cross-year comparisons. Curriculum Research & Development Group, University of Hawai‘i at Mānoa.
Shoesmith, A., Hall, A., Wolfenden, L., et al. (2021). Barriers and facilitators influencing the sustainment of health behaviour interventions in schools and childcare services: A systematic review. Implementation Science, 16, 62. https://doi.org/10.1186/s13012-021-01134-y
Trochim, W., & Kane, M. (2005). Concept mapping: An introduction to structured conceptualization in health care. International Journal for Quality in Health Care, 17(3), 187–191. https://doi.org/10.1093/intqhc/mzi038
Trochim, W. M., & McLinden, D. (2017). Introduction to a special issue on concept mapping. Evaluation and Program Planning, 60, 166–175. https://doi.org/10.1016/j.evalprogplan.2016.10.006
Van Pelt, A., Paniagua-Avila, A., Sanchez, A., Sila, S., Lowenthal, E., Powell, B. J., & Beidas, R. (2023). Spanish translation of the consolidated framework for implementation research (CFIR) and the Expert recommendations for implementing change (ERIC) taxonomy. Implementation Research and Practice, 4(S1), S164–S166.
Vaughn, L. M., & Jacquez, F. (2020). Participatory research methods–Choice points in the research process. Journal of Participatory Research Methods, 1(1), 1–13. https://doi.org/10.35844/001c.13244
Waltz, T. J., Powell, B. J., Fernandez, M. E., Abadie, B., & Damschroder, L. J. (2019). Choosing implementation strategies to address contextual barriers: Diversity in recommendations and future directions. Implementation Science, 14(42), 1–15. https://doi.org/10.1186/s13012-019-0892-4
Waltz, T. J., Powell, B. J., Matthieu, M. M., Damschroder, L. J., Chinman, M. J., Smith, J. L., Proctor, E. K., & Kirchner, J. E. (2015). Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the expert recommendations for implementing change (ERIC) study. Implementation Science, 10(109), 1–8. https://doi.org/10.1186/s13012-015-0295-0
Wang, Y., Wong, E. L. Y., Nilsen, P., et al. (2023). A scoping review of implementation science theories, models, and frameworks—an appraisal of purpose, characteristics, usability, applicability, and testability. Implementation Science, 18, 43. https://doi.org/10.1186/s13012-023-01296-x
Wong, M. M., Klingle, R. S., & Price, R. K. (2004). Alcohol, tobacco, and other drug use among Asian American and Pacific Islander adolescents in California and Hawaii. Addictive Behaviors, 29(1), 127–141. https://doi.org/10.1016/S0306-4603(03)00079-0
Wortham, W. K., Rodwin, A. H., Purtle, J., Munson, M. R., & Raghavan, R. (2023). Revisiting the policy ecology framework for implementation of evidence-based practices in mental health settings. Implementation Science: IS, 18(1), 58. https://doi.org/10.1186/s13012-023-01309-9
Yakovchenko, V., Chinman, M. J., Lamorte, C., et al. (2023). Refining expert recommendations for implementing change (ERIC) strategy surveys using cognitive interviews with frontline providers. Implementation Science Communication, 4, 42. https://doi.org/10.1186/s43058-023-00409-3
Zhang, Y., Cook, C., Fallon, L., Corbin, C., Ehrhart, M., Brown, E., & Lyon, A. (2023). The interaction between general and strategic leadership and climate on their multilevel associations with implementer attitudes toward universal prevention programs for youth mental health: A cross-sectional study. Administration and Policy in Mental Health and Mental Health Services Research, 50(3), 427–449. https://doi.org/10.1007/s10488-022-01248-5
Funding
This work was supported by R34 DA046735 (PI: Okamoto), L60DA059132 (PI: Okamura), U54GM138062 (Pilot PI: Okamura), and Powell was supported in part by the NIMH through R25MH080916.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Okamura and Palafu. The first draft of the manuscript was written by Okamura, and all authors commented on versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Ethical Approval
Approval was obtained from the ethics committee of the University of Hawaiʻi at Mānoa (2019-00443) and the Hawaiʻi State Department of Education (Application #RES2019023). The procedures used in this study adhere to the tenets of the Declaration of Helsinki.
Consent to Participate
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original online version of this article was revised:The original version of this article unfortunately contained mistakes. The masked text for Peer Reviewing was not replaced with the actual text in the Author Contributions and Funding sections. The corrected Author Contributions and Funding statements are given below.
Author Contributions All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Okamura and Palafu. The first draft of the manuscript was written by Okamura, and all authors commented on versions of the manuscript. All authors read and approved the final manuscript.
Funding This work was supported by R34 DA046735 (PI: Okamoto), L60DA059132 (PI: Okamura), U54GM138062 (Pilot PI: Okamura), and Powell was supported in part by the NIMH through R25MH080916.
The original article has been corrected.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Okamura, K.H., Palafu, T., An, K. et al. “Allowing Space for Voice…All Our Voices”: Understanding Ho‘ouna Pono Implementation Through Educational Leadership Perspectives in Rural Hawai‘i Schools. School Mental Health (2024). https://doi.org/10.1007/s12310-024-09660-y
Accepted:
Published:
DOI: https://doi.org/10.1007/s12310-024-09660-y