Abstract
Many neuroprotective and other therapies for treatment of acute ischemic stroke have failed in translation to human studies, indicating a need for more rigorous, multidimensional quality assessment of the totality of preclinical evidence supporting a therapy prior to conducting human trials. A consensus panel of stroke preclinical model and human clinical trial experts assessed candidate items for the translational readiness scale, compiled from prior instruments (STAIR, ARRIVE, CAMARADES, RoB 2) based on importance, reliability, and feasibility. Once constructed, the tool was applied by two independent raters to four current candidate acute stroke therapies, including two pharmacologic agents [nerinetide and trans-sodium crocetinate] and two device interventions [cathodal transcranial direct current stimulation and fastigial nucleus stimulation]. The Preclinical evidence of Readiness In stroke Models Evaluating Drugs and Devices (PRIMED2) assessment tool rates the totality of evidence available from all reported preclinical animal stroke model studies in 11 domains related to diversity of tested animals, time windows, feasibility of agent route of delivery, and robustness of effect magnitude. Within each content domain, clearly operationalized rules assign strength of evidence ratings of 0–2. When applied to the four assessed candidate agents, inter-rater reliability was high (kappa = 0.88), and each agent showed a unique profile of evidentiary strengths and weaknesses. The PRIMED2 assessment tool provides a multidimensional assessment of the cumulative preclinical evidence for a candidate acute stroke therapy on factors judged important for successful basic-to-clinical translation. Further evaluation and refinement of this tool is desirable to improve successful translation of therapies for acute stroke.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Many neuroprotective and other therapies for treatment of acute ischemic stroke have failed in translation despite demonstrating apparent preclinical evidence of efficacy [1,2,3,4,5]. These failures have been attributed to deficiencies in the design and conduct of both preclinical studies and of human clinical trials. For preclinical studies, identified methodologic weaknesses in study quality include absence of randomization, lack of blinded assessment of outcome, failure to prespecify a primary endpoint, use of clinically unattainable treatment times, and publication bias [6,7,8,9].
To address these deficiencies, consensus groups have developed two categories of methods to assess the quality of preclinical acute stroke studies to increase the odds of translational success. The first category encompasses tools that only or primarily assess the quality of individual preclinical studies. These include (1) the Stroke Therapy Academic Industry Roundtable (STAIR) criteria; (2) the Animal Research: Reporting In Vivo Experiments (ARRIVE); and (3) the Preclinical Risk of Bias (P-RoB) assessment tool. The second category is tools that assess the quality of meta-analyses of preclinical studies. The leading example is the Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies (CAMARADES) checklist. These tools have had a substantial beneficial effect on the preclinical literature, with the quality of studies improving over time [10].
However, in addition to these tools, there is a recognized need for an instrument that would complement and extend them by more broadly assessing a treatment’s readiness for translation from preclinical studies to initial human trials, as was highlighted by discussions at the 2020 STAIR XI meeting (October 2020 virtual conference) [11]. Such a tool would rate the strength of the cumulative evidence across all studies in the literature with regard to key aspects needed for treatment success, including effect reproducibility; studies across animals of different ages, sex, species, and comorbidities; presence of a dose–response effect; evaluation of feasible time windows and routes of delivery; and assessment of the magnitude of demonstrated beneficial imaging and behavioral effects. Ideally, the tool would also be designed for rapid assimilation both numerically and figurally. To address this need, we developed a novel assessment tool: the Preclinical evidence of Readiness In stroke Models Evaluating Drugs and Devices (PRIMED2) translational assessment rating system.
Methods
The assessment tool was developed through consensus-building discussions, tool drafting, and iterative review and revision by the author group. The consensus panel was selected to include both experts in preclinical stroke models and experts in human stroke clinical trials.
To identify potential topic domains to be analyzed by the instrument, first, existing content items were compiled from three existing preclinical study assessments (STAIR criteria, ARRIVE, CAMARADES) and two existing assessments of human clinical trial quality—the Consolidated Standards of Reporting Trials (CONSORT) checklist and the Cochrane Collaboration Risk of Bias 2 (RoB 2) assessment tool. Experts were then asked to identify additional candidate topic domains not covered by the compilation from existing instruments. From this extended topic list, the group then identified a final set of domains to be assessed based on judgment of each domain’s importance, reliability, and feasibility. Importance is defined as the value of the domain in showing a genuine biological benefit of an agent. Reliability is defined as the capacity of the domain to be congruently measured by independent raters. Feasibility is defined based on the availability of the data for detailed review.
Next, within each content domain, strength of evidence ratings of 0–2 were developed with clearly operationalized rules for assigning strength scores. For visual presentation, the color coding of red, yellow, and green representing low, intermediate, and high strength was adapted from the RoB 2 tool scoring format.
To assess tool performance, the PRIMED2 tool was then applied by two independent raters to rate the cumulative preclinical literature regarding four candidate acute stroke therapies currently in development. Disagreements between raters were resolved by consensus discussion and inter-rater reliability assessed with the kappa statistic. The two pharmacologic interventions analyzed were nerinetide (NA1), a post-synaptic density-95 protein inhibitor; and trans-sodium crocetinate (TSC), an oxygen diffuser enhancer [12,13,14,15,16,17,18,19,20]. The two device interventions analyzed were cathodal transcranial direct current stimulation (C-tDCS), direct current applied to ischemic hemisphere with anti-excitatory and collateral enhancing effects; and deep cerebellar fastigial nucleus stimulation (FNS), activating cerebral blood flow enhancing and neuroprotective central cholinergic pathways [21,22,23,24,25,26,27,28]. These interventions were chosen from among agents in current development to assess applicability of the scale to diverse types of intervention. Accordingly, two of the treatments selected were drug agents and two were device interventions. In addition, the pharmacologic agents analyzed were chosen to have different mechanisms of action and the device interventions were selected to include an approach involving direct stimulation of the ischemic zone and an approach involving focal stimulation remote from the ischemic zone. Lastly, as a positive control, we also wished to include among the 4 assessed interventions a treatment that had successfully transitioned from demonstrated preclinical efficacy to demonstrated clinical efficacy to use in clinical practice [29]. However, unfortunately, no such agent has yet completed the full development pathway. We, therefore, selected, as one of the pharmalogic treatments, the currently investigated agent that has shown the strongest signal of potential beneficial effect in initial, completed human trials (nerinetide).
Data sharing is not applicable to this article as no datasets were generated during the current study.
Results
After compilation and generation of candidate domains followed by prioritization based on importance, reliability, and feasibility, 11 assessment items were selected for final inclusion. The items characterize the diversity of animals in which benefit was demonstrated (4 items—age, sex, species, comorbidities); the time windows in which benefit was demonstrated (2 items—number of time epochs and attainability of times in clinical settings); the feasibility of agent route of delivery (1 item); and the robustness of effect magnitude (3 items—infarct volume, behavioral outcomes, and dose response).
With regard to operationalized rules for assigning scores of 0–2, four of the items had dichotomous responses and were assigned possible scores of 0 or 2. Seven of the items had graded responses and were assigned possible scores of 0, 1, and 2. For eight items, the operationalized rules could be stated clearly with brief phrases. For three items of treatment time epoch, feasible time window, and feasible route of delivery, more detailed text guidance was judged needed.
For the treatment timing item, five epochs were delineated based on the timing relation of treatment administration to start and stop of ischemia induction: (1) pre-ischemia onset administration (treatment start prior to or concurrent with ischemic period onset); (2) transient ischemia, post-onset administration (treatment starts after transient ischemia onset and treatment continuation throughout the ischemia period; (3) permanent ischemia, post-onset administration (treatment starts after permanent ischemia onset); (4) transient ischemia, post-reperfusion administration (treatment starts after the end of ischemic period). Absence of benefit in any time epoch is scored 0, presence of benefit in one time epoch scored 1, and presence of benefit in two or more time epochs scored 2.
For the feasible time window item, a feasible time window is defined as 45 min or more from ischemia onset (scored 2) and an infeasible time window as less than 45 min (scored 0). The 45-min cutpoint was used for the feasibility of time window definition because it is the earliest start time of therapy that has been achieved in a pivotal human acute stroke clinical trial [30].
To classify the effect size of observed infarct reductions, meta-analytic Cohen’s d values (preferred) or cross-study median Cohen’s d values (less preferred but more easily performed) are employed. Effect sizes are categorized as (a) small: Cohen’s d = 0.2–0.39; (b) medium: Cohen’s d = 0.4–0.69; or (c) large: Cohen’s d ≥ 0.7 [31, 32].
The separate scoring of each of the 11 PRIMED2 elements and their visual-figural display is the primary component of the instrument, arraying agent evidence along multiple dimensions. An optional summary score totaling all component items is provided, but with caution as it fails to convey the granular information of each scale item.
In applying the assessment tool to the preclinical literature regarding the four exemplar therapies, inter-rater reliability showed a kappa statistic of 0.88. The PRIMED2 scores and figural output for all four agents are shown in Table 1. For two items, treatment time epoch and infarct volume reduction magnitude, high scores of 2 were achieved by all four agents. Items with three agents rating at high scores of 2 were age of animals, reproducibility across species and laboratories, and feasible time window. Conversely, for one item, sex of animals, low scores of 0 were present for three of the four agents. For another item, baseline comorbidities, all four agents showed either low 0 or intermediate 1 scores. Each of the four agents showed distinctive profiles of evidence of readiness. High and moderately high total readiness for translation scores were achieved by the two pharmacologic agents. The two device interventions achieved low (deep FNS) and high (C-tDCS) intermediate scores.
Discussion
The Preclinical evidence of Readiness In stroke Models Evaluating Drugs and Devices (PRIMED2) is a novel multidimensional tool assessing experimental acute stroke agent readiness to proceed to human clinical trial testing. For a candidate agent, the assessment instrument delineates the strength of the collective evidence from all preclinical studies in the literature in 11 domains judged important for translational success. These key domains have the capacity to be reliably rated and to be feasibly abstracted from published studies.
The PRIMED2 has several strengths. It is a multidimensional assessment evaluating the demographic, clinical, and species diversity of animals in which benefit has been shown. Also, the tool uses explicit, operationalized criteria to assign evidential strength ratings in each assessed domain. Furthermore, like the Cochrane RoB 2 instrument for human clinical trials, the PRIMED2 provides a visual-figural profile of evidence for an agent that is rapidly assimilable by readers and it is applicable to both drugs and device neuroprotective therapies. In addition, the instrument assesses magnitude of treatment effect with Cohen’s d, a standardized, unit-less effect size metric that accounts for study sample size and variability, and allows for the harmonization of studies that report absolute volume reduction and studies that report relative volume reduction [31, 32].
When applied to four current acute stroke agents in development, the PRIMED2 showed high inter-rater reliability. The instrument has good capacity to delineate evidence profiles, displaying unique patterns of item evidence for each of the four interventions. The PRIMED2 additionally demonstrated absence of floor and ceiling effects and good discriminative validity, assigning the four agents a wide range of overall scores.
The PRIMED2 tool differs from and complements available widely employed rating tools to assess preclinical stroke studies. Compared with the STAIR checklist, the PRIMED2 focuses solely upon rating the cumulative findings across all available studies, rather than a mix of items for rating individual studies and cumulative findings. It also provides a formal, operationalized, graded scoring system, whereas the STAIR checklist characterizes items qualitatively and somewhat dichotomously. These same features differentiate PRIMED2 from the ARRIVE checklist, with added distinction that PRIMED2 incorporates items specific to the acute stroke disease state while ARRIVE is a generic instrument with items applicable to all disease states and no stroke-specific domains. While the CAMARADES checklist is also a whole-literature assessment like PRIMED2, the current instrument incorporates items specific to the acute stroke disease state while CAMARADES is a generic instrument with items applicable to all disease states and no stroke-specific domains.
We have also provided an overall quantitative summary score option for the PRIMED2, but this total value should be used with caution as it can obscure the important granular information contained in the individual items. Final translational decision-making requires a detailed assessment of all the domains and their elements. A detailed assessment of all the tool’s domains particularly applies to the therapies that score in the intermediate readiness for translation range. The total score and individual items may also fail to reflect aspects of a therapy not assessable in animal models. For example, in this study, fastigial nucleus stimulation did not receive a high feasible route of delivery item score because all animal studies were performed by implantation of depth electrodes, an unattractive approach for human studies. However, in humans, non-invasive stimulation of deep structures is feasible, e.g., with focused ultrasound.
This study has limitations. The expert panel, though diverse in expertise, came from two institutions. Evaluation, and potentially revision, of the PRIMED2 by a larger, international consensus group is desirable. The PRIMED2 was applied to four agents currently in development; application to a broader set of interventions would be advantageous. The component items of the PRIMED2 tool are based on expert judgment of properties likely to be important to the successful clinical translation of acute stroke therapies, rather than empirically determined properties, as few agents have been successfully translated to date.
Conclusion
The PRIMED2 assessment tool provides a multidimensional assessment of the cumulative preclinical evidence for a candidate acute stroke therapy on factors judged important for successful basic-to-clinical translation. Further evaluation and refinement of this tool is desirable to improve successful translation of therapies for acute stroke.
Data Availability
Not applicable.
Code Availability
Not applicable.
References
Nerses Sanossian JLS. Neuroprotection for acute brain ischaemia. In: Hankey JSG, editor. Stroke prevention and treatment: an evidence-based approach. Cambridge University Press; 2020. p. 214–38.
Schmidt-Pogoda A, Bonberg N, Koecke MHM, Strecker JK, Wellmann J, Bruckmann NM, et al. Why most acute stroke studies are positive in animals but not in patients: a systematic comparison of preclinical, early phase, and phase 3 clinical trials of neuroprotective agents. Ann Neurol. 2020;87(1):40–51. https://doi.org/10.1002/ana.25643.
Dhir N, Medhi B, Prakash A, Goyal MK, Modi M, Mohindra S. Pre-clinical to clinical translational failures and current status of clinical trials in stroke therapy: a brief review. Curr Neuropharmacol. 2020;18(7):596–612. https://doi.org/10.2174/1570159X18666200114160844.
Grupke S, Hall J, Dobbs M, Bix GJ, Fraser JF. Understanding history, and not repeating it. Neuroprotection for acute ischemic stroke: from review to preview. Clin Neurol Neurosurg. 2015;129:1–9. https://doi.org/10.1016/j.clineuro.2014.11.013.
O’Collins VE, Macleod MR, Cox SF, Van Raay L, Aleksoska E, Donnan GA, et al. Preclinical drug evaluation for combination therapy in acute stroke using systematic review, meta-analysis, and subsequent experimental testing. J Cereb Blood Flow Metab. 2011;31(3):962–75. https://doi.org/10.1038/jcbfm.2010.184.
Lapchak PA, Zhang JH, Noble-Haeusslein LJ. RIGOR guidelines: escalating STAIR and STEPS for effective translational research. Transl Stroke Res. 2013;4(3):279–85. https://doi.org/10.1007/s12975-012-0209-2.
Ford GA. Clinical pharmacological issues in the development of acute stroke therapies. Br J Pharmacol. 2008;153(Suppl 1):S112–9. https://doi.org/10.1038/sj.bjp.0707654.
Ginsberg MD. Current status of neuroprotection for cerebral ischemia: synoptic overview. Stroke. 2009;40(3 Suppl):S111–4. https://doi.org/10.1161/STROKEAHA.108.528877.
Liebeskind DS, Kasner SE. Neuroprotection for ischaemic stroke: an unattainable goal? CNS Drugs. 2001;15(3):165–74. https://doi.org/10.2165/00023210-200115030-00001.
Minnerup J, Zentsch V, Schmidt A, Fisher M, Schabitz WR. Methodological quality of experimental stroke studies published in the Stroke Journal: time trends and effect of the basic science checklist. Stroke. 2016;47(1):267–72. https://doi.org/10.1161/STROKEAHA.115.011695.
STAIR CONSENSUS CONFERENCE-CYBER STAIR XI. 2020. https://www.thestair.com/conference-summary-image. Accessed January 25 2021.
Lapchak PA. Efficacy and safety profile of the carotenoid trans sodium crocetinate administered to rabbits following multiple infarct ischemic strokes: a combination therapy study with tissue plasminogen activator. Brain Res. 2010;1309:136–45. https://doi.org/10.1016/j.brainres.2009.10.067.
Manabe H, Okonkwo DO, Gainer JL, Clarke RH, Lee KS. Protection against focal ischemic injury to the brain by trans-sodium crocetinate Laboratory investigation. J Neurosurg. 2010;113(4):802–9. https://doi.org/10.3171/2009.10.JNS09562.
Wang Y, Yoshimura R, Manabe H, Schretter C, Clarke R, Cai Y, et al. Trans-sodium crocetinate improves outcomes in rodent models of occlusive and hemorrhagic stroke. Brain Res. 2014;1583:245–54. https://doi.org/10.1016/j.brainres.2014.08.013.
Deng J, Xiong L, Zuo Z. Trans-sodium crocetinate provides neuroprotection against cerebral ischemia and reperfusion in obese mice. J Neurosci Res. 2015;93(4):615–22. https://doi.org/10.1002/jnr.23522.
Ballarin B, Tymianski M. Discovery and development of NA-1 for the treatment of acute ischemic stroke. Acta Pharmacol Sin. 2018;39(5):661–8. https://doi.org/10.1038/aps.2018.5.
Aarts M, Liu Y, Liu L, Besshoh S, Arundine M, Gurd JW, et al. Treatment of ischemic brain damage by perturbing NMDA receptor- PSD-95 protein interactions. Science. 2002;298(5594):846–50. https://doi.org/10.1126/science.1072873.
Cook DJ, Teves L, Tymianski M. Treatment of stroke with a PSD-95 inhibitor in the gyrencephalic primate brain. Nature. 2012;483(7388):213–7. https://doi.org/10.1038/nature10841.
Teves LM, Cui H, Tymianski M. Efficacy of the PSD95 inhibitor Tat-NR2B9c in mice requires dose translation between species. J Cereb Blood Flow Metab. 2016;36(3):555–61. https://doi.org/10.1177/0271678X15612099.
Sun HS, Doucette TA, Liu Y, Fang Y, Teves L, Aarts M, et al. Effectiveness of PSD95 inhibitors in permanent and transient focal ischemia in the rat. Stroke. 2008;39(9):2544–53. https://doi.org/10.1161/STROKEAHA.107.506048.
Zhang F, Iadecola C. Fastigial stimulation increases ischemic blood flow and reduces brain damage after focal ischemia. J Cereb Blood Flow Metab. 1993;13(6):1013–9. https://doi.org/10.1038/jcbfm.1993.127.
Yamamoto S, Golanov EV, Reis DJ. Reductions in focal ischemic infarctions elicited from cerebellar fastigial nucleus do not result from elevations in cerebral blood flow. J Cereb Blood Flow Metab. 1993;13(6):1020–4. https://doi.org/10.1038/jcbfm.1993.128.
Reis DJ, Kobylarz K, Yamamoto S, Golanov EV. Brief electrical stimulation of cerebellar fastigial nucleus conditions long-lasting salvage from focal cerebral ischemia: conditioned central neurogenic neuroprotection. Brain Res. 1998;780(1):161–5.
Reis DJ, Berger SB, Underwood MD, Khayata M. Electrical stimulation of cerebellar fastigial nucleus reduces ischemic infarction elicited by middle cerebral artery occlusion in rat. J Cereb Blood Flow Metab. 1991;11(5):810–8. https://doi.org/10.1038/jcbfm.1991.139.
Glickstein SB, Ilch CP, Reis DJ, Golanov EV. Stimulation of the subthalamic vasodilator area and fastigial nucleus independently protects the brain against focal ischemia. Brain Res. 2001;912(1):47–59.
Glickstein SB, Golanov EV, Reis DJ. Intrinsic neurons of fastigial nucleus mediate neurogenic neuroprotection against excitotoxic and ischemic neuronal injury in rat. J Neurosci. 1999;19(10):4142–54.
Peruzzotti-Jametti L, Cambiaghi M, Bacigaluppi M, Gallizioli M, Gaude E, Mari S, et al. Safety and efficacy of transcranial direct current stimulation in acute experimental ischemic stroke. Stroke. 2013;44(11):3166–74. https://doi.org/10.1161/STROKEAHA.113.001687.
Notturno F, Pace M, Zappasodi F, Cam E, Bassetti CL, Uncini A. Neuroprotective effect of cathodal transcranial direct current stimulation in a rat stroke model. J Neurol Sci. 2014;342(1–2):146–51. https://doi.org/10.1016/j.jns.2014.05.017.
Hill MD, Goyal M, Menon BK, Nogueira RG, McTaggart RA, Demchuk AM, et al. Efficacy and safety of nerinetide for the treatment of acute ischaemic stroke (ESCAPE-NA1): a multicentre, double-blind, randomised controlled trial. Lancet. 2020;395(10227):878–87. https://doi.org/10.1016/S0140-6736(20)30258-0.
Saver JL, Starkman S, Eckstein M, Stratton SJ, Pratt FD, Hamilton S, et al. Prehospital use of magnesium sulfate as neuroprotection in acute stroke. N Engl J Med. 2015;372(6):528–36. https://doi.org/10.1056/NEJMoa1408827.
Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, N.J.: L. Erlbaum Associates; 1988.
Schäfer T, Schwarz MA. The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology. 2019;10. https://doi.org/10.3389/fpsyg.2019.00813.
Funding
Partial financial support was received from American Heart Associaton Career Development Award.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Mersedeh Bahr-Hosseini (MBH) receives funding from the American Heart Associaton for developing transcranial direct current stimulation as neuroprotection for treatment of acute cerebral ischemia. MBH has an intellectual property on Transcranial Electrical Stimulation In Stroke Early After Onset.
Marom Bikson (MB): The City University of New York holds patents on brain stimulation with MB as inventor. MB has equity in Soterix Medical Inc. MB consults, received grants, assigned inventions, and/or serves on the SAB of Boston Scientific, GlaxoSmithKline, Biovisics, Mecta, Halo Neuroscience, X. MB is supported by grants from the National Institutes of Health: R01NS101362 (MB), R01NS095123 (MB), R01NS112996 (MB), R01MH111896 (MB), R01MH109289 (MB).
Jeffrey L Saver (JLS) receives consulting fees for service on clinical trial design committees from Astrocyte Pharma and Hybernia Medical.
All the other authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Bahr-Hosseini, M., Bikson, M., Iacoboni, M. et al. PRIMED2 Preclinical Evidence Scoring Tool to Assess Readiness for Translation of Neuroprotection Therapies. Transl. Stroke Res. 13, 222–227 (2022). https://doi.org/10.1007/s12975-021-00922-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12975-021-00922-4