Abstract
There is growing interest in regulatory use of randomized pragmatic trials and noninterventional real-world (RW) studies of effectiveness and safety, but there is no agreed-on framework for assessing when this type of evidence is sufficiently reliable. Rather than impose a clinical trial-like paradigm on RW evidence, like blinded treatments or complete, source-verified data, the framework for assessing the utility of RW evidence should be grounded in the context of specific study objectives, clinical events that are likely to be detected in routine care, and the extent to which systematic error (bias) is likely to impact effect estimation. Whether treatment is blinded should depend on how well the outcome can be measured objectively. Qualification of a data source should be based on (1) numbers of patients of interest available for study; (2) if “must-have” data are likely to be recorded, and if so, how and where; (3) the accessibility of systematic follow-up data for the time period of interest; and (4) the potential for systematic errors (bias) in data collection and the likely magnitude of any such bias. Accessible data may not be representative of an entire population, but still may provide reliable evidence about the experience of typical patients treated under conditions of conventional care. Similarly, RW data that falls short of optimal length of follow-up or study size may still be useful in terms of its ability to provide evidence for regulators for subgroups of special interest. Developing a framework to qualify RW evidence in the context of a particular study purpose and data asset will enable broader regulatory use of RW data for approval of new molecular entities and label changes. Reliable information about diverse populations and settings should also help us move closer to more affordable, effective health care.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Gottlieb S. Speech to Regulatory Affairs Professionals Society 2017 Regulatory Conference; September 11, 2017; Washington DC.
Gottlieb S. Written remarks provided at the conference “Examining the impact of real-world evidence on medical product development.” National Academy of Sciences, Washington DC; September 20, 2017.
US Public Law No: 114-255.
Anderson ML, Griffin J, Goldkind SE, et al. The Food and Drug Administration and pragmatic clinical trials of marketed products. Clin Trials 2015;12:511–519.
Cerrata F, Eichler HG, Rasi G. Drug policy for an aging population: The European Medicines Agency’s Geriatric Medicines Strategy. N Engl J Med. 2012:36;1972–1974.
Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology. 1990;1:43–46.
The SPRINT Research Group. A randomized trial of intensive versus standard blood-pressure control. N Engl J Med. 2015; 373:2103–2116.
NEJM. The Sprint Data Analysis Challenge. https://challenge.nejm.org/pages/about. Accessed September 17, 2017.
Burns NS, Miller PW. Learning what we didn’t know—the SPRINT Data Analysis Challenge. N Engl J Med. 2017;376:2205–2207.
Berger M, Daniel G, Frank K, et al. A framework for regulatory use of real-world evidence. White paper prepared by the Duke Margolis Center for Health Policy, September 13, 2017.
Jarow JP, LaVange L, Woodcock J. Multidimensional evidence generation and FDA regulatory decision making. Defining and using “real-world” data. JAMA 2017;318:703–704.
US Food and Drug Administration. FDA D.I.S.C.O.: Avelumab in Merkel cell carcinoma transcript. https://www.fda.gov/Drugs/InformationOnDrugs/ApprovedDrugs/ucm559876.htm. Accessed March 2, 2018.
Broderick JM. Avelumab approved in Europe for Merkel cell carcinoma. OncLive. http://www.onclive.com/web-exclusives/avelumab-approved-in-europe-for-merkel-cell-carcinoma. Published September 21, 2017.
Cowey CL, Mahnke L, Espirito J, Helwig C, Oksen D, Bharmal M. Real-world treatment outcomes in patients with metastatic Merkel cell carcinoma following treated with chemotherapy in USA. Future Oncol. 2017;13:1699–1710.
Becker JC, Lorenz E, Ugruel S, et al. Evaluation of real-world treatment outcomes in patients with distant metastatic Merkel cell carcinoma following second-line chemotherapy in Europe. Oncotarget. 2017;8:79731–79741.
European Medicines Agency. New cell-based therapy to support stem cell transplantation in patients with high-risk blood cancer [Press Release]. 2016, June 24. Retrieved from http://www.ema.europa.eu/ema/index.jsp?curl=pages/news_and_events/news/2016/06/news_detail_002555.jsp&mid=WC0b01ac058004d5c1.
Alexion Receives Approval for Important Updates to the European Label for Soliris (eculizumab). Business Wire. http://www.businesswire.com/news/home/20150409006360/en/Alexion-Receives-Approval-Important-Updates-European-Label. Accessed October 29, 2017.
Mack C, Pavesio A, Kelly K, et al. Making the most of external comparators. A study of fracture healing in patients at risk of non-union. Poster presented at: International Society of Pharmacoeconomics & Outcomes Research, May 2017.
Velentgas P, Dreyer NA, Nourjah P, Smith SR, Torchia MM, eds. Developing a Protocol for Observational Comparative Effectiveness Research. A User’s Guide. Rockville, MD: Agency for Healthcare Research and Quality; 2013. AHRQ Publication No. 12(13)-EHC099.
US Food and Drug Administration. Promoting safe and effective drugs for 100 years. http://www.fda.gov/AboutFDA/WhatWeDo/History?ProductRegulation/PromotionSafetandEffectiveDrugsfor100Years/.
Woodcock A, Bakerly ND, New JP, et al. The Salford Lung Study protocol: a pragmatic, randomised phase III real-world effectiveness trial in asthma. BMC Pulm Med. 2015;15:160.
New JP, Bakerly ND, Leather D, Woodcock A. Obtaining real-world evidence: the Salford Lung Study. Thorax. 2014;69:1152–1154.
Diuretic Comparison Project. https://clinicaltrials.gov/ct2/show/NCT02185417. Accessed 9 November 2017.
Tinnerman A, Geuter S, Sprenger C, Finsterbusch J, Büchel C. Interactions between brain and spinal cord mediate value effects in nocebo hyperalgesia. Science. 2017;358:105–108.
FDA Guidance for Industry. Diabetes Mellitus—Evaluating cardiovascular risk in new antidiabetic therapies to treat type 2 diabetes. December 2008.
Hlatky MA, Ray RM, Burwen DR, et al. Use of Medicare data to identify coronary heart disease outcomes in the Women’s Health Initiative. Circ Cardiovasc Qual Outcomes. 2014;7:157–162.
Rosenfeld M. An overview of endpoints for cystic fibrosis. One size does not fit all. Proc Am Thorac Soc. 2007;4:299–301.
Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. 3rd ed., 2 vols. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) Rockville, MD: Agency for Healthcare Research and Quality; 2014. AHRQ Publication No. 13(14)-EHC111.
Bourke A, Bate A, Sauer BC, Brown JS, Hall GC. Evidence generational from healthcare databases: recommendations for managing change. Pharmacoepidemiol Drug Saf. 2016;25:749–754.
Dreyer NA, Schneeweiss S, McNeil B, et al. GRACE principles: recognizing high-quality observational studies of comparative effectiveness. Am J Manag Care. 2010;16:467–471.
Dreyer NA, Velentgas P, Westrich K, Dubois R. The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution. J Manag Care Spec Pharm. 2014;20:301–308.
Center for Drug Evaluation and Research Approval Package for Application No. 761049Orig1s000 for Bavencio Injection, Approval Date March 23, 2017.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Dreyer, N.A. Advancing a Framework for Regulatory Use of Real-World Evidence: When Real Is Reliable. Ther Innov Regul Sci 52, 362–368 (2018). https://doi.org/10.1177/2168479018763591
Published:
Issue Date:
DOI: https://doi.org/10.1177/2168479018763591