Abstract
Background
The National Institute for Health and Care Excellence (NICE) may approve health technologies on condition of more evidence generated only in research (OiR) or only with research (OwR). NICE specifies the information needed to comply with its request, although it may not necessarily guarantee good quality and timely evidence for re-appraisal, before reaching a final decision.
Aim
This study aimed to critically appraise the methods, quality and risk of bias of evidence generated in response to NICE OiR and OwR technology appraisal (TA) and highly specialised technologies (HSTs) recommendations.
Methods
NICE TAs (between March 2000 and September 2020) and HST evaluations (to October 2023) of medicines were reviewed. Conditional recommendations were analysed to identify the evidence requested by NICE for re-appraisal. The new evidence was analysed for compliance with NICE’s request and assessed using the Cochrane Collaboration’s tools for risk of bias in randomised trials and the ROBINS-I tool for non-randomised evidence.
Results
NICE made 54 conditional recommendations from TAs (13 OiR and 41 OwR) and five conditional recommendations for HSTs (all OwR). Of these, 16 TAs presented additional evidence for re-appraisal (9 OiR [69%] and 7 OwR [17%]) and three HSTs (3 OwR [60%]). Two of the nine re-appraised TAs with OiR recommendation and four of the seven OwR complied fully with NICE’s request for further evidence, while all three from the HSTs complied. The majority of re-appraised TAs and HSTs included evidence that was deemed to be at serious, high, moderate or unclear risk of bias. Among the 26 randomised controlled trials from TAs assessed, eight were categorised as having low risk of bias in all domains and ten had at least one domain as a high risk of bias. Reporting was unclear for the remainder. Twenty-two non-randomised studies, primarily single-arm studies, were susceptible to biases mostly due to the selection of participants and to confounding. Two HSTs provided evidence from randomised controlled trials which were classified as unclear or high risk of bias. All non-randomised evidence from HSTs were categorised as moderate or serious risk of bias.
Conclusions
There is widespread non-compliance with agreed data requests and important variation in the quality of evidence submitted in response to NICE conditional approval recommendations. Quality standards ought to be stipulated in respect to evidence contributing to re-appraisals following NICE conditional approval recommendations.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
There is widespread non-compliance with agreed data requests following conditional NICE technology appraisal recommendations. |
The majority of evidence submitted for re-appraisal was judged to be at serious, high, moderate or unclear risk of bias. |
Quality standards ought to be stipulated in respect to evidence contributing to re-appraisals following conditional NICE approval recommendations. |
1 Introduction
Pharmaceutical purchasing decisions have historically been based on prices set by their manufacturers and the evidence available at the time of application for reimbursement [1]. More recently, mechanisms for approving medicines conditional on further evidence being generated has gained traction as a mechanism for earlier patient access to new and sometimes innovative medicines [2,3,4,5,6,7]. Approaches to conditional approval include only in research (OiR) and only with research (OwR) recommendations; with the main distinction being that in OiR, coverage is limited to participants enrolled in research studies, whereas the whole eligible population is covered following an OwR recommendation, but with pre-specified data collected [1, 4, 8].
The National Institute for Health and Care Excellence (NICE) has always had the option to recommend health technologies OiR where there is uncertainty in clinical parameters (along with ‘recommended’, ‘optimised’ [that is, recommended for a sub-group of patients for whom the medicine is clinically indicated] and ‘not recommended’ appraisal decision outcomes [9]). OiR recommendations were more common in the early years of the NICE appraisal process, contrasting with OwR which has become more commonplace following provisions for performance-based risk-sharing arrangements in the Pharmaceutical Price Regulation Scheme and more recently the Voluntary Scheme for Branded Medicines Pricing and Access. Most notably, OwR recommendations have featured extensively in managed access agreements (MAAs) for Cancer Drugs Fund (CDF) medicines, a trend set to continue with the Innovative Medicines Fund for England. MAAs consist of two main components—a data collection arrangement to address clinical uncertainty and a commercial agreement that controls drug costs whilst data are collected [10]. Medicines included in the CDF or the Innovative Medicines Fund list are funded for a specified period and considered for routine commissioning upon review of additional data in a subsequent appraisal.
One source of concern about NICE’s conditional approval process is the quality and reliability of the methods of evidence generation, analysis and reporting [3, 11]. Morrell et al. [12] identified the most common sources of uncertainty among a sample of technology appraisals (TAs) of cancer medicines as being the immaturity of survival data and issues relating to comparators. They commented that these could not be readily resolved by real-world data collection where there are no ongoing trials to provide longer-term data. However, subsequent reviews of cancer medicines appraised by NICE found more extensive use of later follow-up data from clinical trials than from real-world data for addressing uncertainty related to survival estimates [13, 14].
A second area of concern is the increasing focus on OwR in preference to OiR. While the research following OiR recommendation is predominantly funded by manufacturers, OwR incurs a cost to the NHS. The relative value of both approaches ought to be established when conditional approval recommendations are made. Central to this is the need to consider explicitly the health gain foregone by patients denied access to a medicine until the research is completed (OiR) and the health gain foregone by other patients when a medicine is reimbursed that is subsequently found to be insufficiently effective (OwR) [15].
NICE’s conditional approval mechanisms facilitate patient access to specific categories of medicines based on immature evidence and lower certainty around their clinical and cost effectiveness. Providing access to ineffective treatments or treatments with questionable evidence imposes an opportunity cost in terms of impacting on access to medicines and services in other areas of the NHS [16]. The drive to expedite patient access has also led to a wider acceptance of inferior evidence based on non-randomised research [17]. While the collection of further evidence can reduce the uncertainty of re-appraisal decisions, the reversibility of such decisions is potentially limited, although in practice treatment would continue for responders, and not initiated for new patients.
Recognising the limitations of conditional approval, and the variable nature of the evidence available for re-appraisal, the aim of this study was to critically appraise the methods, quality and risk of bias of evidence generated in response to OiR and OwR recommendations by NICE in TAs and evaluations of highly specialised technologies (HSTs).
2 Methods
2.1 Data Sources
All medicines appraised by NICE between March 2000 and September 2020 were identified from a published list of TAs [18] and HSTs up to October 2023 [19]. The final appraisal determinations (or if unavailable, the assessment reports) for TAs and the final evaluation determinations for HSTs were reviewed for whether they were approved conditionally (OiR or OwR). Relevant committee papers and MAAs (where applicable) were retrieved from NICE’s website or, in the case of historical TAs (which had been deleted from the website), from the UK Web Archive online or the UK government National Web Archive, respectively (see Supplementary Table 1 in the electronic supplementary material [ESM]).
2.2 Inclusion Criteria
The review was limited to pharmacological interventions, which represent the majority of technologies with an MAA and all technologies on the CDF list. For the purposes of this review, TAs and HSTs were classified as either OiR or OwR based on the following definitions of Walker et al. [8]: “OiR—Coverage of a technology is available only to patients involved in research. OwR—A positive coverage decision is conditioned upon the collection of additional evidence to support continued, expanded or withdrawal of coverage.”
2.3 Exclusion Criteria
TAs and HSTs were excluded from the review if they were not recommended or had been terminated due to non-submission by the pharmaceutical company.
2.4 Data Extraction
A database was constructed in MS Excel to record details of each TA and HST, including the name of the medicine, clinical indication, NICE recommendation, CDF and MAA status and whether the recommendation was OiR or OwR. Data were extracted from NICE TA and HST guidances on the areas of uncertainty highlighted by the appraisal committees and the recommendations made for further research (study type and data sources). MAAs were also checked to identify any agreement or recommendation that may not have been mentioned in the TA or HST.
Medicines that were re-appraised after conditional approval were identified and the TAs or HSTs reviewed for reference to the new evidence generated. The original (X) and re-appraised (Y) TAs (HSTs) are referenced as TAX → TAY (HSTX → HSTY). Further information about the evidence was then identified from published journal articles and NICE committee papers. Data were subsequently extracted on the source of the clinical evidence (e.g. trials, audits, registries, experts), the research design (randomised controlled trials [RCTs], observational studies, opinion) and methods (randomisation, blinding, consideration of confounding, outcome reporting, data missingness and adherence to intervention). The principal findings of the additional research studies were summarised.
2.5 Analysis
The risk of bias from randomised evidence identified for re-appraised medicines was analysed using the Cochrane Collaboration’s tool for assessing risk of bias in randomised trials [20]. The tool considers six domains of bias: selection bias, performance bias, detection bias, attrition bias, reporting bias and other biases. Each domain of bias could be categorised as low risk, high risk, or unclear risk.
Non-randomised evidence was analysed using the Cochrane Collaboration’s Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool [21], which considers seven domains of bias: two pre-intervention (confounding, selection of participants), one at intervention (classification of intervention) and four post-intervention (deviations from intended interventions, missing data, measurement of outcomes and selection of the reported result). Each domain could be categorised as low, moderate, serious, or critical risk, or no information provided.
Risk of bias assessments were undertaken by YPP and reviewed by DAH.
3 Results
3.1 Included Appraisals
For the period of analysis, NICE conducted 651 TAs and made 968 individual recommendations. Six hundred TAs were of medicines, from which 88 were not recommended and 61 related to terminated appraisals. Of the remaining TAs, 53 were included for analysis. These made 54 recommendations—13 were OiR and 41 satisfied the criteria for classification as OwR. Additionally, some TAs were misclassified in NICE’s database, including TA070 which was listed as recommended, but with a condition of further data collection; TA397, which was listed as optimised and not highlighted as OiR; and TA588 which was an optimised recommendation, but linked to an MAA with conditions for further data collection.
NICE also conducted 28 HST evaluations, of which five were conditionally approved OwR. HST1 was classified as OwR because one of the conditions for approval included a research programme to evaluate when to stop the treatment, although it was the only HST conditionally approved without an MAA (see Fig. 1 and Supplementary Table 2 [in the ESM]).
Flow diagram of Technology Appraisals (TAs) and Highly Specialised Technologies (HSTs) included in the review
3.2 NICE Recommendations for Further Research
NICE specified the need to address uncertainty in clinical and cost effectiveness in all conditionally approved TAs and HSTs. Recommendations for further research included RCTs, observational studies, or a combination of both (Table 1). For example, among OiR recommendations, in TA093 the committee recommended obtaining data from well-designed (unspecified method) clinical studies and an audit, while in TA111, the committee recommended research, “preferably in the form of RCTs”, and an audit. For TAs with OwR recommendations, data were requested mainly from ongoing trials, and from the Systemic Anti-Cancer Therapy (SACT) and Blueteq registries as secondary sources of information. Only TA581 recommended a new RCT, linked to a post-marketing condition for approval by the European Medicines Agency. Recommendations for all conditionally approved HSTs included the collection of auditable data through specific databases. For instance, for HST1 and HST2, NICE recommended the creation of a national registry; for HST3, data would be collected via the NorthStart database; for HST6, through a disease-specific MAA database; while for HST12 it would be collected through the creation of an Excel SharePoint repository of registering patients.
3.3 NICE Technology Appraisals (TAs) and Highly Specialised Technologies (HSTs) Subjected to Re-appraisal
Sixteen TAs that included 17 individual recommendations were subsequently re-appraised by NICE.
Nine TAs were originally recommended OiR (including TA193 with two individual recommendations) and of these, NICE requested RCT evidence for three TAs (TA033, TA037 and TA111) and further research from unspecified sources in six TAs (TA065, TA072, TA093, TA193 [for both indications], TA244 and TA467).
There were seven re-appraised TAs that were originally recommended OwR (TA070, TA416, TA446, TA447, TA465, TA472 and TA483). For these, NICE mainly requested evidence from ongoing RCTs, SACT and/or Blueteq databases, with the exception of TA070, which made reference to a national registry, an audit and “good quality studies”; and TA446 which referred to a retrospective, non-interventional trial with data from SACT and Blueteq.
Three HSTs were subjected to NICE re-appraisal. Following HST1, for example, NICE requested national monitoring systems to record the number of people diagnosed; and for HST2, NICE specified that the marketing authorisation holder had to create a patient registry accessible to NHS England.
3.3.1 Evidence Considered in Re-appraisal Following Only in Research (OiR) Recommendation
There were some inconsistencies in what research NICE requested in OiR recommendations and what was subsequently considered during re-appraisal. While two TAs complied fully with NICE’s request for further evidence (TA033 → TA093 and TA288 → TA418), four were only partially compliant (TA072 → CG79, TA093 → CG131, TA111 → TA217 and TA244 → TA461).
RCT evidence was provided in three instances following NICE recommendations (TA033 → TA093, TA111 → TA217 and TA244 → TA461), although RCT data were considered in three other re-appraisals: TA072 → CG79, TA093 → CG131 and TA288 → TA418 (Table 2). However, while TA461 included evidence from two RCTs, there were no data on the comparison of interest that NICE specified in TA244. NICE specified audit data along with additional studies in their OiR recommendations for TA065, TA072 and TA093, but evidence from audits were not available in any re-appraisals. TA217 did not include evidence requested in TA111 of how the technology contributed to health-related quality of life.
No new evidence was presented in three re-appraisals (TA037 → TA137, TA065 and TA193). NICE commented in TA137 that it would be difficult to gather relevant information for the OiR indication in question because the medicine was approved as a third-line treatment. NICE decided TA065 was no longer relevant to clinical practice; and a review of TA193 did not find any new evidence.
3.3.2 Evidence Considered in Re-appraisal Following Only with Research (OwR) Recommendation
Of the seven TAs identified as OwR, four complied with NICE’s request for further evidence on re-appraisal (TA416 → TA653, TA446 → TA524, TA465 and TA483 → TA655) and three TAs complied partially (TA070 → TA241, TA447 → TA531 and TA472 → TA629). TA241 did not present the audit data requested and, while TA531 and TA629 included SACT data, this was immature or not considered by the committee to be sufficiently robust.
RCT evidence was available on re-appraisal for five TAs for which NICE requested such data following OwR recommendations (TA416 → TA653, TA447 → TA531, TA465, TA472 → TA629 and TA483 → TA655) (Table 2). TA653 included a survival analysis based on SACT data relating to CDF patients, in addition to the evidence from the two studies requested in TA416. For TA465, NICE requested data from an ongoing trial and from SACT, but the results from the trial indicated no survival advantage, and the TA was withdrawn. TA655 presented a combination of data from an RCT, follow-up data from a previous study and a cohort study.
NICE’s re-appraisal of TA070 in TA241 included RCT evidence in addition to non-randomised studies, but no audit. The re-appraisal of TA446 in TA524 provided the evidence requested and, in addition, a single-arm, open-label, multicentre study (Table 3).
All re-appraised HSTs complied with NICE’s requested of further evidence. In HST6 → HST23, however, while the company presented descriptive MAA data on clinical outcomes, these were not initially analysed in accordance with NICE’s request (a comparison with registry data), on the basis that there were major baseline differences between populations. A subsequent analysis of the registry data, which compared treatment naïve and experienced patients, was redacted.
3.4 Risks of Bias in Evidence Presented in Re-appraisals
Among technology re-appraisals where new data were considered, TA461 and TA465 included evidence that was all judged to be of low risk of bias, two TAs (TA217 and TA418) included at least some evidence that was of low risk of bias and 11 TAs (TA093, TA217, TA241, TA418, TA524, TA531, TA629, TA653 and TA655, including evidence from CG79, CG131) contained evidence with a critical, serious, high, moderate or unclear risk of bias.
3.4.1 Randomised Evidence
Based on the Cochrane risk of bias tool, randomised studies were susceptible mostly to biases due to a lack of blinding of participants and researchers, or blinding of outcome assessment (see Fig. 2).
Risk of bias from (A) randomised controlled trials and (B) non-randomised studies, that were considered during re-appraisal of Technology Appraisals (TAs) and Highly Specialised Technologies (HSTs) with Only in Research (OiR) or Only with Research (OwR) conditional approvals
Eight RCTs reported in four TAs (three TAs previously recommend OiR and one TA previously recommended as OwR) were categorised as having low risk of bias in all domains: TA217 [41,42,43,44], TA418 [53], TA461 [50, 51] and TA465 [63].
Eight RCTs from four TAs (all previously recommended OiR) had at least one domain categorised as unclear: TA093 [22], CG79 [25,26,27,28], CG131 [29, 30] and TA418 [52]. Selection, performance and detection bias were identified as the three main sources of risk of bias that led to this classification.
Ten RCTs from seven TAs (two previously recommended OiR and five previously recommended OwR) were categorised in at least one domain as a high risk of bias: TA093 [23, 24], TA241 [54, 55], TA217 [39, 40], TA531 [62], TA629 [64], TA653 [59] and TA655 [65]. Performance, detection and other bias were identified as the domains that contributed the most to this categorisation.
Evidence presented in HSTs was categorised as unclear or high risk. For instance, trial MOR-004 [67] in HST2 had most domains as low risk of bias, but random sequence generation and allocation concealment were unclear. Trials MOR-05 [68] in HST2 and ENB-009-10 [72] in HST6 had more than two domains classified as high risk of bias.
3.4.2 Non-randomised Evidence
There were six re-appraised TAs with evidence from 22 non-randomised studies—primarily single-arm studies, with the exception of Atri et al. [46] and Lopez et al. [49] in TA217. The evidence came from a combination of studies, including, for example, a case series [56], a phase II single-arm study [60], three phase IV or post-marketing surveillance studies that used patient-reported outcomes [45, 48, 57], a real-world UK observational study (TA524) and evidence from SACT data (TA653 and TA655) (Table 4).
Based on the ROBINS-I tool, the non-randomised studies were susceptible mostly to biases due to the selection of participants and to confounding (Fig. 2). One study (Aparicio et al. [36]) presented in CG131 (previously recommended OiR) had one domain classified as a critical risk of bias. Thirteen studies from five TAs (two previously recommended OiR and three OwR) presented evidence categorised as high risk of bias in at least one domain: TA241 [56,57,58], CG131 [31, 32, 35, 37], TA217 [45,46,47,48,49], TA524 (real-world observational study) and TA655 [66]. All the other evidence was categorised as a moderate risk of bias (one TA previously recommended OiR and three TAs OwR).
Evidence submitted in re-appraised HSTs were also categorised as moderate or serious risk of bias. All evidence but one (study ENB-003-08 [73] in HST6) was considered to have serious risk of bias in more than one domain.
3.5 NICE Recommendations Following Re-appraisal
There were nine TAs with OiR recommendation, of which three (TA037 → TA137, TA111 → TA217 and TA288 → TA418) were recommended in full for at least one indication. Two TAs were optimised (TA111 → TA217 and TA244 → TA461), four remained recommended OiR (TA033 → TA093, TA072 → CG79, TA093 → CG131 and TA193) and TA065 was withdrawn. Of the seven TAs with OwR recommendation, two TAs were subsequently recommended in full (TA472 → TA629 and TA447 → TA531), three TAs were optimised (TA416 → TA653, TA446 → TA524 and TA483 → TA655), one TA was not recommended (TA070 → TA241) and TA465 was withdrawn.
Of the six TAs that complied fully with NICE’s recommendations for further research (either OiR or OwR), one (TA418) was recommended in line with marketing authorisation, three TAs (TA524, TA653 and TA655) were recommended optimised, TA093 remained OiR and TA465 was withdrawn. Among the seven TAs that complied partially with NICE’s recommendations, TA531 and TA629 were recommended in line with marketing authorisation and TA461 was recommended optimised. TA217 received full recommendation for patients with severe disease but optimised for those with moderate disease. CG79 and CG131 remained OiR and TA241 was not recommended. Three TAs did not provide new evidence, but of those, TA137 was recommended, TA193 remained OiR for both indications and TA065 was withdrawn.
All HSTs were recommended after re-appraisal with HST19 and HST2 recommended according to marketing authorisation, while HST23 was recommended optimised.
4 Discussion
4.1 Statement of Principal Findings
Over the period of analysis, NICE conditionally recommended more medicines in technology appraisals OwR (41) than OiR (12). However, more TAs with OiR recommendations were subsequently re-appraised (9) than OwR recommendations (7). Upon re-appraisal, only six TAs (two originally with OiR and four with OwR recommendations) complied fully with NICE’s request for further evidence; three TAs did not comply.
The majority of re-appraised TAs included evidence that was deemed to be at serious, high, moderate or unclear risk of bias, with Aparicio et al. [36] in CG131 categorised as critical. Only eight randomised studies from four appraisals were supported by evidence that was judged to have low risk of bias in all domains. Reliance on observational studies and audits was commonplace among TAs that were subsequently re-appraised. However, the majority of these studies were judged as having high or critical risk of bias. The increasing use of SACT and Blueteq for real-world evidence of benefit from cancer and high-cost medicines is fraught with issues resulting from missing data and confounding, and which could be mitigated through well-designed RCTs.
All five conditionally approved HSTs were recommended OwR. Of these, three were re-appraised and were compliant with NICE’s request of further information. However, all the evidence submitted was categorised as unclear, moderate or high/serious risk of bias.
Overall, the quality of evidence submitted in response to OiR and OwR recommendations was poor.
4.2 Comparison with Other Studies
To our best knowledge, this is the first systematic study of the quality and biasedness of evidence resulting from NICE OiR and OwR recommendations. Other researchers, however, have (1) highlighted the challenges for health technology assessment in the face of higher clinical uncertainty relating to conditional marketing authorisation pathways [76]; (2) investigated the use of real-world data within single technology appraisals of cancer medicines [77] or as part of MAA associated with CDF medicines [12, 13] and (3) compared real-world data relating to targeted and non-targeted cancer therapies [14].
4.3 Strengths and Limitations
This review provides a comprehensive assessment of the nature and quality of evidence requested and subsequently evaluated by NICE following conditional recommendation. It benefits from the systematic application of two widely utilised tools for the determination of the risks of bias in randomised and observational data. Our findings should be of value in informing the application of the NICE real-world evidence framework [78].
One limitation of this review has been our reliance on the availability of documents on NICE’s website. Some superseded documents have been removed from the website, while many documents are highlighted as being commercially sensitive with text redacted [79]. The lack of transparency in aspects of the CDF has been noted previously [80], although some historical documents could be retrieved from UK web archives [81, 82]. Additionally, as NICE does not explicitly label OwR recommendations as such, assumptions were necessary in relation to those TAs for medicines conditionally recommended via processes other than OiR [8, 83,84,85]. A third limitation related to the application of tools used to assess the risk of bias being restricted to the information publicly available for those reviewed studies. This was compounded by analyses of SACT and Blueteq data not being available as publications in peer-reviewed journals. Finally, as our study was limited to NICE, its findings may not be fully generalisable to other jurisdictions.
4.4 Policy and Research Implications
There are four important policy implications to our research. Firstly, while OwR recommendations were more likely to result in re-appraisal, they rely heavily on data from potentially biased studies. The increasing use of non-randomised studies for providing evidence on clinical effectiveness is problematic, not least as confounding, selection and information bias can undermine the reliability of such data. The cost of incorrect decisions upon re-appraisal can be significant—there is a fallacy to expediting wider access to treatment through reliance on observational data. The limitations of non-randomised data are acknowledged in the NICE real-world evidence framework, which has the specific purpose and aim of improving the quality of real-world evidence to inform NICE’s guidance.
Secondly, non-compliance with NICE-requested evidence generation is widespread. The reasons for this were not apparent in most instances; however, it is concerning both that manufacturers do not always provide the evidence requested and that NICE recommendations are seemingly unaffected by this.
Thirdly, NICE has resisted using value of information methods to help select the specific design and sample size of a proposed study, as well as the value of OiR versus OwR [86]. Instead, NICE assesses the nature and methods of evidence generation (in respect to MAA) according to whether data collection and analyses are feasible in a reasonable timeframe, do not represent an unreasonable burden on patients and the NHS and are likely to support the case for a positive recommendation upon re-appraisal [87]. While conceptually attractive, value of information analyses require detailed specification of the decision problem and quantification of the associated uncertainty. This typically entails a level of complexity that may obfuscate, and a requirement for data that may not be reliably available—although a valid outcome from such an analysis would be evidence to inform the choice of whether a recommendation ought to be OiR or OwR.
Fourthly is the timeliness of data. Others have commented on the delays and costs in acquiring evidence following OiR recommendations—such as in the extreme example of treatments for multiple sclerosis [88]. Our review found evidence to be either immature or had been superseded by RCTs in some re-appraised TAs following OwR recommendations. While the duration of data collection is agreed within data collection agreements, it is well recognised that some medicines with MAA remain on the CDF for extended periods of time [80].
5 Conclusions
This review highlights important variation in the quality of evidence submitted in response to NICE conditional approval (OiR and OwR) recommendations. It further identifies non-compliance with agreed data requests and discusses the implications in respect to pharmaceutical policy. The increased reliance on real-world evidence raises concerns about the risks of incorrect decisions being made based on inaccurate data. This may be mitigated through more careful considerations of the limitations of observational evidence, and potentially pre-empted by conducting value of information analyses that could inform the nature and choice of additional data collection methods. As a minimum, quality standards ought to be stipulated in respect to evidence contributing to NICE re-appraisals following conditional approval recommendations.
References
Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, risk sharing, or patient access scheme? A framework for coverage decisions. Value Health. 2012;15:570–9.
Martelli N, van den Brink H, Borget I. New French coverage with evidence development for innovative medical devices: improvements and unresolved issues. Value Health. 2016;19:17–9.
Hutton J, Trueman P, Henshall C. Coverage with evidence development: an examination of conceptual and policy issues. Int J Technol Assess Health Care. 2007;23:425–32.
Reckers-Droog V, Federici C, Brouwer W, Drummond M. Challenges with coverage with evidence development schemes for medical devices: a systematic review. Health Policy Technol. 2020;9:146–56.
Levin L, Goeree R, Levine M, Krahn M, Easty T, Brown A, et al. Coverage with evidence development: the Ontario experience. Int J Technol Assess Health Care. 2011;27:159–68.
Lu CY, Lupton C, Rakowsky S, Babar ZUD, Ross-Degnan D, Wagner AK. Patient access schemes in Asia-pacific markets: current experience and future potential. J Pharm Policy Pract. 2015. https://doi.org/10.1186/s40545-014-0019-x.
Zampirolli Dias C, Godman B, Peres Gargano L, Santos Azevedo P, Morgado Garcia M, Souza Cazarim M, et al. Integrative review of managed entry agreements: chances and limitations. Pharmacoeconomics. 2020;38:1165–85.
Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, risk sharing or patient access scheme? | Centre for Health Economics—The University of York [Research paper 77]. 2012. https://pure.york.ac.uk/portal/en/publications/coverage-with-evidence-development-only-in-research-risk-sharing-or-patient-access-scheme(47787ac9-d258-42e0-89a0-dc19b52939d2).html. Accessed 2020 Jul 14.
NICE. Types of technology appraisal recommendation—National Institute for Health and Care Excellence. https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-technology-appraisal-guidance/types-of-recommendation. Accessed 22 Jun 2020.
NHS. Cancer drug fund list. What has changed in the CDF? National Health System England. https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-technology-appraisal-guidance/cancer-drugs-fund. Accessed 2020 Jun 22.
Trueman P, Grainger DL, Downs KE. Coverage with evidence development: applications and issues. Int J Technol Assess Health Care. 2010;26:79–85.
Morrell L, Wordsworth S, Schuh A, Middleton MR, Rees S, Barker RW. Will the reformed cancer drugs fund address the most common types of uncertainty? An analysis of NICE cancer drug appraisals. BMC Health Serv Res. 2018. https://doi.org/10.1186/s12913-018-3162-2.
Kang J, Cairns J. “Don’t think twice, it’s all right”: Using additional data to reduce uncertainty regarding oncologic drugs provided through managed access agreements in England. PharmacoEconomics Open. 2022;7:77–91.
Kang J, Cairns J. Exploring uncertainty and use of real-world data in the national institute for health and care excellence single technology appraisals of targeted cancer therapy. BMC Cancer. 2022. https://doi.org/10.1186/s12885-022-10350-8.
Hall PS, Edlin R, Kharroubi S, Gregory W, McCabe C. Expected net present value of sample information. Med Decis Mak. 2012;32:E11-21.
Charlton V, Lomas J, Mitchell P. NICE’s new methods: putting innovation first, but at what cost? BMJ. 2022. https://doi.org/10.1136/bmj-2022-071974.
Davis C, Naci H, Gurpinar E, Poplavska E, Pinto A, Aggarwal A. Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009–13. BMJ. 2017;359: j4530. https://doi.org/10.1136/bmj.j4530.
NICE. Technology appraisal data: appraisal recommendations—National Institute for Health and Care Excellence. https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-technology-appraisal-guidance/data/appraisal-recommendations. Accessed 8 Jan 2021.
NICE. Guidance programme: highly specialized technologies—National Institute for Health and Care Excellence. https://www.nice.org.uk/guidance/published?ndt=Guidance&ngt=Highly%20specialized%20technologies%20guidance&ps=9999. Accessed 30 Oct 2023
Higgins JPT, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The cochrane collaborations tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928–38.
Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016. https://doi.org/10.1136/bmj.i4919.
Cunningham D, Zalcberg JR, Rath U, Oliver I, van Cutsem E, Svensson C, et al. Final results of a randomised trial comparing “tomudex” (raltitrexed) with 5-fluorouracil plus leucovorin in advanced colorectal cancer. Ann Oncol. 1996;7:961–5.
Cocconi G, Cunningham D, Van Cutsem E, Francois E, Gustavsson B, van Hazel G, et al. Open, randomized, multicenter trial of raltitrexed versus fluorouracil plus high-dose leucovorin in patients with advanced colorectal cancer. Tomudex colorectal cancer study group. J Clin Oncol. 1998;16:2943–52.
Maughan T, James R, Kerr D, Ledermann J, McArdle C, Seymour M, et al. Comparison of survival, palliation, and quality of life with three chemotherapy regimens in metastatic colorectal cancer: a multicentre randomised trial. Lancet. 2002;359:1555–63.
Nuki G, Bresnihan B, Bear MB. Long-term safety and maintenance of clinical improvement following treatment with anakinra (recombinant human interleukin-1 receptor antagonist) in patients with rheumatoid arthritis: extension phase of a randomized, double-blind, placebo-controlled trial. Arthritis Rheum. 2002;46:2838–46.
Cohen SB, Woolley JM, Chan W. Interleukin 1 receptor antagonist anakinra improves functional status in patients with rheumatoid arthritis. J Rheumatol. 2003;30:225–31.
Genovese MC, Cohen S, Moreland L, Lium D, Robbins S, Newmark R, et al. Combination therapy with etanercept and anakinra in the treatment of patients with rheumatoid arthritis who have been treated unsuccessfully with methotrexate. Arthritis Rheum. 2004;50:1412–9.
Bresnihan B, Newmark R, Robbins S, Genant HK. Effects of anakinra monotherapy on joint damage in patients with rheumatoid arthritis. Extension of a 24-week randomized, placebo-controlled trial. J Rheumatol. 2004;31:1103–11.
Popov I, Carrato A, Sobrero A, Vincent M, Kerr D, Labianca R, et al. Raltitrexed (Tomudex) versus standard leucovorin-modulated bolus 5-fluorouracil: results from the randomised phase III Pan-European Trial in Adjuvant Colon Cancer 01 (PETACC-1). Eur J Cancer. 2008;44:2204–11.
Feliu J, Castañón C, Salud A, Mel JR, Escudero P, Pelegrín A, et al. Phase II randomised trial of raltitrexed-oxaliplatin vs raltitrexed-irinotecan as first-line treatment in advanced colorectal cancer. Br J Cancer. 2005;93:1230–5.
Feliu J, Salud A, Escudero P, López-Gómez L, Pericay C, Castañón C, et al. Irinotecan plus raltitrexed as first-line treatment in advanced colorectal cancer: a phase II study. Br J Cancer. 2004;90:1502–7.
Cortinovis D, Bajetta E, Di Bartolomeo M, Dognini G, Beretta E, Ferrario E, et al. Raltitrexed plus oxaliplatin in the treatment of metastatic colorectal cancer. Tumori. 2004;90:186–91.
Laudani A, Gebbia V, Leonardi V, Savio G, Borsellino N, Cusimano MP, et al. Activity and toxicity of oxaliplatin plus raltitrexed in 5-fluorouracil refractory metastatic colorectal adeno-carcinoma. Anticancer Res. 2004;24:1139–42.
Santini D, Massacesi C, DAngelillo RM, Marcucci F, Campisi C, Vincenzi B, et al. Raltitrexed plus weekly oxaliplatin as first-line chemotherapy in metastatic colorectal cancer: a multicenter non-randomized phase II study. Med Oncol. 2004;21:59–66.
Chiara S, Nobile MT, Tomasello L, Acquati M, Taveggia P, Murolo C, et al. Phase II trial of irinotecan and raltitrexed in chemotherapy-naive advanced colorectal cancer. Anticancer Res. 2005;25:1391–6.
Aparicio J, Vicent JM, Maestu I, Bosch C, Galán A, Busquier I, et al. First-line treatment with irinotecan and raltitrexed in metastatic colorectal cancer. Oncology. 2005;68:58–63.
Maroun JA, Jonker D, Seymour L, Goel R, Vincent M, Kocha W, et al. A national cancer institute of Canada clinical trials group study IND.135: Phase i/II study of irinotecan (camptosar), oxaliplatin and raltitrexed (tomudex) (COT) in patients with advanced colorectal cancer. Eur J Cancer. 2006;42:193–9.
Vyzula R, Kocáková I, Demlová R, Kiss I, Dusek L, Jarkovský J. Raltitrexed plus oxaliplatin in the second-line treatment of metastatic colorectal cancer. Neoplasma. 2006;53:119–27.
Reisberg B, Doody R, Stöffler A, Schmitt F, Ferris S, Möbius HJ. Memantine in moderate-to-severe Alzheimers disease. N Engl J Med. 2003;348:1333–41.
Tariot PN, Farlow MR, Grossberg GT, Graham SM, McDonald S, Gergel I, et al. Memantine treatment in patients with moderate to severe Alzheimer disease already receiving donepezil. JAMA. 2004;291:317.
Peskind ER, Potkin SG, Pomara N, Ott BR, Graham SM, Olin JT, et al. Memantine treatment in mild to moderate Alzheimer disease: a 24-week randomized, controlled trial. Am J Geriatr Psychiatry. 2006;14:704–15.
van Dyck CH, Tariot PN, Meyers B, Resnick EM. A 24-week randomized, controlled trial of memantine in patients with moderate-to-severe Alzheimer disease. Alzheimer Dis Assoc Disord. 2007;21:136–43.
Porsteinsson A, Grossberg G, Mintzer J, Olin J. Memantine treatment in patients with mild to moderate alzheimers disease already receiving a cholinesterase inhibitor: a randomized, double-blind, placebo-controlled trial. Curr Alzheimer Res. 2008;5:83–9.
Bakchine S, Loft H. Memantine treatment in patients with mild to moderate Alzheimers disease: results of a randomised, double-blind, placebo-controlled 6-month study. J Alzheimers Dis. 2007;11:471–9.
Hartmann S, Möbius HJ. Tolerability of memantine in combination with cholinesterase inhibitors in dementia therapy. Int Clin Psychopharmacol. 2003;18:81–5.
Atri A, Shaughnessy LW, Locascio JJ, Growdon JH. Long-term course and effectiveness of combination therapy in Alzheimer disease. Alzheimer Dis Assoc Disord. 2008;22:209–21.
Vidal J-S, Lacombe J-M, Dartigues J-F, Pasquier F, Robert P, Tzourio C, et al. Evaluation of the impact of memantine treatment initiation on psychotropics use: a study from the French national health care database. Neuroepidemiology. 2008;31:193–200.
Clerici F, Vanacore N, Elia A, Spila-Alegiani S, Pomati S, Cas RD, et al. Memantine in moderately-severe-to-severe Alzheimer’s disease. Drugs Aging. 2009;26:321–32.
Lopez OL, Becker JT, Wahed AS, Saxton J, Sweet RA, Wolk DA, et al. Long-term effects of the concomitant use of memantine with cholinesterase inhibition in Alzheimer disease. J Neurol Neurosurg Psychiatry. 2009;80:600–7.
Martinez FJ, Calverley PMA, Goehring U-M, Brose M, Fabbri LM, Rabe KF. Effect of roflumilast on exacerbations in patients with severe chronic obstructive pulmonary disease uncontrolled by combination therapy (REACT): a multicentre randomised controlled trial. Lancet. 2015;385:857–66.
Martinez FJ, Rabe KF, Sethi S, Pizzichini E, McIvor A, Anzueto A, et al. Effect of roflumilast and inhaled corticosteroid/long-acting β2-agonist on chronic obstructive pulmonary disease exacerbations (RE^2SPOND). A randomized clinical trial. Am J Respir Crit Care Med. 2016;194:559–67.
Mathieu C, Marmolejo MH, González JGG, Hansen L, Chen H, Johnsson E, et al. Efficacy and safety of triple therapy with dapagliflozin add-on to saxagliptin plus metformin over 52 weeks in patients with type 2 diabetes. Diabetes Obes Metab. 2016;18:1134–7.
Matthaei S, Catrinoiu D, Celiński A, Ekholm E, Cook W, Hirshberg B, et al. Randomized, double-blind trial of triple therapy with saxagliptin add-on to dapagliflozin plus metformin in patients with type 2 diabetes. Diabetes Care. 2015;38:2018–24.
Kantarjian H, Pasquini R, Hamerschlak N, Rousselot P, Holowiecki J, Jootar S, et al. Dasatinib or high-dose imatinib for chronic-phase chronic myeloid leukemia after failure of first-line imatinib: a randomized phase 2 trial. Blood. 2007;109:5143–50.
Kantarjian H, Pasquini R, Lévy V, Jootar S, Holowiecki J, Hamerschlak N, et al. Dasatinib or high-dose imatinib for chronic-phase chronic myeloid leukemia resistant to imatinib at a dose of 400 to 600 milligrams daily. Cancer. 2009;115:4136–47.
Breccia M, Stagno F, Vigneri P, Latagliata R, Cannella L, Fabro VD, et al. Imatinib dose escalation in 74 failure or suboptimal response chronic myeloid leukaemia patients at 3-year follow-up. Am J Hematol. 2010;85(5):375–7.
Koh Y, Inho K, Yoon S-S, Kim BK, Kim D-Y, Lee J-H, et al. Phase IV study evaluating efficacy of escalated dose of imatinib in chronic myeloid leukemia patients showing suboptimal response to standard dose imatinib. Ann Hematol. 2010;89:725–31.
Rajappa S, Mallavarapu KM, Gundeti S, Paul TR, Jacob RT, Digumarti R. Kinase domain mutations and responses to dose escalation in chronic myeloid leukemia resistant to standard dose imatinib mesylate. Indian J Med Paediatr Oncol. 2013;34:221–3.
Mok TS, Wu Y-L, Ahn M-J, Garassino MC, Kim HR, Ramalingam SS, et al. Osimertinib or platinum-pemetrexed in EGFR T790-positive lung cancer. N Engl J Med. 2017;376:629–40.
Goss G, Tsai C-M, Shepherd FA, Bazhenova L, Lee JS, Chang G-C, et al. Osimertinib for pretreated EGFR Thr790Met-positive advanced non-small-cell lung cancer (AURA2): a multicentre, open-label, single-arm, phase 2 study. Lancet Oncol. 2016;17:1643–52.
Walewski J, Hellmann A, Siritanaratkul N, Ozsan GH, Ozcan M, Chuncharunee S, et al. Prospective study of brentuximab vedotin in relapsed/refractory Hodgkin lymphoma patients who are not suitable for stem cell transplant or multi-agent chemotherapy. Br J Haematol. 2018;183:400–10.
Reck M, Rodríguez-Abreu D, Robinson AG, Hui R, Csőszi T, Fülöp A, et al. Pembrolizumab versus chemotherapy for PD-L1-positive non-small-cell lung cancer. New Engl J Med. 2016;375:1823–33.
Tap WD, Wagner AJ, Schöffski P, Martin-Broto J, Krarup-Hansen A, Ganjoo KN, et al. Effect of doxorubicin plus olaratumab vs doxorubicin plus placebo on survival in patients with advanced soft tissue sarcomas. JAMA. 2020;323:1266.
Cheson BD, Chua N, Mayer J, Dueck G, Trněný M, Bouabdallah K, et al. Overall survival benefit in patients with rituximab-refractory indolent non-Hodgkin lymphoma who received obinutuzumab plus bendamustine induction and obinutuzumab maintenance in the GADOLIN study. J Clin Oncol. 2018;36:2259–66.
Brahmer J, Reckamp KL, Baas P, Crinò L, Eberhardt WEE, Poddubskaya E, et al. Nivolumab versus docetaxel in advanced squamous-cell nonsmall-cell lung cancer. N Engl J Med. 2015;373:123–35.
Gettinger S, Horn L, Jackman D, Spigel D, Antonia S, Hellmann M, et al. Five-year follow-up of nivolumab in previously treated advanced nonsmall-cell lung cancer: Results from the CA209-003 study. J Clin Oncol. 2018;36:1675–84.
Hendriksz CJ, Burton B, Fleming TR, Harmatz P, Hughes D, Jones SA, et al. Efficacy and safety of enzyme replacement therapy with BMN 110 (elosulfase alfa) for morquio a syndrome (mucopolysaccharidosis IVA): a phase 3 randomised placebo-controlled study. J Inherit Metab Dis. 2014;37:979–90.
Hendriksz CJ, Berger KI, Parini R, AlSayed MD, Raiman J, Giugliani R, et al. Impact of long-term elosulfase alfa treatment on respiratory function in patients with morquio a syndrome. J Inherit Metab Dis. 2016;39:839–47.
Harmatz P, Mengel KE, Giugliani R, Valayannopoulos V, Lin S-P, Parini R, et al. The morquio a clinical assessment program: Baseline results illustrating progressive, multisystemic clinical impairments in Morquio a subjects. Mol Genet Metab. 2013;109:54–61.
Cleary M, Davison J, Gould R, Geberhiwot T, Hughes D, Mercer J, et al. Impact of long-term elosulfase alfa treatment on clinical and patient-reported outcomes in patients with mucopolysaccharidosis type IVA: results from a managed access agreement in England. Orphanet J Rare Dis. 2021;16:38.
Mercuri E, Muntoni F, Osorio AN, Tulinius M, Buccella F, Morgenroth LP, et al. Safety and effectiveness of ataluren: Comparison of results from the STRIDE registry and CINRG DMD natural history study. J Comp Effect Res. 2020;9:341–60.
GmbH AP. Asfotase alfa for infants and young children with hypophosphatasia: 7 year outcomes of a single-arm, open-label, phase 2 extension trial. European Union Clinical Trials Register. 2018;1–83. https://www.clinicaltrialsregister.eu/ctr-search/search?query=ENB-009-10.
Whyte MP, Simmons JH, Moseley S, Fujita KP, Bishop N, Salman NJ, et al. Asfotase alfa for infants and young children with hypophosphatasia: 7 year outcomes of a single-arm, open-label, phase 2 extension trial. Lancet Diabetes Endocrinol. 2019;7:93–105.
Hofmann CE, Harmatz P, Vockley J, Högler W, Nakayama H, Bishop N, et al. Efficacy and safety of asfotase alfa in infants and young children with hypophosphatasia: a phase 2 open-label study. J Clin Endocrinol Metab. 2019;104:2735–47.
Whyte MP, Madson KL, Phillips D, Reeves AL, McAlister WH, Yakimoski A, et al. Asfotase alfa therapy for children with hypophosphatasia. JCI Insight. 2016;1(9): e85971.
Mills M. HTA barriers for conditional approval drugs. Pharmacoeconomics. 2023;41:529–45.
Bullement A, Podkonjak T, Robinson MJ, Benson E, Selby R, Hatswell AJ, et al. Real-world evidence use in assessments of cancer drugs by NICE. Int J Technol Assess Health Care. 2020;36:388–94.
NICE. National Institute for Health and Care Excellence—NICE real-world evidence framework. 2022. https://www.nice.org.uk/corporate/ecd9/chapter/overview. Accessed 28 Mar 2023.
Osipenko L. Audit of data redaction practices in NICE technology appraisals from 1999 to 2019. BMJ Open. 2021;11: e051812.
Wood EM, Hughes DA. The new and non-transparent cancer drugs fund. Pharmacoeconomics. 2019;38:1–4.
UKWA. United Kingdom Web Archive. https://www.webarchive.org.uk/. Accessed 2 Apr 2 2021.
UKGWA. United Kingdom government Web Archive. https://webarchive.nationalarchives.gov.uk/search/. Accessed 30 Oct 2023.
Dhalla IA, Garner S, Chalkidou K, Littlejohns P. Perspectives on the National Institute for Health and Clinical Excellence’s recommendations to use health technologies only in research. Int J Technol Assess Health Care. 2009;25:272–80.
Longworth L, Youn J, Bojke L, Palmer S, Griffin S, Spackman E, et al. When does NICE recommend the use of health technologies within a programme of evidence development? Pharmacoeconomics. 2013;31:137–49.
Claxton K, Palmer S, Longworth L, Bojke L, Griffin S, Soares M, et al. A comprehensive algorithm for approval of health technologies with, without, or only in research: the key principles for informing coverage decisions. Value Health. 2016;19:885–91.
Drummond M, Federici C, Reckers-Droog V, Torbica A, Blankart CR, Ciani O, et al. Coverage with evidence development for medical devices in Europe: can practice meet theory? Health Econ. 2022;31(Suppl 1):179–94.
The National Institute for Health and Care Excellence—NICE. NICE health technology evaluations: the Manual. https://www.nice.org.uk/process/pmg36/chapter/introduction-to-health-technology-evaluation. Accessed 15 Feb 2022.
Giovannoni G, Brex P, Dhiraj D, Fullarton J, Freddi M, Rodgers-Gray B, et al. Glatiramer acetate as a clinically and cost-effective treatment of relapsing multiple sclerosis over 10 years of use within the national health service: final results from the UK risk sharing scheme. Mult Scler J Exp Transl Clin. 2019;5:205521731989310.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
This research was supported by the MRC-NIHR Trials Methodology Research Partnership (MR/S014357/1) via a Bangor University hosted PhD studentship. DAH is a Health and Care Research Wales Senior Research Leader (SRL 2022-25-20). Dyfrig Hughes is an Editorial Board Member of PharmacoEconomics. He was not involved in the selection of peer reviewers for the manuscript nor any of the subsequent editorial decisions.
Data availability
The datasets analysed during the current study are available via https://www.nice.org.uk/.
Author contributions
Dyfrig Hughes conceived the study, secured the funding and provided supervision. Dyfrig Hughes and Yankier Pijeira Perez contributed to the study design. Data collection and analysis were performed by Yankier Pijeira Perez. The first draft of the manuscript was written by Yankier Pijeira Perez. Dyfrig Hughes contributed to subsequent drafts, and both authors approved the final manuscript.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
About this article
Cite this article
Pijeira Perez, Y., Hughes, D.A. Evidence Following Conditional NICE Technology Appraisal Recommendations: A Critical Analysis of Methods, Quality and Risk of Bias. PharmacoEconomics (2024). https://doi.org/10.1007/s40273-024-01418-3
Accepted:
Published:
DOI: https://doi.org/10.1007/s40273-024-01418-3