FormalPara Key Points for Decision Makers

Over 99% of non-oncology medication reports from the Canadian Agency for Drugs and Technologies in Health (CADTH) included an effectiveness critique. The most common critiques were related to evidence uncertainty, insufficient clinical evidence, and clinical data uncertainty beyond the available clinical evidence.

There was no observed association between evidence uncertainty and the final CADTH recommendation. The proportion of reports with highly uncertain evidence did not significantly differ between clinical criteria/condition and do not reimburse recommendations.

In structure critiques, lack of transparency and programming error shows significant association with a ‘do not reimburse’ recommendation.

CADTH reanalyses revealed higher median incremental costs and lower median incremental quality-adjusted life-years (QALYs) compared with manufacturers’ submitted reports. Furthermore, the discrepancies in incremental costs and incremental QALYs led to a higher incremental cost-utility ratio (ICUR) in CADTH reanalyses.

The most common budgetary impact critiques were related to miscalculations of patient population, market share percentage, critiques due to clinical data uncertainty, and the percentage of patients covered by the public plan. Furthermore, CADTH reanalyses indicated a higher 3-year budgetary impact compared with the manufacturers’ submissions.

1 Introduction

Canada is among the high-spending countries that have a publicly funded healthcare, and ranks above the Organisation for Economic Co-operation and Development (OECD) average per-person healthcare expenditure [1]. Total healthcare expenditures experienced an increase from $269 billion and 11.5% of the total gross domestic product (GDP) in 2015, to $305 billion and 13.2% of the GDP in 2020, and is projected to increase to $344 billion in 2023 [1, 2]. Medication costs in Canada, including expenditures by provincial and territorial governments, private insurance, and patients’ out-of-pocket costs, have increased by 1.6% and 4.1% in 2020 and 2021, respectively. Although drug costs for 2022 have not yet been published, they are projected to rise by 5.4% in 2022, with a total projected cost of $14.5 billion [1].

To address the financial constraints of finite resources, the Canadian Agency for Drugs and Technologies in Health (CADTH), which operates across Canada (excluding Quebec), and the Institut National d'Excellence en Santé et en Services Sociaux (INESSS) in Quebec play a crucial role in evaluating cost effectiveness and budget impact analysis (BIA) when they conduct health technology assessment (HTA) reviews. Until the end of 2022, manufacturers were required to submit an HTA report to CADTH’s Common Drug Review for non-oncology drugs, while HTA reports for oncology drugs were submitted to the Pan-Canadian Oncology Drug Review. Recently, CADTH merged its procedures and provided a harmonized procedure for both oncology and non-oncology medications [3].

Since 2015, CADTH has included drugs with evidence-based expanded use, based on clinical effectiveness and safety data, even if Health Canada had not yet approved the medication for that indication [4]. CADTH evaluates clinical evidence dossier through a process that involves performing a literature review and collecting values and preferences of clinicians and patients. Additionally, CADTH assesses the health economic model structure and data used for model development [5]. Upon completion of its internal process, CADTH issues a funding recommendation. To support these recommendations, CADTH publishes both a clinical appraisal report and an economic appraisal report. These reports include a critical appraisal of the manufacturers’ report, a reanalysis with enhanced input data and model structure, along with revised cost-utility and BIA results. Subsequently, CADTH provides a non-binding recommendation for each submission to be used by government drug reimbursement programs. The recommendations can be ‘reimburse’, ‘do not reimburse’, or ‘reimburse with clinical criteria and/or conditions’. The latter is applicable when a medication or technology’s incremental cost-utility ratio (ICUR) exceeds CADTH’s willingness to pay (WTP) threshold or when CADTH announces clinical constraints [5].

In 2017, CADTH released the Fourth Edition of the guideline for the Economic Evaluation of Health Technologies. This edition included additional details for manufacturers to consider in their HTA reports, particularly regarding modeling, measurements, costs, and handling of uncertainty. Since the release of this edition, two major changes have been implemented: all the cost-effectiveness analyses have transitioned to probabilistic, and submission of BIA reports has become mandatory [5]. Manufacturers are obliged to comply with CADTH guidelines and procedures when submitting a product for reimbursement.

While the guidelines on economic evaluation provided by CADTH are comprehensive, it could be advantageous for manufacturers to examine the critiques of previous drug submissions by CADTH. This could help them avoid recurring issues. Furthermore, understanding the implications of insufficient evidence and the importance of robust effectiveness evidence could prevent manufacturers from submitting a dossier to CADTH prematurely, prior to accumulating sufficient clinical evidence. While some studies have assessed CADTH reports on non-oncology medications, all of these were published prior to 2014. Moreover, no studies have yet assessed CADTH criticism in the context of the most recent CADTH guideline for economic evaluation [5,6,7]. We propose that there is a need for a more refined categorization of CADTH criticisms, and a subsequent assessment of these critics. This could lead to better organization and understanding of the criticisms raised by CADTH. Our study is specifically focused on CADTH decisions made between 2018 and 2022. We examined the final CADTH recommendation rates for non-oncology drugs across four key categories: those addressing unmet needs, drugs for rare diseases (DRD), various therapeutic categories, and the overall recommendation rate. We also detailed CADTH critiques, reporting them based on final recommendations with a focus related to clinical and economic data, as well as budget impact models. Additionally, we investigated whether the HTA models submitted by manufacturers generate lower ICUR and net budgetary impact compared with the reanalysis models by CADTH. Evaluating these differences is crucial for informed decision making in healthcare resource allocation.

2 Methods

2.1 Selection of Reports and Data Extraction

We included all the clinical and economic reports from CADTH for non-oncology medications that received final recommendations between 1 January 2018 and 31 December 2022. Given that CADTH only releases clinical and pharmacoeconomic reports upon completion of the evaluation process, we restricted our inclusion to reports marked as ‘complete’. To develop the abstraction sheet, we used the critiques highlighted in existing literature as a foundation and enriched these with other frequently encountered points in CADTH reports [6,7,8,9,10]. Two reviewers independently extracted data using the pilot-tested standardized abstraction sheet in Microsoft Excel (Microsoft Corporation, Redmond, WA, USA). Data extractions conducted by two reviewers were then compared and any disagreements were addressed through discussion. If a consensus could not be reached, a third reviewer was brought in to resolve the issue. We extracted the following information from the CADTH website: project number, generic drug name, brand name, therapeutic area, final recommendation, and recommendation date. We also gathered information on study type (cost-utility analysis [CUA] vs. cost-minimization analysis [CMA] and cost comparison), model type (Markov, semi-Markov, partitioned survival model, decision tree, mixed model with repeated measures, or not reported), data source (randomized controlled trials, observational studies, or combination of randomized trials and observational studies, and not reported), and utility measurement (EuroQol-5 Dimensions [EQ-5D], Short Form 36 [SF-36], Health Utilities Index [HUI], others, and not reported). We extracted results data on incremental cost, incremental quality-adjusted life-year (QALY), ICUR, and BIA for manufacturers’ submissions and the CADTH reanalysis. As CADTH’s guidelines mandate manufacturers to provide comparisons with multiple medications, we extracted data for the lower and upper extremes of the ICUR and BIA to calculate the mean values for ICURs and BIAs. We extracted the cumulative 3-year BIA reported by the manufacturer and the reanalysis by CADTH, and recorded the percentage of price reduction requested by CADTH. We excluded reports with dominance for one of the lower or upper extremes of ICUR.

We obtained data related to therapeutic categories based on the International Classification of Diseases, 11th revision (ICD-11) codes. We directly extracted data for the unmet need from the discussion of the final CADTH recommendations and derived values for the WTP threshold from the price reduction statement of the CADTH recommendations. We adopted the definition of a DRD from the CADTH’s website [11]. The CADTH final recommendation to ‘reimburse with clinical criteria and/or conditions’ is a broad recommendation that does not provide specific data on whether the criteria relate to the product’s cost or clinical limitations. Consequently, we collected additional data on final recommendations by identifying (1) any further limitations on clinical criteria, such as restricted indications, the requirement for additional diagnostic tests, or administration in specific settings (i.e., hospital), and (2) the need for price reduction.

We categorized CADTH critiques pertaining to methodological issues of manufacturers’ submissions into different groups: (1) model structure; (2) costs; (3) utility score; and (4) effectiveness. Details related to CADTH critiques on cost utility, cost minimization and cost comparison reports are summarized in Table 1 [5, 12]. Since high uncertainty level could be a potential risk for the rejection of new drugs, we also extracted level of uncertainty from the reports (Table 2). The BIA guideline of Canada (2020) states that critical points of calculating BIA should be listed as the real target population and its growth over time, opting for a suitable comparator, cost consideration, sensitivity analysis, and characterizing uncertainty [12]. According to this guideline, the target population should be based on the approved drug indication and its limitation of use, if there is any. In addition to this, a comprehensive list of critiques for BIA was considered in the abstraction sheet (Table 3).

Table 1 CADTH Critiques on cost-utility, cost-minimization, and cost comparison submissions of manufacturers
Full size table
Table 2 Uncertainty level specified in CADTH reports based on the final recommendation
Full size table
Table 3 CADTH critiques on 3-year budget impacts analysis of manufacturers’ reports in the “recommended with clinical criteria/conditions” group, “do not reimburse” group, and in total reports
Full size table

2.2 Data Management and Analysis

We summarized the mean incremental costs, incremental QALY, ICUR, and 3-year BIA in the manufacturers' base-case reports and in the CADTH’s base-case reanalysis using descriptive statistics. We used the Wilcoxon rank-sum test to compare the median differences of the manufacturers’ reports and the CADTH’s reanalysis reports for incremental costs, incremental QALY, ICUR and BIA. Distribution plots were created for ICUR and 3-year BIA to contrast the values reported by the manufacturers with the results of the CADTH reanalysis.

We reported the general characteristics of the studies, the number of drugs and the rate of positive recommendations for rare diseases, drugs addressing an unmet need, and drugs in different therapeutic categories, in a descriptive manner. CADTH defined an unmet need as a medication goal unfulfilled by the currently available treatments [13]. To identify potential predictors, addressing an unmet need and DRD underwent univariate regression modelling. Variables demonstrating a significant relationship with the outcome (p < 0.10) were subsequently included in the multiple variable regression model. All statistical analyses were carried out using SPSS version 26 (IBM Corporation, Armonk, NY, USA).

3 Results

We included 178 clinical and economic reviews, with a decision made between 1 January 2018 and 31 December 2022, on the CADTH website. Of these, 31 (17%) reports received a ‘do not reimburse’ recommendation and 147 (83%) received a ‘recommended with clinical criteria and/or conditions’ recommendation, a phrase used by CADTH to indicate acceptance conditional on either specific clinical condition, a price reduction, or both. Of the conditional recommendations, 128 (87%) reports were recommended with a clinical condition and 138 (95%) were recommended with a price reduction. CADTH’s required conditions included being prescribed by a specialist or an experienced physician (n = 91), or reimbursement for a limited time period, conditional on demonstrating a treatment response or limited to a hospital setting (n = 119). Electronic supplementary (ESM) Table S1 provides the reasons given by CADTH for rejecting the reimbursement requests.

Randomized controlled trials, along with observational studies, were used in model development in 63% of reports (n = 112), while 37% used only randomized controlled trials (n = 66). Most of the reports were CUAs (n = 155, 87%), followed by cost comparison analyses (n = 13, 7%), and CMAs (n = 10, 6%). Markov or semi-Markov models were used in 75% of the models, followed by a combination of decision tree and Markov models (13%), decision tree only (7%), and others (5%). Among pivotal trials, six were single-arm trials and six were phase II trials. ESM Table S2 provides general characteristics of the included reports. Results of the Chi-square test showed that study type (p = 0.79), trial characteristics (p = 0.16), model type (p = 0.43), and utility measurement tools (p = 0.18) were independent of the final CADTH recommendation and that these variables do not have any relationship with the final recommendations.

ESM Table S3 presents the type of reimbursement recommendations in the reports. There was no association between the therapeutic category and the type of CADTH recommendation made in the report (p = 0.32).

3.1 Canadian Agency for Drugs and Technologies in Health (CADTH) Critiques on Manufacturers’ Submissions

Almost all the submissions (99%) had at least one effectiveness critique. The most common was ‘the uncertainty of evidence’, which pertained to 96% of the reports, followed by ‘not enough clinical evidence available’ (76%), ‘clinical data uncertainty beyond clinical evidence’ (74%) and ‘not a proper population captured’ (59%). The results of the Chi-square test analyzing the effectiveness critiques indicate that ‘other effectiveness critiques’, representing critiques we could not categorize, were more prevalent in the ‘do not reimburse’ group compared with the ‘recommended with clinical criteria/conditions’ group (p < 0.05). Although critiques on the uncertainty of evidence were mentioned repeatedly in the CADTH critical appraisals, only 7 (4%) reports mentioned a standardized tool such as Grading of Recommendations Assessment, Development and Evaluation (GRADE) for measuring uncertainty of evidence.

Eighty-nine percent of reports had at least one structure critique, with the most common structure critique being ‘the model does not reflect current practice’, which was responsible for 59% of all structure critiques. Lack of transparency and programming errors were more frequent in the ‘do not reimburse’ group compared with the ‘recommended with clinical criteria/conditions’ group (p < 0.05).

Of the total reports, 79% and 69% had at least one cost and utility score critique, respectively. Using critiques on non-Canadian utility scores and also using unaccepted extra costs were more frequent in the ‘recommended with clinical criteria/conditions’ group compared with the ‘do not reimburse’ group (p < 0.05). Data on frequencies of critiques based on CADTH recommendations are summarized in Table 1.

Among all the reports, 78 (44%) were assessed as highly uncertain, 91 (52%) were assessed as uncertain and only 4 (2%) reports could adequately address the uncertainty levels. Among 31 reports with a ‘do not reimburse’ recommendation, 10 (32%) were rejected merely due to the uncertainty of evidence; however, 64 (44%) reports with a ‘reimburse with clinical criteria/conditions’ recommendation had a high uncertainty level. The Chi-square test results show no association between uncertainty level and final recommendation (p = 0.31). Table 2 shows the uncertainty level specified in CADTH recommendations based on the final recommendations.

All reports had at least one critique on the 3-year BIA analysis, with the two most common critiques of 3-year BIA related to ‘population of patients’ and ‘percentage of market share’, responsible for 80.2% and 73.2% of all 3-year BIA critiques, respectively. Frequency of CADTH critiques were not significantly different based on different final recommendations. The CADTH critiques on 3-year BIA are shown in Table 3.

3.2 Drugs for Rare Disease and Drugs that Address an Unmet Need

All 68 CADTH reports of medications that addressed an unmet need were recommended for ‘reimbursement with clinical criteria/conditions’. For DRD, 47 (92%) of 51 reports recommended medication reimbursement with clinical criteria/conditions. In contrast, only 100 (79%) of 127 non-DRD medication reports received recommendations with clinical criteria/conditions. Addressing an unmet need and DRD were associated with a recommendation supporting reimbursement (p < 0.001 and p < 0.05, respectively) (Table 4). With some submissions, manufacturers claimed their medication would address an unmet need, but CADTH refused to accept that notion. One reason for rejecting these claims would be that the claim was only based on a surrogate outcome.

Table 4 Final reimbursement recommendations in different therapeutic categories
Full size table

3.3 CADTH Reanalysis

The median incremental cost submitted by the manufacturers was $6562, with an interquartile interval of −$246 to $127,607, indicating a wide variability in the cost of interventions produced by different manufacturers. The incremental cost in the CADTH reanalysis was higher, with a median cost of $10,777 and an interquartile interval of $378–$144,505. The median difference between manufacturers and CADTH was $1837, with an interquartile interval of −$679 to $25,295, an increase of 19.6% (interquartile interval −10.1% to 110.5%) [p < 0.001].

Of the 155 CUA reports, CADTH assessed 60 medications as clinically non-inferior, indicating the new drug does not provide a superior efficacy compared with existing interventions, and therefore assumed zero incremental utility for these medications. The median incremental QALY in manufacturers’ reports was 0.29 (interquartile interval 0.04–1.36) and the median incremental QALY in the CADTH reanalysis was 0.11 (interquartile interval 0.01–0.76). The difference between the manufacturers’ analysis and the CADTH reanalysis was −0.11 (interquartile interval −0.94 to 0.00) and the percentage difference was −50.0%, with an interquartile interval of −83.7% to −1.0% (p < 0.001). The data for incremental costs and QALYs are presented in Table 5.

Table 5 Difference between manufacturers’ submissions and the CADTH reanalysis regarding incremental cost, incremental QALY, and ICUR of non-oncology medications
Full size table

Of the medications with an improved QALY, 28 (29%) reported dominance. For the remaining 67 reports (71%), the ICUR from the manufacturer and the CDR’s reanalysis were assessed. The median ICUR was significantly higher in the CADTH reanalysis compared with the manufacturers’ reports—$138,658/QALY (interquartile interval $43,203/QALY–$394,076/QALY) for manufacturers and $380,251/QALY (interquartile interval $149,197–$1,347,825) for the CADTH reanalysis (p < 0.001). The difference in the median ICURs was $169,299/QALY (interquartile interval $56,040–$574,073), i.e. 151% (interquartile interval 33–353%) higher in the CADTH reanalysis than the manufacturers’ reports (Table 5).

The distributions of the manufacturers’ and CADTH’s ICURs are presented in Fig. 1; in 5 (7%) of 67, the manufacturers’ ICUR was less than the CADTH reanalysis.

Fig. 1
figure 1

Distribution of manufacturers’ and CADTH reanalysis ICUR values for 67 CADTH reports from 2018 to 2022 for non-oncology medications

Full size image

A total of 81 medications reported a 3-year BIA, with 44 (54.3%) assessed in 2022, 30 (37%) in 2021, and 7 (9%) in 2020. The CADTH reanalyzed 76 of these reports (94%). The median 3-year BIA for manufacturers was $13,666,621 (interquartile interval $551,393–$72,640,799) and was significantly lower than the median value reported in the CADTH reanalysis, i.e. $19,104,299 (interquartile interval $3,125,747–$98,198,139) [p < 0.001]. The difference in the median values between manufacturers and CADTH was $4,575,102 (interquartile interval $9170–$26,330,512), with a percentage reduction of 27.0% (interquartile interval 0–182%) (Table 6). The distributions of the manufacturers’ and CADTH’s 3-year BIAs are summarized in Fig. 2. The 3-year BIA in the CADTH reanalysis was higher in 66 (87%) of the 76 reports.

Table 6 Three-year BIA from the manufacturers’ report and CADTH reanalysis
Full size table
Fig. 2
figure 2

Distribution of 76 3-year budget impact analysis results of manufacturer’s reports compared to the CADTH reanalysis reports from 2018 to 2022 for non-oncology medications

Full size image

The results of the univariate logistic regression model are shown in ESM Table S4. The results showed that DRD is a predictor of receiving a recommendation with criteria/conditions from CADTH. To test the robustness of the results, we ran a multiple variable logistic regression that showed that DRD and addressing an unmet need are not predictors of receiving a recommendation with clinical criteria/conditions (ESM Table S5).

3.4 Price Reduction Requested and Willingness to Pay Threshold Mentioned in CADTH Reports

CADTH requested a median price reduction of 64% (interquartile interval 35–88%) for non-oncology medications. The price reduction requested was lowest for drugs in the ‘certain infectious or parasitic diseases’ category, ‘diseases of the skin’, and ‘diseases of the musculoskeletal system or connective tissue’. The highest median CADTH requested price reduction was for ‘diseases of the blood or blood-forming organs’, ‘diseases of the respiratory system’, and ‘endocrine, nutritional or metabolic diseases’ (Fig. 3).

Fig. 3
figure 3

Price reduction requested by CADTH based on therapeutic categories for non-oncology medications

Full size image

The CADTH reports for non-oncology medications from 2018 through 2020 indicated that the most requested price reduction was a consequence of the ICUR being greater than the WTP of $50,000/QALY. Among the WTP thresholds mentioned in 2018, 2019, and 2020, the most frequent were $50,000 and $100,000. In 2018 and 2019, other WTP values were also mentioned by CADTH, including $25,000, $200,000, $400,000, and $500,000. In 2021 and 2022, $50,000 was the only WTP threshold mentioned in CADTH reports. Figure 4 shows the frequency of WTP thresholds mentioned in CADTH reports from 2018 to 2022. In 39 (22%) CADTH reports, no specific WTP threshold was specified. These reports were mostly those in which CADTH failed to find the medications to be superior to their comparators. For these reports, CADTH stated that the price of the new intervention should not exceed that of the currently available options.

Fig. 4
figure 4

Willingness to pay threshold mentioned in CADTH price reduction requests from 2018 to 2022 for non-oncology medications

Full size image

4 Discussion

CADTH is a not-for-profit, independent organization dedicated to assessing the value of new health technologies. CADTH is committed to generating evidence through an unbiased process to advise health policymakers whether to reimburse new health technologies or not [14, 15]. CADTH consistently showed that it does not make recommendations based solely on drug prices but rather focuses on their effectiveness, as was shown in previous literature [7, 9, 10]. However, reimbursement decisions can be influenced by political considerations, lobbyists, pressure groups and media attention [16,17,18,19,20]. The results of our study showed that the most important effectiveness critique, the uncertainty of evidence, was not significantly different between the recommended ‘do not reimburse’ group and the ‘recommended with clinical criteria/conditions’ group. Drugs with a high uncertainty level received ‘do not reimburse’ recommendations (44%) and were ‘recommended with clinical criteria/conditions’ (45%). This means that the level of uncertainty does not have any association with any particular final reimbursement recommendation. One possible reason could be that CADTH does not use a standardized tool such as GRADE to quantify the certainty of evidence, and therefore the uncertainty levels reported by CADTH are not measured precisely. Another reason could be the difficult environment of Canadian policymaking, which resulted in the acceptance of a proportion of drugs with highly uncertain clinical effectiveness.

Regarding model structure, we observed that lack of transparency and programming errors have an association with ‘do not reimburse’ recommendations, and shows that it is crucial to maintain transparency and avoid programming errors in economic models in CADTH submissions.

The final CADTH recommendations on new technologies that used non-Canadian utility values show that CADTH is not sensitive to the use of Canadian utility values. This might be due to the fact that many of the clinical trials of new drugs are not conducted in Canada and no Canadian utility scores are available for use in the analysis. Regarding costs, the results of our study show that the use of unacceptable extra costs by the manufacturers is associated with the ‘recommended with clinical criteria/conditions’ group, which can include a price reduction. In these cases, CADTH provides a revised economic value and bases its final recommendations on these revised values.

This is the first study to assess CADTH recommendations after release of the 4th edition of the CADTH guidelines on CADTH recommendations [5]. The results of this study showed that there are differences in both incremental costs and incremental QALYs between the manufacturers’ submission and CADTH’s reanalyses, resulting in a significant difference in ICUR.

The results of other published studies are not fully aligned with the results of this current study. The 2012 study by Rocchi et al. analyzed predictors associated with negative recommendations in CADTH non-oncology reports from 2003 to 2009. The study found that 48% of the 138 reports were rejected, with significant variance across different therapeutic categories. Factors predicting drug rejection included clinical factors, a higher price compared with comparators, a request for reconsideration, and the use of price as the only economic evidence [7]. The 2013 study by Rocchi et al. evaluated the impact of surrogate outcomes on CADTH recommendations, finding that 44% of 156 studies involved surrogate outcomes. The majority of surrogates were accepted by CADTH, but the non-accepted surrogates were significantly associated with clinical uncertainty and a higher likelihood of rejection [6]. In 2017, Masucci et al. evaluated 39 economic evaluations of oncology drugs published on the CADTH website between 2011 and 2014. The authors identified several methodological flaws in these evaluations, including issues with time horizon, duration of treatment benefit, costing, utility estimation, model structure, extrapolation techniques, uncertainty in indirect comparison, calculation errors, and the quantity of clinical data available [8]. In 2022, Ball et al. assessed HTA reports for oncology medicines in Canada, the UK, and Australia from 2019 to 2020 and reported common criticisms as critiques on costing, time horizon, utility value, treatment benefit, extrapolation, comparator, and subgroup analysis [21]. We believe this study marks the first instance of a comprehensive categorization of CADTH critiques, potentially accounting for the differences in our results compared with prior studies. For instance, our identification of transparency issues and programming errors as significant factors was novel, revealing their association with ‘do not reimburse’ recommendations.

Regarding differences in ICUR, a previous study of CADTH non-oncology reports from 2010 to 2017 showed similar results; therefore, in this regard, little has drastically changed [22]. Rocchi and Mills reported that ICUR in the CADTH reanalysis was $70K/QALY more than ICURs submitted by manufacturers [22]. In our study, the difference between the manufacturers’ submitted ICURs and the CADTH reanalysis was $170K/QALY (95% confidence interval $56K/QALY–$570K/QALY), indicating a 2.5-fold increase in the differences between the CADTH reanalysis and the manufacturers’ submissions compared with the study by Rocchi and Mills [22]. In an assessment specifically focused on oncology drug submissions from 2012 to 2018, the median ICUR in manufacturers’ submissions was lower than the CADTH reanalysis ICUR. This was driven by a difference in incremental QALY but not in incremental costs [23]. In another assessment of oncology reports from 2015 to 2018, the elevated ICUR in the reanalysis was driven by a difference in incremental costs only [24]. An evaluation of HTA reports for oncology medicines from 2019 and 2020 showed that the ICUR in the reanalysis increased by 82% compared with the manufacturers’ submissions, and that the difference was a result of differences in both incremental QALY and incremental costs [21]. In our study, like all previously published studies on CADTH reports, manufacturers’ estimates on the value of their new interventions were higher than those assessed by the CADTH [21,22,23].

In 3-year BIAs, the frequency of critiques did not show a statistically significant difference between the two final reimbursement recommendations. All reports (100%) had at least one BIA critique, indicating that CADTH evaluates BIAs with scrutiny. No previous studies have assessed CADTH critiques on the 3-year BIA of manufacturers’ submissions, and no prior studies have investigated the differences between the BIA calculated by the manufacturers versus the CADTH reanalysis, although it has been reported that pharmaceutical companies have a tendency to underreport the budget impact of their drugs in a BIA [25].

CADTH may propose a request for price reduction for costly medications; however, no medication was singled out for a ‘do not reimburse’ recommendation based solely on its cost. This observation aligns with findings from previous studies [11,12,13,14,15]. All CADTH recommendations fell into one of two categories: ‘reimburse with clinical criteria/conditions’, implying a potential price reduction or the imposition of additional clinical conditions, or ‘do not reimburse’. From 2018 to 2020, $100,000/QALY was the second most frequent WTP threshold mentioned in CADTH reports, after $50,000/QALY. From the beginning of 2021, CADTH decided to move on to the more harmonized approach and has consequently kept $50,000/QALY as the only WTP threshold for all medications, including DRD. Keeping $50,000 as the WTP threshold after 2021 was also observed in oncology medications in Canada [26]. Some studies have shown that addressing an unmet need can override the ICUR threshold in other jurisdictions [27]; however, in this study, WTP thresholds were not different between drugs that address an unmet need versus others. Since 2018, there have been no studies that have evaluated clinical and economic appraisals by therapeutic categories, DRD, or addressing an unmet need in Canada [9]. In our study, DRD showed a higher proportion of reimbursement recommendations compared with non-DRD. Although some HTA agencies may consider a higher WTP threshold for DRD, CADTH is strict on its $50,000/QALY WTP threshold [27].

The median percentage price reduction requested for DRD was found to be higher than that for non-DRD. Since CADTH had previously used a WTP of $100,000/QALY for some ultra-rare diseases, using a WTP of $50,000/QALY for all DRDs, including ultra-rare diseases, has resulted in a higher percentage price reduction in the DRD [28]. The CADTH price reduction is only the starting point of the negotiations between the drug plans and the manufacturer, and manufacturer negotiations with the pan-Canadian Pharmaceutical Alliance (pCPA) can start after a recommendation by the CADTH. The pCPA negotiates behind closed doors and publishes the results of its negotiations only as ‘concluded without agreement’ or ‘concluded with a letter of intent (LoI)’ and does not provide any details about its negotiations [29]. While CADTH provides a ‘reimburse with clinical criteria/conditions’ recommendation, the significant percentage price reduction might prevent pharmaceutical companies from successfully negotiating with the pCPA, especially for DRD. While noting that 1 in 12 Canadians are affected by rare diseases, considering a WTP of $50,000 for all DRDs might limit access to some critical medications for rare diseases [26, 30].

This study could serve as a resource for CADTH to enhance its internal procedures and adopt standard tools for reporting critiques on the uncertainty of evidence. Manufacturers can utilize this report to gauge the robustness of their submissions based on CADTH’s criteria. Regarding limitations of this study, it worth mentioning that when CADTH provides a ‘reimburse with clinical criteria/conditions’ recommendation, it only advises the policymakers, and policymakers can reject the new technology due to the high costs. Therefore, a ‘reimburse with clinical criteria/conditions’ recommendation should not be considered as a firm reimbursement command, and of course, the manufacturer may choose to not accept the conditions whether they be a price reduction or a clinical factor. In this study, we only assessed the frequency of individual critiques and the level of uncertainty in clinical evidence in CADTH reports. Nonetheless, there might be a combined effect of critiques, and CADTH might respond differently when a manufacturer presents multiple effectiveness critiques compared with when there are only a few critiques; however, our study did not capture this.

This study was limited to secondary data that are made public by CADTH, and access to primary data was not possible. CADTH also provides a detailed ‘confidential’ report to manufacturers that is not publicly available. We did not review oncology reports, as our study was limited to the non-oncology medications. We faced limitations in our analysis due to the large number of non-inferior drugs and the resulting unavailability of ICER values. Consequently, we were only able to analyze and report ICURs for approximately 40% of the submissions. Finally, the ‘reimburse with clinical criteria/conditions’ recommendation was treated as a positive recommendation from CADTH, but that could be overly optimistic since medications that receive such a recommendation with considerable price reduction requirements might not subsequently receive reimbursement during price negotiation with the pCPA.