Abstract
Background
In October 2020, digital health applications (DiGAs) became part of standard care in Germany. For approval, DiGA manufacturers must demonstrate medical benefit or patient-relevant improvement of structure and processes (PISP). PISP refers to an innovative outcome core area in terms of proof of benefits and reimbursement decisions. These are subdivided into 9 outcome domains, including for example health literacy, facilitating access to care, and coping with illness-related difficulties in everyday life. Their implementation aims at empowering patients, encouraging shared decision-making, and increasing patient-centeredness in healthcare delivery. Given the novelty of PISP, no standardized set of outcomes and outcome measurement instruments currently exists to operationalize the domains. Learning from previous evaluation studies can help operationalize and standardize PISPs for evaluation studies of digital health applications. Therefore, we investigated the outcomes and outcome measurement instruments, used in controlled trials to assess DiGA-compliant applications, published before the Digital Health Applications Ordinance of April 2020.
Methods
We conducted a systematic review of studies published between 01/2015 and 04/2020, via MEDLINE and Embase, complemented by forward/backward searches. Controlled trials assessing interventions adhering to the definition of DiGA were eligible, if they applied a validated outcome measurement instrument, and if results were presented in German or English. Title-abstract screening, full-text screening, data extraction and narrative synthesis were conducted independently by two researchers.
Results
Out of 2,671 references identified, 6 studies collecting a total of 48 outcomes were included. 14 outcomes (29.2%) addressed PISP by using 13 different measurement instruments. The outcomes corresponded to 5 of 9 PISP outcome domains with health literacy being the most common (7/14, 50.0%).
Conclusions
This review provides an overview of the characteristics of PISPs used in previous evaluation studies of DiGA-compliant applications. It shows which outcomes and validated outcome measurement instruments can be used to measure PISP and where knowledge is still lacking. These results serve as a starting point for operationalizing and standardizing PISPs and help to increase the outcome measurement quality of PISPs.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Background
Since October 2020, digital health applications (so-called DiGAs) have been part of standard care for people covered by statutory health insurance in Germany. Given Germany’s pioneering role in international comparison, other countries and health systems follow with great interest the coverage policies and assessment processes for digital health applications implemented in Germany [1, 2]. The legal and regulatory requirements were set by the Digital Healthcare Act (Digitale-Versorgung-Gesetz; DVG) from December 2019 and the Digital Health Applications Ordinance (Digitale-Gesundheitsanwendungen-Verordnung; DiGAV) from April 2020.
1.1 Definition of DiGA
DiGAs are active, lower-risk medical devices, classified as class I or IIa medical devices according to the European Union (EU) Medical Device Regulation 2017/745 (MDR) [3]. Therefore, a medical purpose to be achieved by the digital main function of the DiGA must be defined by the manufacturer. This function is intended to support the recognition, monitoring, treatment, or alleviation of diseases or the recognition, compensation, treatment, or alleviation of injuries or disabilities. The target group for DiGAs is patients with a confirmed diagnosis, which must be assignable to a 3- or 4-digit ICD-10 code [4]. All features necessary for a digital health application to qualify as a DiGA can also be found in the DiGA Guide for Manufacturers, Service Providers and Users [5] in accordance with Section 139e of Book V of the Social Security Code (Sozialgesetzbuch V) and are summarized in Fig. 1. DiGAs can be used following a prescription of treating physicians and psychotherapists or after patients request their use and gain approval directly from their statutory health insurance.
At the request of the manufacturer and as a precondition for reimbursement, the applications are reviewed, assessed, and approved by the Federal Institute for Drugs and Medical Devices (Bundesinstitut für Arzneimittel und Medizinprodukte; BfArM) [5]. In addition to the required characteristics of DiGAs outlined above, the BfArM also reviews whether the manufacturer has provided proof that their application fulfills basic requirements such as patient safety, functionality, data protection, data security, quality, and interoperability of the medical device [5]. If the manufacturer has provided proof that these basic requirements are fulfilled, the BfArM will review whether the manufacturer has demonstrated a positive healthcare effect of their application. If all these requirements are met, the DiGA will be approved and included in the DiGA directory maintained by the BfArM [6]. Listing in the DiGA directory is a prerequisite for the prescription of a DiGA and reimbursement of its costs by the statutory health insurance.
1.2 Evaluation of positive healthcare effects
Positive healthcare effects can be demonstrated by proof of medical benefit or patient-relevant improvement of structure and processes (PISP). The term medical benefit refers to outcomes that are known from trials of clinical or pharmaceutical interventions, including (a) improvement of the state of health, (b) reduction of the duration of a disease, (c) prolongation of survival, or (d) improvement in the quality of life. In contrast, PISP refers to an outcome core area which is innovative in Germany and the international comparison both in the context of proof of benefits and in terms of reimbursement, (see Fig. 2) [1]. The 9 PISP outcome domains are (a) coordination of treatment procedures, (b) alignment of treatment with guidelines and recognized standards, (c) adherence, (d) facilitating access to care, (e) patient safety, (f) health literacy, (g) patient autonomy, (h) coping with illness-related difficulties in everyday life, and (i) reduction of therapy-related efforts and strains for patients and their relatives [5]. Table 2 describes each outcome domain, as defined by the DiGA Guide of the BfARM [5].
In the context of assessment and approval, positive healthcare effects in terms of both medical benefits and PISPs are now equally important. Therefore, a positive healthcare effect compared to standard care must be demonstrated for only one of these [5]. The motivation behind this decision was to empower patients to become more active and informed, encourage shared decision-making, and promote health literacy [5]. In addition, the integration of this new outcome core area reflects a more comprehensive understanding of the quality of care and its evaluation for the purpose of patient benefits. Therefore, it increases patient-centeredness in healthcare delivery and contributes to the principles of value-based healthcare [7, 8]. Furthermore, this approval process and the coverage of costs by statutory health insurances provide the basis for patients having low-threshold access to quality-assured digital health applications [5].
The perspective of patients on their health status and healthcare delivery, and their participation in their therapy, have become increasingly relevant in evidence-based medicine [9,10,11]. This is reflected by the significant increase in the use of patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) [12,13,14] that can also be observed regarding digital health interventions [10, 15]. In particular, all 9 PISP outcome domains can be assigned to PROMs. PROMs are used to monitor health conditions and the effectiveness of treatments and interventions, whereas PREMs are used to evaluate and monitor experiences during the delivery and use of healthcare services, both measured from the patient perspective.
1.3 Research gaps concerning PISP
Patient perspective is key in the various evaluation frameworks and guidelines on digital health interventions. However, they differ in their definition and operationalization of the patient perspective. For example, the authors of the Model for the Assessment of Telemedicine Applications (MAST) recommend that the patient perspective should be considered by measuring usability and acceptance [16]. The National Institute for Health and Care Excellence (NICE) framework proposes within evidence tier C, which relates to applications that fit the definition of a DiGA, that the focus should be on measuring effectiveness in terms of quality of life or symptom severity [17]. Frameworks also exist that recommend the measurement of particular PISP outcome domains as part of the evaluation of digital health applications, such as the Khoja–Durrani–Scott Evaluation Framework [18] and the design and evaluation framework for digital health interventions (DEDHI) [19], which covers aspects such as “improved access to care,” “equity of care,” “effects on the delivery of care,” and “service quality.”
Thus, some elements of the PISP domains are considered by existing frameworks for evaluating health applications. However, to date, none covers all 9 domains. Whether the developers of the nine PISP domains considered any of the frameworks listes above when further defining PISP for the DiGAV is unclear, and so is the decision process which led to the nine domains. Consequently, given the novelty of PISPs as outcome domains for the evaluation of DiGA, no standardized set of outcomes exists to operationalize the 9 domains, let alone a set of measurement instruments to assess them. Given the fact, however, that PISP alone are relevant criteria for the evaluation of a DiGA, this research gap needs to be closed.
As demonstrated by the aforementioned frameworks, some of the aspects that define the 9 PISP domains have been considered important for evaluating health applications. Therefore, learning from previous evaluation studies can help standardize outcomes and measurement instruments for the collection of PISP in the course of evaluation studies of digital health applications. For medicial benefits of analogue as well as digital interventions, comprehensive measurement tools exist, some of which are codified within Core Outcome Sets [20].
Against this background, this study aimed to examine the characteristics of PISP measured in previous prospective controlled evaluation studies of DiGA-compliant digital health applications, published internationally before the DiGAV.
1.4 Research questions
This systematic review examined the following research questions:
-
1.
What were the characteristics of evaluation studies in which PISPs were collected?
-
2.
Which outcome domains, outcomes, and outcome measurement instruments were used to assess PISP and medical benefits?
-
3.
How frequently were different outcome domains, outcomes, and outcome measurement instruments used in the included evaluation studies?
2 Methods
2.1 Search strategy
To investigate these research questions, we conducted an in-depth review of the evaluation studies of telemedicine applications included in the systematic review of Knapp et al. published in November 2021 [15]. The review investigated the use of PROMs and PREMs in those evaluation studies. Since PISPs belong to the field of PROMs as outlined above, this review provided an appropriate basis to investigate our research questions. The preceding review was performed as an electronic database search on MEDLINE and Embase. The inclusion and exclusion criteria, as well as the search string can be found in Additional file 1. The search string was based on previous works of other research groups on the topics of telemedicine [21] and PROMs [22]. The screening of the 2671 hits in the databases, the extraction as well as the data analysis were each performed independently by two reviewers.
The initial review included 303 studies providing the basis for our in-depth review. Studies up to April 2020 were included in the review because the Digital Health Applications Ordinance became effective then and we aimed to examine evaluation studies published before it. In addition, we performed an in-depth hand search, as well as a comprehensive forward and backward reference search starting from the research items finally included in the original review in order to ensure that we considered all relevant articles in our review. Back and forward as well as hand searches were finalized in September 2021.
2.2 Eligibility criteria
We defined inclusion and exclusion criteria for the in-depth review presented in the current paper in order to identify evaluation studies for telemedicine applications complying with the definition of a DiGA in accordance with the requirements mentioned within the introduction. The inclusion and exclusion criteria were based on participant, intervention, comparison, outcome, and study type (PICOS scheme) as shown in Table 1. Given the explicit focus on PISP, further inclusion criteria beyond those of the initial review were added in order to identify studies appropriate to answer our research questions.
2.3 Study selection and data extraction
Based on the eligibility criteria, two researchers (MS, AK) independently conducted a title and abstract screening of all articles included in the initial review as well as those gained from hand, forward and backward searches. They then independently assessed the full texts of the preselected articles for inclusion. Any disagreements were discussed between both reviewers and resolved in discussion.
Data extraction was conducted by both researchers (MS, AK) independently. To ensure a consistent approach to data extraction, the matrix for data extraction was jointly developed in advance. The selection of relevant characteristics for the extraction matrix was based on characteristics commonly used in systematic reviews, including author, title, year of publication, journal, study country, study type, and intended medical purpose indicated by 3-digit ICD-10. These were supplemented by characteristics relevant to answering our research questions, including (1) main intended use of the DiGA, describing the type of study intervention; (2) assignment of outcome(s) to the categories of medical benefit, patient-relevant improvement of structure and processes, or others; (3) outcome domain(s); (4) outcome(s); and (5) outcome measurement instrument(s). The latter were divided into validated and non-validated instruments, considering that the DiGAV requires the use of validated outcome measurement instruments to demonstrate the healthcare effects of digital health applications. Figure 2 illustrates this subdivision, based on examples from the initial review by Knapp et al. [15].
For the purpose of characterizing the main intended use of the DiGAs, we used the terminology in the DiGA Guide, which in turn matches the MDR [3]. The main intended area of usage included recognition, monitoring, treatment, alleviation, and compensation.
Due to a lack of existing binding definitions of the different PISPs and a lack of transparemcy on the development of the PISP domains, we used the DIGA Guide as a reference in order to assign outcomes to the respective PISPs [5]. Table 2 includes all PISP domains and corresponding explanations as direct citations from the German original DiGA guide and translated by the authors.
The results of the independent data extraction were discussed by both reviewers (MS, AK). Any disagreements were discussed and resolved by consensus. Additional file 2 shows the entire data extraction table.
As our review explicitly did not aim to prove the effectiveness of DiGAs, but rather to focus on the allocation and application of different outcome domains for the evaluation of DiGAs, we did not perform a risk of bias analysis which would have been necessary in the context of an effectiveness analysis.
3 Results
3.1 Study selection
The initial review by Knapp et al. identified 2,671 studies, which resulted in 303 included studies [15]. After applying the inclusion and exclusion criteria of the in-depth review presented here, 133 studies remained for title and abstract screening, and 17 were included in full-text screening. Six studies met all inclusion criteria and were included in data extraction. 11 studies were excluded, because no PISPs were measured as outcome domains of the intervention (n = 6) [23,24,25,26,27,28], PISPs were not measured by validated outcome measurement instruments as required in the approval process for DiGAs in Germany (n = 3) [29,30,31], a non-controlled study design was used (n = 2) [27, 32] or the record was a study protocol (n = 1) [33]. The flow chart in Fig. 3 shows the entire study selection process. Hand, backward and foward searches did not yield any additional studies which met all inclusion criteria.
3.2 Study characteristics
Of the 6 included studies, 4 were randomized controlled trials [34,35,36,37], 1 was a controlled pragmatic pilot trial [38], and 1 was a controlled trial [39]. The studies were published between 2016 and 2019 and covered a study period from 2009 to 2017. Three studies were conducted in the United States [35, 37, 39], and 1 each was conducted in Sweden [38], India [34], and Canada [36]. The indications addressed by the DiGA-compliant telemedicine applications were chronic obstructive pulmonary disease (COPD) [38], type 2 diabetes mellitus [34], osteoarthritis [35], bronchial asthma [36], chronic heart failure, and spinal cord injury [37]. Of the included studies, five investigated a monitoring intervention [34,35,36,37, 39], and one investigated an intervention for alleviating symptom burden [38]. Further characteristics of the included studies can be found in Additional file 2 but are not shown here because they were not relevant to answer our research questions.
3.3 Characteristics of outcome core areas, outcome domains, outcomes, and outcome measurement instruments
A total of 48 outcomes were collected in the 6 included studies, contributing to a mean of 8.0 outcomes per study. Of these, 14 (29.2%) outcomes addressed PISP, and 29 (60.4%) addressed medical benefit. The remaining 5 (8.3%) outcomes could not be assigned to the core areas of PISP or medical benefit and hence were assigned to the category “other.” They included outcome domains such as satisfaction and usability.
PISPs are divided into 9 outcome domains as outlined in the introduction. The 14 outcomes identified in the studies of our review could be assigned to 5 of these domains. The most commonly used PISP outcome domain was health literacy, with a frequency of 7 (50.0%), followed by coping with illness-related difficulties in everyday life, with a frequency of 3 (21.0%) and adherence with a frequency of 2 (14.0%) (Table 3). Within our studies, we found no outcomes fitting to the domains coordination of treatment procedures, facilitating access to care, patient autonomy, or patient safety.
A total of 13 different measurement instruments were used for the assessment of PISP outcome domains. One outcome measurement instrument, the Patient Activation Measure (PAM)-13 questionnaire [40], was used twice. Table 3 and Additional file 2 provide a comprehensive overview of all PISP and medical benefit outcome domains, outcomes, and outcome measurement instruments used, presented according to the individual studies.
The majority of PISP outcomes (71.4%, 10/14) were assessed by validated questionnaires, as shown in Fig. 4. One questionnaire (7.1%) was self-developed for evaluation purposes in a single study. In 3 cases (21.4%), PISPs were not collected by questionnaires but by patient-reported data such as self-reported medication or frequency of blood glucose testing according to schedule.
Table 4 summarizes for which PISP outcome domains we found validated outcome measurement instruments and for which PISP outcome domains knowledge is lacking.
All of the 6 included studies investigated outcomes from the core area of PISP in addition to outcomes from the core area of medical benefits. We found no study investigating solely outcomes from the core area of PISP. Furthermore, outcomes from the core area of PISP were investigated as secondary outcomes across all included studies except the one by Evangelista et al., where no difference for primary and secondary endpoints was made [39].
Table 3 provides a comprehensive overview of all included studies detailing the investigated outcome domains, the outcome including their priority, and the applied outcome measurement instruments.
4 Discussion
Our study aimed to examine the characteristics of PISP use in the context of previous prospective evaluation studies of DiGA-compliant digital health applications, published internationally before the DiGAV in April 2020.
4.1 Main findings in the context of previous research
The core area of PISP was introduced to strengthen the role of patients and take their assessments and benefits into greater account in the approval process of digital health applications [5]. Notably, PISP outcome categories are primarily process quality indicators and not outcome quality indicators as represented by medical benefit. Considering process quality indicators in addition to outcome quality indicators, which mutually influence each other, provides the basis for a more holistic, not to mention patient-centered, evaluation of digital health applications [56].
Altogether, we included 6 studies in our review. All included studies used a controlled study design, 4 (4/6) of which were randomized controlled trials. The approval process as a DiGA in Germany also requires a controlled study design, underlining the appropriateness of this design for demonstrating the effectiveness of digital interventions compared to non-controlled study designs [57,58,59]. The majority of evaluated applications focused on patients with chronic conditions (4/6) and offered monitoring of the corresponding disease as a feature (5/6). Both findings reflect the current range of telemedicine applications, which primarily offer monitoring for patients with chronic conditions [15].
The most commonly used PISP outcome domain in evaluation studies was health literacy (7/14, 50.0%), followed by coping with illness-related difficulties in everyday life (3/14, 21.0%). One possible reason for this is that health literacy [60] is a widespread and well-known outcome domain and various established outcome measurement instruments already exist in the form of validated questionnaires. For instance, the Health Literacy Tool Shed online database listed a total of 240 health literacy measurement instruments as of April 28th 2023 [61].
Within our studies, we found no outcomes belonging to the PISP domains of patient autonomy, coordination of treatment procedures, facilitating access to care, or patient safety. Since validated measurement instruments for most of these domains exist [62,63,64], and increasing patient empowerment and access to care are among of the key promises of digital health use [65], this result is surprising.
Regarding the PISP outcome domains in our evaluation studies, self-developed questionnaires as well as process-generated, tracked, or directly measured data were used for only 4 out of 14 (28.5%) of the outcomes measured. In this regard, our study provides evidence that for some PISP outcome categories, validated outcome measurement instruments may be lacking. This is also critical because validated questionnaires are mandatory in the DiGA evaluation studies for measuring and demonstrating a positive healthcare effect [5].
In addition, the analysis showed that clearly assigning the outcomes from evaluation studies to the PISP outcome domains is sometimes difficult. This is due to a lack of detailed guidance on which outcomes can be assigned to each domain and which outcome measurement instruments should be used for assessment [5]. Since the development process of the nine PSIP outcome domains is unclear, this result is not suprising. Using performance management models as suggested for the public sector could help in increasing transparency in this context [66].
Notably, all included studies evaluated outcomes from the core area of PISPs in addition to outcomes from the core area of medical benefits and only as secondary outcomes. No study solely evaluated positive effects for outcomes from the core area of PISPs. This is likely due to the novelty of PISP outcome domains in the context of approval trials and the fact that positive healthcare effects in terms of medical benefits and PISPs are now of equal importance.
4.2 Implications for future research
Our study is the first systematic review of evidence concerning the characteristics of PISPs in the context of previous evaluation studies of DiGA-compliant digital health applications. Therefore, our results are a starting point concerning the guidance needed on which outcomes and outcome measurement instruments can be used in evaluation studies of such applications to measure PISP outcome domains. The results also highlight PISP outcome domains where knowledge is still lacking about outcomes and outcome measurement instruments that can be used and thus help sharpen the focus on results of DiGA ratification in Germany and beyond. Thus, our findings can be used to further develop existing evaluation frameworks and outcome taxonomies. The discussion of our findings also provides several valuable ideas for targeted implementation and dissemination of PISP in Germany, as well as other countries that would like to strengthen the patient perspective in the evaluation and implementation of digital health applications.
Future research should advance the distinct and transparent assignment of outcomes and validated outcome measurement instruments to all 9 of the existing PISP categories based on the results of our review. Research is especially needed concerning the 6 domains for which we found no insights concerning validated outcome measurement instruments.
Another topic for future research is the further inclusion of PISP outcome categories in existing evaluation frameworks and outcome taxonomies. The questions of how PISPs can be classified within existing taxonomies, how taxonomies could be adapted, or even if new ones should be developed hold comprehensive research potential.
Additionally, updating the review in one or two years will be of great interest as it will help to analyze how the characteristics of PISP measurement in evaluation studies have developed since the first DiGA was approved in Germany in October 2020.
4.3 Implications for practice
With DiGAs now being part of standard care for people with statutory health insurance in Germany and with PISPs now relevant in terms of approval and reimbursement decisions, Germany is breaking ground. These innovations, the accompanying discussions, and thus, the results of the present study are of international interest for different target groups:
-
(I)
International digital health application manufacturers and vendors who plan to have their application approved as a DiGA in Germany, an important market in the field of mobile health applications [2].
-
(II)
Representatives of governments, and healthcare systems worldwide concerning the significance of PISP in evaluation studies of digital health applications and the associated DiGA approval and reimbursement process [2].
-
(III)
Patients, who gain low-threshold access to quality-assured and evidence-based digital health applications and whose perspective and potential benefits have become significantly more important through innovative German legislation. This greater consideration of patient benefits when implementing and evaluating care interventions follows the principles of value-based healthcare [7].
4.4 Limitations
Currently, no legally binding document elaborates on the single PISP outcome categories, the corresponding outcome domains, outcomes, or outcome measurement instruments. This lack of explanation made it difficult for us to assign the single outcomes to the respective PISP outcome categories. However, to sharpen our understanding, we used the explanations from the DiGA Guide (Table 2) [5] and insights from a text book published by an expert committee of the Federal Ministry of Health for digital health interventions [67]. Furthermore, the interventions and study settings described in the articles were considered for the assignment of outcomes to the respective PISP outcome categories, and the entire data extraction was done by two reviewers independently. Nonetheless, the difficulties we faced illustrate again the need for further clarification of which outcomes and outcome measurement instruments can be assigned to the 9 PISP outcome domains.
As this was the follow-up analysis of an existing review not directed at measuring the effectiveness of an intervention, no a priori registration of the review was filed. However, the previous review used validated search strategies as well as a piloted data extraction strategy.
Given the fact that we did not aim to measure intervention effectiveness, we refrained from any quality assessment. Therefore, no statement can be made on potential risks of bias within the included studies.
Since we were unable to obtain any additional references from the extensive aditional searches, we are confident that the initial data set and thus the conduct of a subreview was an appropriate method to address our research questions.
5 Conclusions
Our review provides an overview of the characteristics of PISP use in the context of previous prospective controlled evaluation studies of DiGA-compliant digital health applications, published internationally before the DiGAV in April 2020. Our findings also show which outcomes and outcome measurement instruments can be used in evaluation studies of digital health applications to measure PISP outcome domains. Concurrently, we have highlighted PISP outcome domains where knowledge is still lacking about outcomes and validated outcome measurement instruments that can be used. Given the few studies included, the results are a starting point for operationalizing and standardizing PISPs and, therefore, increase the outcome measurement quality of PISPs.
The possibility of demonstrating positive healthcare effects by proof of PISP, and their relevance for cost reimbursement decisions by statutory health insurance funds, both underline the active role and personal responsibility of patients in dealing with their health. This should become even more important in terms of patient-centered care. Prescription and care practice will show to what degree DiGAs and outcomes from the PISP core area are presumed relevant by healthcare providers and patients alike. The outlined need for a clear assignment of outcomes to individual PISP outcome categories is also crucial to enable informed decision-making by physicians and patients for or against a DiGA.
Abbreviations
- BfArM:
-
Federal Institute for Drugs and Medical Devices (Bundesinstitut für Arzneimittel und Medizinprodukte)
- DiGAV:
-
Digital Health Applications Ordinance (Digitale-Gesundheitsanwendungen-Verordnung)
- DVG:
-
Digital Healthcare Act (Digitale-Versorgung-Gesetz)
- MDR:
-
Medical Device Regulation 2017/745
- NICE:
-
National Institute for Health and Care Excellence (NICE)
- PISP:
-
Patient-relevant improvement of structure and processes
- PROMs:
-
Patient-reported outcome measures
- PREMs:
-
Patient-reported experience measures
References
Lauer W, Löbker W, Höfgen B. Digitale Gesundheitsanwendungen (DiGA): Bewertung der Erstattungsfähigkeit mittels DiGA-Fast-Track-Verfahrens im Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM). Bundesgesundheitsbl. 2021;64(10):1232–40.
Gerke S, Stern AD, Minssen T. Germany’s digital health reforms in the COVID-19 era: lessons and opportunities for other countries. Npj Digit Med. 2020;3:94.
The European Parliament and the Council of the European Union. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC. 2017. https://eur-lex.europa.eu/eli/reg/2017/745/oj. Accessed 28 Apr 2023.
World Health Organization. ICD-10: International statistical classification of diseases and related health problems: 10th revision. Geneva: World Health Organization; 2011.
Federal Institute for Drugs and Medical Devices (BfArM). The fast-track process for digital health applications (DiGA) according to section 139e SGBV. A guide for manufacturers, service providers and users. Bonn: Federal Institute for Drugs and Medical Devices (BfArM). 2020. https://www.bfarm.de/SharedDocs/Downloads/EN/MedicalDevices/DiGA_Guide.pdf. Accessed 28 Apr 2023.
Federal Institute for Drugs and Medical Devices (BfArM). DiGA-Verzeichnis. 2023. https://diga.bfarm.de/de/verzeichnis. Accessed 28 Apr 2023.
Porter ME. What is value in health care. N Engl J Med. 2010;363:2477–81.
Teisberg E, Wallace S, O’Hara S. Defining and implementing value-based health care: a strategic framework. Acad Med. 2020;95:682–5.
Schlieter H, Timpel P, Otto L, Richter P, Wollschlaeger B, Knapp A, Harst L. Digitale Gesundheitsanwendungen—Forderungen für deren Entwicklung. Implement Begleitende Eval Monit Versorgungsforschung. 2021;14:76–9.
European Commission. Assessing the impact of digital transformation of health services. Luxembourg: Publications Office of the European Union. 2019. https://ec.europa.eu/health/sites/default/files/expert_panel/docs/022_digitaltransformation_en.pdf. Accessed 28 Apr 2023.
McGee RG. How to include patient-reported outcome measures in clinical trials. Curr Osteoporos Rep. 2020;18:480–5.
Kargo AS, Coulter A, Jensen PT, Steffensen KD. Proactive use of PROMs in ovarian cancer survivors: a systematic review. J Ovarian Res. 2019;12:63.
Siljander MP, McQuivey KS, Fahs AM, Galasso LA, Serdahely KJ, Karadsheh MS. Current trends in patient-reported outcome measures in total joint arthroplasty: a study of 4 major orthopaedic journals. J Arthroplasty. 2018;33:3416–21.
Weszl M, Rencz F, Brodszky V. Is the trend of increasing use of patient-reported outcome measures in medical device studies the sign of shift towards value-based purchasing in Europe? Eur J Health Econ. 2019;20(Suppl 1):133–40.
Knapp A, Harst L, Hager S, Schmitt J, Scheibe M. Use of patient-reported outcome measures and patient-reported experience measures within evaluation studies of telemedicine applications: systematic review. J Med Internet Res. 2021;23:e30042.
Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A, et al. A model for assessment of telemedicine applications: MAST. Int J Technol Assess Health Care. 2012;28:44–51.
National Institute for Health and Care Excellence (NICE). Evidence standards framework for digital health technologies. 2018. https://www.nice.org.uk/corporate/ecd7/resources/evidence-standards-framework-for-digital-health-technologies-pdf-1124017457605. Accessed 28 Apr 2023.
Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. Conceptual framework for development of comprehensive e-health evaluation tool. Telemed J E Health. 2013;19:48–53.
Kowatsch T, Otto L, Harperink S, Cotti A, Schlieter H. A design and evaluation framework for digital health interventions. Inf Technology. 2019;61:253–63.
COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments). 2023. https://www.cosmin.nl/. Accessed 28 Apr 2023.
Arnold K, Scheibe M, Müller O, Schmitt J. Principles for the evaluation of telemedicine applications: Results of a systematic review and consensus process. Z Evid Fortbild Qual Gesundhwes. 2016;117:9–19.
Mackintosh A, Comabella C, Hadi M, Gibbons E, Roberts N, Fitzpatrick R. PROM group construct and instrument type filters. Oxford: University of Oxford; 2010.
Kumar D, Tully LM, Iosif AM, Zakskorn LN, Nye KE, Zia A, et al. A Mobile health platform for clinical monitoring in early psychosis: implementation in community-based outpatient early psychosis care. JMIR Ment Health. 2018;5:e15.
van der Meij E, Anema JR, Leclercq WKG, Bongers MY, Consten ECJ, Schraffordt Koops SE, et al. Personalised perioperative care by e-health after intermediate-grade abdominal surgery: a multicentre, single-blind, randomised, placebo-controlled trial. Lancet. 2018;392:51–9.
Merchant R, Inamdar R, Henderson K, Barrett M, Su JG, Riley J, et al. Digital health intervention for asthma: patient-reported value and usability. JMIR Mhealth Uhealth. 2018;6:e133.
Zairina E, Abramson MJ, McDonald CF, Li J, Dharmasiri T, Stewart K, et al. Telehealth to improve asthma control in pregnancy: a randomized controlled trial. Respirology. 2016;21:867–74.
Tran C, Dicker A, Leiby B, Gressen E, Williams N, Jim H. Utilizing digital health to collect electronic patient-reported outcomes in prostate cancer: single-arm pilot trial. J Med Internet Res. 2020;22:e12689.
Kubo A, Kurtovich E, McGinnis M, Aghaee S, Altschuler A, Quesenberry C Jr, et al. A randomized controlled trial of mhealth mindfulness intervention for cancer patients and informal cancer caregivers: a feasibility study within an integrated health care delivery system. Integr Cancer Ther. 2019;18:1534735419850634.
Wills AM, Garry J, Hubbard J, Mezoian T, Breen CT, Ortiz-Miller C, et al. Nutritional counseling with or without mobile health technology: a randomized open-label standard-of-care-controlled trial in ALS. BMC Neurol. 2019;19:104.
Ramkumar PN, Haeberle HS, Ramanathan D, Cantrell WA, Navarro SM, Mont MA, et al. Remote patient monitoring using mobile health for total knee arthroplasty: validation of a wearable and machine learning-based surveillance platform. J Arthroplasty. 2019;34:2253–9.
McElroy I, Sareh S, Zhu A, Miranda G, Wu H, Nguyen M, et al. Use of digital health kits to reduce readmission after cardiac surgery. J Surg Res. 2016;204:1–7.
Le Marne FA, Butler S, Beavis E, Gill D, Bye AME. EpApp: Development and evaluation of a smartphone/tablet app for adolescents with epilepsy. J Clin Neurosci. 2018;50:214–20.
Lidington E, McGrath SE, Noble J, Stanway S, Lucas A, Mohammed K, et al. Evaluating a digital tool for supporting breast cancer patients: a randomized controlled trial protocol (ADAPT). Trials. 2020;21:86.
Kleinman NJ, Shah A, Shah S, Phatak S, Viswanathan V. Improved medication adherence and frequency of blood glucose self-testing using an m-health platform versus usual care in a multisite randomized clinical trial among people with Type 2 diabetes in India. Telemed J E Health. 2017;23:733–40.
Skrepnik N, Spitzer A, Altman R, Hoekstra J, Stewart J, Toselli R. Assessing the impact of a novel smartphone application compared with standard follow-up on mobility of patients with knee osteoarthritis following treatment with hylan g-f 20: a randomized controlled trial. JMIR Mhealth Uhealth. 2017;5:e64.
Ahmed S, Ernst P, Bartlett SJ, Valois MF, Zaihra T, Paré G, et al. The effectiveness of web-based asthma self-management system, my asthma portal (map): a pilot randomized controlled trial. J Med Internet Res. 2016;18:e313.
Kryger MA, Crytzer TM, Fairman A, Quinby EJ, Karavolis M, Pramana G, et al. The effect of the interactive mobile health and rehabilitation system on health and psychosocial outcomes in spinal cord injury: randomized controlled trial. J Med Internet Res. 2019;21:e14305.
Nyberg A, Tistad M, Wadell K. Can the COPD web be used to promote self-management in patients with COPD in swedish primary care: a controlled pragmatic pilot trial with 3 month- and 12 month follow-up. Scand J Prim Health Care. 2019;37:69–82.
Evangelista LS, Lee JA, Moore AA, Motie M, Ghasemzadeh H, Sarrafzadeh M, et al. Examining the effects of remote monitoring systems on activation, self-care, and quality of life in older patients with chronic heart failure. J Cardiovasc Nurs. 2015;30:51–7.
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the patient activation measure (pam): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004;39:1005–26.
Riegel B, Lee CS, Dickson VV, Carlson B. An update on the self-care of heart failure index. J Cardiovasc Nurs. 2009;24:485–97.
Rector TS, Johnson G, Dunkman WB, Daniels G, Farrell L, Henrick A, et al. Evaluation by patients with heart failure of the effects of enalapril compared with hydralazine plus isosorbide dinitrate on quality of life. V-HeFT II. The V-HeFT VA Cooperative Studies Group. Circulation. 1993;87(6 Suppl):vi71-7.
Lorig KR, Sobel DS, Ritter PL, Laurent D, Hobbs M. Effect of a self-management program on patients with chronic disease. Eff Clin Pract. 2001;4:256–62.
Juniper EF, Guyatt GH, Cox FM, Ferrie PJ, King DR. Development and validation of the mini asthma quality of life questionnaire. Eur Respir J. 1999;14:32–8.
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–13.
Brooks R. EuroQol: the current state of play. Health Policy. 1996;37:53–72.
Buran CF, Brei TJ, Sawin KJ, Stevens S, Neufeld J. Further development of the adolescent self management and independence scale: AMIS II. Cerebrospinal Fluid Res. 2006;3:S37.
Hall K, Dijkers M, Whiteneck G, Brooks CA, Krause JS. The Craig Handicap Assessment and Reporting Technique (CHART): Metric Properties and Scoring. Top Spinal Cord Inj Rehabil. 1998;4:16–30.
Kroll T, Kehn M, Ho P-S, Groah S. The SCI Exercise Self-Efficacy Scale (ESES): development and psychometric properties. Int J Behav Nutr Phys Act. 2007;4:34.
Wångdahl JM, Mårtensson LI. The communicative and critical health literacy scale–Swedish version. Scand J Public Health. 2014;42:25–31.
Jones PW. COPD assessment test—rationale, development, validation and performance. COPD. 2013;10:269–71.
Bestall JC, Paul EA, Garrod R, Garnham R, Jones PW, Wedzicha JA. Usefulness of the Medical Research Council (MRC) dyspnoea scale as a measure of disability in patients with chronic obstructive pulmonary disease. Thorax. 1999;54:581–6.
EuroQol. EuroQol–a new facility for the measurement of health-related quality of life. Health Policy. 1990;16:199–208.
Frändin K, Grimby G. Assessment of physical activity, fitness and performance in 76-year-olds. Scand J Med Sci Sports. 1994;4:41–6.
Olsson SJ, Ekblom Ö, Andersson E, Börjesson M, Kallings LV. Categorical answer modes provide superior validity to open answers when asking for level of physical activity: a cross-sectional study. Scand J Public Health. 2016;44:70–6.
Donabedian A. The definition of quality: a conceptual exploration. The definition of quality and approaches to its assessment. Chicago: Health Administration Press; 1980. p. 3–31.
Sachverständigenrat zur Begutachtung der Entwicklung im Gesundheitswesen. Digitalisierung für Gesundheit. Ziele und Rahmenbedingungen eines dynamisch lernenden Gesundheitssystems. 2021. https://www.svr-gesundheit.de/fileadmin/Gutachten/Gutachten_2021/SVR_Gutachten_2021.pdf. Accessed 28 Apr 2023.
Peiris D, Miranda JJ, Mohr DC. Going beyond killer apps: building a better mHealth evidence base. BMJ Glob Health. 2018;3:e000676.
Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med. 2016;51:843–51.
Tavakolikia N, Kheiltash A, Shojaeefar E, Montazeri A, Shariati M, Meysamie A. The most well-known health literacy questionnaires: a narrative review. Soc Determ Health. 2017;3:104–13.
Health Literacy Tool Shed: A database of health literacy measures. https://healthliteracy.bu.edu/. Accessed 28 Apr 2023.
Singla AK, Kitch BT, Weissman JS, Campbell EG. Assessing patient safety culture: a review and synthesis of the measurement tools. J Patient Saf. 2006;2:105–15.
Clement S, Brohan E, Jeffery D, Henderson C, Hatch SL, Thornicroft G. Development and psychometric properties the barriers to access to care evaluation scale (BACE) related to people with mental ill health. BMC Psychiatry. 2012;12:36.
Kasper J, Hoffmann F, Heesen C, Köpke S, Geiger F. MAPPIN’SDM—the multifocal approach to sharing in shared decision making. PLoS ONE. 2012;7:e34849.
Eysenbach G. What is e-health? J Med Internet Res. 2001;3:e20.
Buschor E. Performance Management in the public sector: Past, current and future trends. Rev Appl Manag Stud. 2013;11:4–9.
Brönneke JB, Debatin JF, Hagen J, Kircher P, Matthies H. DiGA VADEMECUM: Was man zu Digitalen Gesundheitsanwendungen wissen muss: Berlin: Medizinisch Wissenschaftliche Verlagsgesellschaft; 2020.
Acknowledgements
The authors would like to thank the European Regional Development Fund and the Free State of Saxony for making this study possible. We would also like to thank Stefanie Deckert for supporting the review with her expertise and project-independent perspective.
Funding
Open Access funding enabled and organized by Projekt DEAL. The review was part of the project “Häusliche Gesundheitsstation,” funded by the European Regional Development Fund and the Free State of Saxony (06/2018–09/2022; Grant number: 100278533).
Author information
Authors and Affiliations
Contributions
MS and AK jointly planned, conducted, and analyzed the review and drafted the manuscript. LH supported the conceptualization of the manuscript and reviewed and revised the first draft. JS obtained funding for the project and supported the conceptualization and realization of the review. All authors reviewed and edited the manuscript and approved the final version of the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
MS, AK and LH declare that there is no competing interest. Unrelated to this study, JS reports institutional grants for investigator-initiated research from the German Federal Joint Committee, Federal Ministry of Health, Federal Ministry of Education and Research, European Union, Federal State of Saxony, Novartis, Sanofi, Allergologisk Laboratorium København (ALK), and Pfizer. He also participated in advisory board meetings as a paid consultant for Sanofi, Lilly, and ALK. JS serves the German Ministry of Health as a member of the Sachverständigenrat Gesundheit und Pflege (expert advisory board on health and care).
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Scheibe, M., Knapp, A., Harst, L. et al. Outcome domains and measurement instruments of patient-relevant improvement of structure and processes as a new set of outcomes for evaluating and approving digital health applications: systematic review. Discov Health Systems 2, 33 (2023). https://doi.org/10.1007/s44250-023-00046-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s44250-023-00046-6