Abstract
Overall quality of radiomics research has been reported as low in literature, which constitutes a major challenge to improve. Consistent, transparent, and accurate reporting is critical, which can be accomplished with systematic use of reporting guidelines. The CheckList for EvaluAtion of Radiomics research (CLEAR) was previously developed to assist authors in reporting their radiomic research and to assist reviewers in their evaluation. To take full advantage of CLEAR, further explanation and elaboration of each item, as well as literature examples, may be useful. The main goal of this work, Explanation and Elaboration with Examples for CLEAR (CLEAR-E3), is to improve CLEAR’s usability and dissemination. In this international collaborative effort, members of the European Society of Medical Imaging Informatics−Radiomics Auditing Group searched radiomics literature to identify representative reporting examples for each CLEAR item. At least two examples, demonstrating optimal reporting, were presented for each item. All examples were selected from open-access articles, allowing users to easily consult the corresponding full-text articles. In addition to these, each CLEAR item’s explanation was further expanded and elaborated. For easier access, the resulting document is available at https://radiomic.github.io/CLEAR-E3/. As a complementary effort to CLEAR, we anticipate that this initiative will assist authors in reporting their radiomics research with greater ease and transparency, as well as editors and reviewers in reviewing manuscripts.
Relevance statement Along with the original CLEAR checklist, CLEAR-E3 is expected to provide a more in-depth understanding of the CLEAR items, as well as concrete examples for reporting and evaluating radiomic research.
Key points
• As a complementary effort to CLEAR, this international collaborative effort aims to assist authors in reporting their radiomics research, as well as editors and reviewers in reviewing radiomics manuscripts.
• Based on positive examples from the literature selected by the EuSoMII Radiomics Auditing Group, each CLEAR item explanation was further elaborated in CLEAR-E3.
• The resulting explanation and elaboration document with examples can be accessed at https://radiomic.github.io/CLEAR-E3/.
Graphical Abstract
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Introduction
Radiomics is a collection of quantitative medical image analysis methods based on sophisticated mathematical approaches for extraction and analysis of biomarkers (i.e., features) to enhance the information currently accessible to physicians [1]. The key premise of radiomics is that disease-specific complex patterns and biological insights are present in medical images but undetectable to the human eye. These are usually revealed with the use of machine learning techniques [2, 3]. Through various hand-crafted and deep learning-based radiomic approaches, numerous research studies have demonstrated encouraging results for a range of prediction tasks, such as diagnosis [4], genomics [5], and clinical outcomes [6, 7].
As of March 2024, a simple search for “radiomics” in PubMed returns around 10,800 papers, of which more than half were published after 2022. This finding provides further evidence that the radiomics literature has been expanding at an exponential rate, with a reported annual growth rate of 29.1% and a doubling time of 2.7 years for the time period between 2017 and 2021 [8]. Despite such an increase in research output over recent years, the vast majority of radiomic studies unfortunately failed to be clinically useful, which is also clearly noticeable in the US Food and Drug Administration clearance of tools related to radiomics [9, 10]. Similarly, a recent overview of the meta-analyses concluded that more evidence is still needed to support the clinical translation of radiomics research [11]. Interestingly, despite these shortcomings, papers with positive results continue to heavily predominate in the radiomics literature [12,13,14], with very few studies presenting negative results [15, 16].
The current lack of clinical translation might be attributable to the fact that radiomics is a complex multi-step process that leads to numerous challenges related to robustness, reproducibility, replicability, and generalizability [17,18,19]. The key to bridging the current translational gap between exploratory radiomics research and clinically validated decision-making tools may lie in the standardization of the radiomics pipeline including its reporting [9, 20]. With its recent extended effort on convolutional filters [21], the Image Biomarker Standardisation Initiative (IBSI) has been a significant driving force to standardize feature computation-related aspects [22]. Regarding transparent reporting and methodological quality evaluation, new consensus guidelines have been published and endorsed by prominent imaging societies [23, 24].
Lack of shared standards and practices in radiomics, especially in the years following its introduction in the research domain may have fostered poor methodological quality [25]. On the other hand, increased leeway in study design might be considered acceptable to some level to develop or test novel methods that can be more robustly validated by future research. Transparent reporting via a systematic approach (i.e., based on established guidelines) has been considered essential to improve the reproducibility of scientific research [26]. According to a recent metaresearch, reporting guidelines and quality scoring tools are not frequently used for self-reporting purposes in radiomics research [27], similar to the findings in another work on an AI reporting guideline for medical imaging [28] and to those related to other general reporting guidelines. This might be related to limited encouragement and endorsement of reporting guidelines by journals for several reasons and to some reluctance of authors to show the limited quality [29, 30].
Developed with a modified Delphi method and published in the first half of 2023, the CheckList for EvaluAtion of Radiomics research (CLEAR) is a 58-item checklist designed specifically for radiomics research [23] (Table 1). It has been endorsed by the European Society of Radiology and the European Society of Medical Imaging Informatics (EuSoMII).
One of the major strengths that distinguishes CLEAR from the other radiomic guidelines is its systematic and transparent development methodology [31]. The CLEAR checklist can be accessed at https://clearchecklist.github.io/clear_checklist/CLEAR.html[23]. The overall aim of CLEAR is to improve the completeness and transparency of radiomics research presentation. Consulting this document at the beginning of radiomics research planning may also be useful in improving study design. On the other hand, it should be used after the research is completed for better documentation of the study methods during manuscript preparation. Similarly, it is useful not only for authors but also for reviewers in the peer review process. However, CLEAR is intended to be a reporting guideline rather than a quality assessment tool. For the latter task, the METhodological RadiomICs Score (METRICS), also endorsed by EuSoMII, has been developed to provide a structured approach for post hoc research evaluation [24].
The value of explanation and elaboration documents for reporting guidelines has been previously highlighted [32]. Well-known checklists have also stressed the importance of publishing explanation and elaboration documents [33,34,35]. Lack of explanation and elaboration documents may result in divergences of opinion over the meaning of specific items, which might be an obstacle to the implementation of reporting checklists [27, 28, 30].
Here, we present an explanation and elaboration paper for CLEAR, including positive examples for each item from published original articles on radiomics. The resulting collection, CLEAR Explanation and Elaboration with Examples (CLEAR-E3), is intended to improve CLEAR’s adoption and dissemination, which will ultimately contribute to more transparent, complete, and accurate reporting of radiomics research.
Development of CLEAR-E3
Contributor recruitment and task definition
The project was proposed by the lead author (B.K.) and members recruited among the EuSoMII Radiomics Auditing Group through an open call.
Each contributor was assigned to 4 to 5 CLEAR items, with specific instructions aimed at ensuring diversity, relevance, and adherence to predefined open-access standards. All contributors were instructed to provide at least two distinct examples for each item, encompassing different aspects and sourced from various papers, without restriction to a specific database (e.g., PubMed, Scopus, and/or Web of Science). The focus was on reporting practices in accordance with CLEAR, rather than assessing the overall methodological quality of the selected articles. Additionally, they were encouraged to explore examples beyond textual content, including tables and figures if necessary. Examples were required to be sourced from articles with an appropriate Creative Commons (CC) license, allowing the reuse of the material with proper citation and attribution of the work. To prevent any misuse of licensed work or copyright infringement, all CC license attributes of the source papers were carefully evaluated for inclusion in this work. Each example was required to be accompanied by a brief explanation and elaboration paragraph that incorporates the theoretical basis of the CLEAR item with references to relevant literature when applicable.
Presentation of examples
Adhering to the principles of the CC license type associated with the referenced paper, certain examples were presented verbatim, while others were modified for enhanced clarity, with omitted text indicated by bracketed ellipses (i.e., “[…]”). Citations in the original CLEAR item explanations and examples were intentionally excluded to avoid potential confusion. To avoid any potential copyright infringement, one exemption for excluding these citations was given to papers with a “nonderivative” attribute in their CC license. These papers were carefully chosen to include excerpts that do not contain any citations.
Finalized CLEAR-E3
To allow easier navigation, the resulting CLEAR-E3 can be accessed at https://radiomic.github.io/CLEAR-E3/. Homepage of the CLEAR-E3’s interactive website is shown in Fig. 1. QR code displayed in Fig. 2 can be used to access the mobile-friendly website of CLEAR-E3. Figure 3 presents an example from CLEAR-E3.
Recommendations for using CLEAR-E3 along with CLEAR
The CLEAR-E3 team strongly advises that users of CLEAR and CLEAR-E3 carefully consider the following recommendations (Fig. 4).
Understand the primary purpose of the tools
Both CLEAR and CLEAR-E3 are tools designed to improve thoroughness and clarity when reporting radiomics research. Conversely, these documents are not intended to judge the quality of the methods used in the radiomic papers. For the latter case, the use of quality assessment tools (i.e., METRICS) is strongly recommended [24]. Furthermore, CLEAR-E3 is intended as a complement and not a substitute for CLEAR. We recommend using the CLEAR-E3 along with CLEAR whenever deemed necessary.
Consult these tools during research planning
Consulting this document when initially designing radiomics research could help researchers in systematically gathering important information, as well as prevent issues that limit the downstream applicability and reliability of the radiomics pipeline.
Keep in mind the scope of the examples
For each CLEAR item, we aimed to present at least two examples from published articles. The examples in CLEAR-E3 are intended to represent appropriate reporting for the corresponding item. Although CLEAR-E3 is expected to offer valuable guidance, by no means do we believe that optimal reporting of radiomics research is solely limited to the examples provided in CLEAR-E3. Furthermore, while these included examples from the literature represent good reporting practice for the corresponding item, this does not guarantee overall CLEAR adherence or methodological quality of the referenced study as a whole.
Report comprehensively
Authors should adhere to all of the CLEAR items and not just partially. Also, reporting should cover as much of what is mentioned in an item definition as possible instead of just partially. If it is not possible to report all the important information, e.g., due to the word limit, researchers can summarize the important information in a table or figure and/or add more information to the supplementary material. Although not mandatory, reporting in line with the order of CLEAR items would make it easier to identify the key information being reported such as in systematic reviews.
Self-report appropriately
When self-reporting items using the CLEAR checklist, “not applicable” should always be used appropriately, particularly when the item definition hints at its use by containing an “if applicable” statement. If an item does not include an “if applicable” statement then it should be conclusively answered with “yes” or “no” (i.e., reported or not reported), as this may otherwise lead to incorrect judgments among evaluators [27, 28].
Submit the filled-out checklist
Authors of radiomics research studies should include a completed CLEAR checklist with their submission to help the editorial process, the peer reviewers, and finally the readers and systematic reviewers of these studies. We suggest using a section heading or its abbreviation along with a paragraph number (e.g., for the second paragraph of the introduction; Intro p2; for the third paragraph of methods, Met p3,) instead of line or page numbers, since changing the page or line number during or after the publishing process can make it hard to keep track of them.
Aid dissemination of transparent reporting practice guidelines
To facilitate dissemination and appropriate use of CLEAR, we recommend that authors refer this open-access CLEAR-E3 publication along with the original CLEAR statement. Moreover, we recommend reviewers assess whether authors have appropriately cited these publications in their articles to reinforce the significance of and to help disseminate transparent reporting practices. When reporting articles, it is strongly advised to add the following statement: “This study was prepared in accordance with CLEAR and CLEAR-E3,” with citations to CLEAR and CLEAR-E3 publications.
Final remarks
While CLEAR was intended to enhance the quality of reporting in radiomics research, its actual impact will only become apparent in the future. While formal evaluation studies are warranted, our focus has been on developing strategies to improve the appropriate use of CLEAR within the radiomics community. This initiative has been epitomized in the international collaborative effort to create CLEAR-E3, aimed to facilitate the understanding of CLEAR items. A key difference of CLEAR-E3 from similar previous explanation and elaboration papers is its website. This interactive website greatly improves navigation and eases the use of CLEAR-E3, given the high number of items in CLEAR. Furthermore, we offered collective recommendations for the effective use of CLEAR-E3 as a complement to the CLEAR statement. We welcome feedback from the readers to continuously refine and enhance CLEAR and CLEAR-E3, ensuring their continued efficacy in advancing reporting standards in radiomics research.
Availability of data and materials
All data and materials are shown in the submission.
Abbreviations
- CC:
-
Creative Commons
- CLEAR:
-
CheckList for EvaluAtion of Radiomics research
- CLEAR-E3:
-
Explanation and Elaboration with Examples for CLEAR
- EuSoMII:
-
European Society of Medical Imaging Informatics
- METRICS:
-
METhodological RadiomICs Score
References
Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577. https://doi.org/10.1148/radiol.2015151169
Koçak B, Durmaz EŞ, Ateş E, Kılıçkesmez Ö (2019) Radiomics with artificial intelligence: a practical guide for beginners. Diagn Interv Radiol 25:485–495. https://doi.org/10.5152/dir.2019.19321
van Timmeren JE, Cester D, Tanadini-Lang S et al (2020) Radiomics in medical imaging—“how-to” guide and critical reflection. Insights Imaging 11:91. https://doi.org/10.1186/s13244-020-00887-2
Liu J, Qi L, Wang Y et al (2024) Development of a combined radiomics and CT feature-based model for differentiating malignant from benign subcentimeter solid pulmonary nodules. Eur Radiol Exp 8:8. https://doi.org/10.1186/s41747-023-00400-6
Zhang L, Pan H, Liu Z et al (2023) Multicenter clinical radiomics-integrated model based on [18F]FDG PET and multi-modal MRI predict ATRX mutation status in IDH-mutant lower-grade gliomas. Eur Radiol 33:872–883. https://doi.org/10.1007/s00330-022-09043-4
Liu C, Li L, Chen X et al (2024) Intratumoral and peritumoral radiomics predict pathological response after neoadjuvant chemotherapy against advanced gastric cancer. Insights Imaging 15:23. https://doi.org/10.1186/s13244-023-01584-6
Li Q, Wang N, Wang Y et al (2024) Intratumoral and peritumoral CT radiomics in predicting prognosis in patients with chondrosarcoma: a multicenter study. Insights Imaging 15:9. https://doi.org/10.1186/s13244-023-01582-8
Kocak B, Baessler B, Cuocolo R et al (2023) Trends and statistics of artificial intelligence and radiomics research in Radiology, Nuclear Medicine, and Medical Imaging: bibliometric analysis. Eur Radiol 33:7542–7555. https://doi.org/10.1007/s00330-023-09772-0
Huisman M, Akinci D’Antonoli T (2024) What a radiologist needs to know about radiomics, standardization, and reproducibility. Radiology 310:e232459. https://doi.org/10.1148/radiol.232459
Huang EP, O’Connor JPB, McShane LM et al (2023) Criteria for the translation of radiomics into clinically useful tests. Nat Rev Clin Oncol 20:69–82. https://doi.org/10.1038/s41571-022-00707-0
Zhong J, Lu J, Zhang G et al (2023) An overview of meta-analyses on radiomics: more evidence is needed to support clinical translation. Insights Imaging 14:111. https://doi.org/10.1186/s13244-023-01437-2
Kocak B, Bulut E, Bayrak ON et al (2023) NEgatiVE results in Radiomics research (NEVER): a meta-research study of publication bias in leading radiology journals. Eur J Radiol 163:110830. https://doi.org/10.1016/j.ejrad.2023.110830
Buvat I, Orlhac F (2019) The Dark Side of Radiomics: On the Paramount Importance of Publishing Negative Results. J Nucl Med 60:1543–1544. https://doi.org/10.2967/jnumed.119.235325
Song J, Yin Y, Wang H et al (2020) A review of original articles published in the emerging field of radiomics. Eur J Radiol 127:108991. https://doi.org/10.1016/j.ejrad.2020.108991
Peisen F, Gerken A, Hering A et al (2023) Can Whole-body baseline CT radiomics add information to the prediction of best response, progression-free survival, and overall survival of stage IV melanoma patients receiving first-line targeted therapy: a retrospective register study. Diagnostics 13:3210. https://doi.org/10.3390/diagnostics13203210
van der Reijd DJ, Guerendel C, Staal FCR et al (2023) Independent validation of CT radiomics models in colorectal liver metastases: predicting local tumour progression after ablation. Eur Radiol. https://doi.org/10.1007/s00330-023-10417-5
Fusco R, Granata V, Grazzini G et al (2022) Radiomics in medical imaging: pitfalls and challenges in clinical management. Jpn J Radiol 40:919–929. https://doi.org/10.1007/s11604-022-01271-4
Lafata KJ, Wang Y, Konkel B et al (2022) Radiomics: a primer on high-throughput image phenotyping. Abdom Radiol (NY) 47:2986–3002. https://doi.org/10.1007/s00261-021-03254-x
Rizzo S, Botta F, Raimondi S et al (2018) Radiomics: the facts and the challenges of image analysis. Eur Radiol Exp 2:36. https://doi.org/10.1186/s41747-018-0068-z
Zwanenburg A (2023) Standardisation and harmonisation efforts in quantitative imaging. Eur Radiol 33:8842–8843. https://doi.org/10.1007/s00330-023-09921-5
Whybra P, Zwanenburg A, Andrearczyk V et al (2024) The Image Biomarker Standardization Initiative: standardized convolutional filters for reproducible radiomics and enhanced clinical insights. Radiology 310:e231319. https://doi.org/10.1148/radiol.231319
Zwanenburg A, Vallières M, Abdalah MA et al (2020) The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology 295:328–338. https://doi.org/10.1148/radiol.2020191145
Kocak B, Baessler B, Bakas S et al (2023) CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. Insights Imaging 14:75. https://doi.org/10.1186/s13244-023-01415-8
Kocak B, Akinci D’Antonoli T, Mercaldo N et al (2024) METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII. Insights Imaging 15:8. https://doi.org/10.1186/s13244-023-01572-w
Spadarella G, Stanzione A, Akinci D’Antonoli T et al (2023) Systematic review of the radiomics quality score applications: an EuSoMII Radiomics Auditing Group Initiative. Eur Radiol 33:1884–1894. https://doi.org/10.1007/s00330-022-09187-3
Simera I, Moher D, Hirst A et al (2010) Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med 8:24. https://doi.org/10.1186/1741-7015-8-24
Kocak B, Akinci D’Antonoli T, Ates Kus E et al (2024) Self-reported checklists and quality scoring tools in radiomics: a meta-research. Eur Radiol. https://doi.org/10.1007/s00330-023-10487-5
Kocak B, Keles A, Akinci D’Antonoli T (2023) Self-reporting with checklists in artificial intelligence research on medical imaging: a systematic review based on citations of CLAIM. Eur Radiol. https://doi.org/10.1007/s00330-023-10243-9
Zhong J, Xing Y, Lu J et al (2023) The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study. BMC Med Res Methodol 23:292. https://doi.org/10.1186/s12874-023-02117-x
Koçak B, Keleş A, Köse F (2024) Meta-research on reporting guidelines for artificial intelligence: are authors and reviewers encouraged enough in radiology, nuclear medicine, and medical imaging journals? Diagn Interv Radiol. https://doi.org/10.4274/dir.2024.232604
Lambin P, Leijenaar RTH, Deist TM et al (2017) Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 14:749–762. https://doi.org/10.1038/nrclinonc.2017.141
Moher D, Schulz KF, Simera I, Altman DG (2010) Guidance for developers of health research reporting guidelines. PLoS Med 7:e1000217. https://doi.org/10.1371/journal.pmed.1000217
Moher D, Hopewell S, Schulz KF et al (2012) CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg 10:28–55. https://doi.org/10.1016/j.ijsu.2011.10.001
Chan A-W, Tetzlaff JM, Gøtzsche PC et al (2013) SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 346:e7586. https://doi.org/10.1136/bmj.e7586
Moons KGM, Altman DG, Reitsma JB et al (2015) Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162:W1-73. https://doi.org/10.7326/M14-0698
Acknowledgements
This study was endorsed by the European Society of Medical Imaging Informatics (EuSoMII).
Funding
This study has not directly received any funding. RoCa: co-funding by the European Union - FESR or FSE, PON Research and Innovation 2014-2020 - DM 1062/2021.
Author information
Authors and Affiliations
Contributions
BK and ReCu organized and supervised the study. All other authors collected examples from the literature and wrote explanations and elaborations, which were further edited by BK and ReCu. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable
Competing interests
FV: travel grant from GE Healthcare and Bracco Imaging; lecture fee from Guerbet; occasional consultant for Bracco Imaging. RoCa: support for attending meetings from Bracco and Bayer; research collaboration with Siemens Healthineers. AS and ReCu are scientific editorial board members of European Radiology Experimental; they have not taken part in the review and decision process of this paper. The other authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kocak, B., Borgheresi, A., Ponsiglione, A. et al. Explanation and Elaboration with Examples for CLEAR (CLEAR-E3): an EuSoMII Radiomics Auditing Group Initiative. Eur Radiol Exp 8, 72 (2024). https://doi.org/10.1186/s41747-024-00471-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s41747-024-00471-z