Abstract
As part of their daily workload, clinicians examine patient cases in the process of formulating a diagnosis. These large multimodal patient datasets stored in hospitals could help in retrieving relevant information for a differential diagnosis , but these are currently not fully exploited. The VISCERAL Retrieval Benchmark organized a medical case-based retrieval algorithm evaluation using multimodal (text and visual) data from radiology reports. The common dataset contained patient CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) scans and RadLex term anatomy–pathology lists from the radiology reports. A content-based retrieval method for medical cases that uses both textual and visual features is presented. It defines a weighting scheme that combines the anatomical and clinical correlations of the RadLex terms with local texture features obtained from the region of interest in the query cases. The visual features are computed using a 3D Riesz wavelet texture analysis performed on a common spatial domain to compare the images in the analogous anatomical regions of interest in the dataset images. The proposed method obtained the best mean average precision in 6 out of 10 topics and the highest number of relevant cases retrieved in the benchmark. Obtaining robust results for various pathologies, it could further be developed to perform medical case-based retrieval on large multimodal clinical datasets.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Arsigny V, Fillard P, Pennec X, Ayache N (2006) Log-Euclidean metrics for fast and simple calculus on diffusion tensors. Magn Reson Med 56(2):411–421
Cirujeda P, Mateo X, Dicente Y, Binefa X (2014) MCOV: a covariance descriptor for fusion of texture and shape features in 3D point clouds. In: International conference on 3D vision (3DV)
Depeursinge A, Foncubierta-Rodriguez A, Ville D, Müller H (2011) Lung texture classification using locally-oriented riesz components. In: Fichtinger G, Martel A, Peters T (eds) MICCAI 2011. LNCS, vol 6893. Springer, Heidelberg, pp 231–238. doi:10.1007/978-3-642-23626-6_29
García Seco de Herrera A, Kalpathy-Cramer J, Demner Fushman D, Antani S, Müller H (2013) Overview of the ImageCLEF 2013 medical tasks. In: Working notes of CLEF 2013 (Cross Language Evaluation Forum)
García Seco de Herrera A, Foncubierta-Rodríguez A, Müller H (2015) Medical case-based retrieval: integrating query MeSH terms for query-adaptive multi-modal fusion. In: SPIE medical imaging. International Society for Optics and Photonics
García Seco de Herrera A, Müller H, Bromuri S (2015) Overview of the ImageCLEF 2015 medical classification task. In: Working notes of CLEF 2015 (Cross Language Evaluation Forum)
Hanbury A, Müller H, Langs G, Weber MA, Menze BH, Fernandez TS (2012) Bringing the algorithms to the data: cloud–based benchmarking for medical image analysis. In: Catarci T, Forner P, Hiemstra D, Peñas A, Santucci G (eds) CLEF 2012. LNCS, vol 7488. Springer, Heidelberg, pp 24–29. doi:10.1007/978-3-642-33247-0_3
Jiménez del Toro OA, Foncubierta-Rodríguez A, Vargas Gómez MI, Müller H, Depeursinge A (2013) Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons. In: Mori K, Sakuma I, Sato Y, Barillot C, Navab N (eds) MICCAI 2013. LNCS, vol 8150. Springer, Heidelberg, pp 353–360. doi:10.1007/978-3-642-40763-5_44
Jiménez del Toro OA, Foncubierta-Rodríguez A, Depeursinge A, Müller H (2015) Texture classification of anatomical structures in CT using a context-free machine learning approach. In: SPIE medical imaging 2015
Jiménez-del-Toro OA, Cirujeda P, Cid YD, Müller H (2015) RadLex terms and local texture features for multimodal medical case retrieval. In: Müller H, Jimenez del Toro OA, Hanbury A, Langs G, Foncubierta Rodríguez A (eds) Multimodal retrieval in the medical domain. LNCS, vol 9059. Springer, Cham, pp 144–152. doi:10.1007/978-3-319-24471-6_14
Jiménez-del-Toro OA, Hanbury A, Langs G, Foncubierta–Rodríguez A, Müller H (2015) Overview of the VISCERAL retrieval benchmark 2015. In: Müller H, Jimenez del Toro OA, Hanbury A, Langs G, Foncubierta Rodríguez A (eds) Multimodal retrieval in the medical domain. LNCS, vol 9059. Springer, Cham, pp 115–123. doi:10.1007/978-3-319-24471-6_10
Klein S, Pluim JP, Staring M, Viergever MA (2009) Adaptive stochastic gradient descent optimisation for image registration. Int J Comput Vis 81(3):227–239
Klein S, Staring M, Murphy K, Viergever MA, Pluim JP (2010) Elastix: a toolbox for intensity-based medical image registration. IEEE Trans Med Imaging 29(1):196–205
Kurtz C, Depeursinge A, Napel S, Beaulieu CF, Rubin DL (2014) On combining visual and ontological similarities for medical image retrieval applications. Med Image Anal 18(7):1082–1100
Langlotz CP (2006) RadLex: a new method for indexing online educational materials. Radiographics 26(6):1595–1597
Langs G, Hanbury A, Menze B, Müller H (2013) VISCERAL: towards large data in medical imaging — challenges and directions. In: Greenspan H, Müller H, Syeda-Mahmood T (eds) MCBR-CDS 2012. LNCS, vol 7723. Springer, Heidelberg, pp 92–98. doi:10.1007/978-3-642-36678-9_9
Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medicine-clinical benefits and future directions. Int J Med Inform 73(1):1–23
Rubin GD (2000) Data explosion: the challenge of multidetector-row CT. Eur J Radiol 36(2):74–80
Rubin D, Napel S (2010) Imaging informatics: toward capturing and processing semantic information in radiology images. Yearb Med Inform 2010:34–42
Spanier AB, Joskowicz L (2015) Medical case-based retrieval of patient records using the RadLex hierarchical lexicon. In: Müller H, Jimenez del Toro OA, Hanbury A, Langs G, Foncubierta Rodríguez A (eds) Multimodal retrieval in the medical domain. LNCS, vol 9059. Springer, Cham, pp 129–138. doi:10.1007/978-3-319-24471-6_12
Unser M, Van De Ville D (2010) Wavelet steerability and the higher-order Riesz transform. IEEE Trans Image Process 19(3):636–652
Acknowledgements
This work was supported by the EU in FP7 through VISCERAL (318068), Khresmoi (257528) and the Swiss National National Foundation (SNF grant 205320–141300/1).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is distributed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (http://creativecommons.org/licenses/by-nc/2.5/), which permits any noncommercial use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this book are included in the work’s Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work’s Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.
Copyright information
© 2017 The Author(s)
About this chapter
Cite this chapter
Jimenez-del-Toro, O., Cirujeda, P., Müller, H. (2017). Combining Radiology Images and Clinical Metadata for Multimodal Medical Case-Based Retrieval. In: Hanbury, A., Müller, H., Langs, G. (eds) Cloud-Based Benchmarking of Medical Image Analysis. Springer, Cham. https://doi.org/10.1007/978-3-319-49644-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-49644-3_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-49642-9
Online ISBN: 978-3-319-49644-3
eBook Packages: Computer ScienceComputer Science (R0)