Abstract
The aim of this study was to measure intra- and interobserver agreement among radiologists in the assessment of pancreatic perfusion by computed tomography (CT). Thirty-nine perfusion CT scans were analyzed. The following parameters were measured by three readers: blood flow (BF), blood volume (BV), mean transit time (MTT) and time to peak (TTP). Statistical analysis was performed using the Bland-Altman method, linear mixed model analysis, and intraclass correlation coefficient (ICC). There was no significant intraobserver variability for the readers regarding BF, BV or TTP. There were session effects for BF in the pancreatic body and MTT in the pancreatic tail and whole pancreas. There were reader effects for BV in the pancreatic head, pancreatic body and whole pancreas. There were no effects for the interaction between session and reader for any perfusion parameter. ICCs showed substantial agreement for the interobserver measurements and moderate to substantial agreement for the intraobserver measurements, with the exception of MTT. In conclusion, satisfactory reproducibility of measurements was observed for TTP in all pancreatic regions, for BF in the head and BV in the tail, and these parameters seem to ensure a reasonable estimation of pancreatic perfusion.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Introduction
Ultrasound, computed tomography (CT) and magnetic resonance imaging (MRI) can be used in tissue perfusion studies1,2,3,4,5,6,7. In CT, however, there is a linear relationship between the concentration of iodinated contrast media and the recorded density in Hounsfield units8,9,10, and this could be considered the preferred technique for the acquisition of perfusion images11. Perfusion CT is a relatively recent technique and can provide qualitative and quantitative information regarding tissue perfusion parameters in a non-invasive way. In comparing perfusion CT and dynamic contrast enhanced MRI, the major drawback of the first method is the use of ionizing radiation, while the second is a more complex and time-consuming method. However, CT is faster and more available than MRI. Furthermore, there are more restrictions in MRI use comparing to CT (e.g., implanted devices, metallic foreign bodies and prostheses). Variability of biomarkers in perfusion CT and dynamic contrast enhanced MRI are similar12,13.
In 1995, Miles et al.14 conducted the first study on the feasibility of pancreatic perfusion CT. Since then, a number of studies have used CT to observe normal pancreatic perfusion values, pancreatic perfusion impairments in pancreatic and hepatic diseases and modifications in pancreatic perfusion after oncologic therapy8,11,15,16,17,18,19,20,21,22,23,24,25,26,27.
However, the effects of observer variability in diagnostic testing can have a potentially large impact28. In clinical practice, radiologist-dependent factors may contribute to measurement inconsistencies due to variations in measurement technique or experience29,30,31,32,33,34. Therefore, intra- and interobserver variability must be assessed to guarantee the accuracy of radiologic readings.
Little information is available about intra- and interobserver variability in pancreatic perfusion CT35,36. Therefore, the purpose of this study is to measure intra- and interobserver agreement among radiologists with different levels of experience while assessing pancreatic perfusion by CT.
Material and Methods
Between October 2015 and September 2016, we prospectively analyzed 12 scans from subjects who were referred for abdominal perfusion CT at the Jules Bordet Institute (Brussels, Belgium) for reasons unrelated to pancreatic symptoms or disease. Twenty-seven scans from a CT archive were also included. Informed consent was obtained from all participants that were prospectively included. The study was approved by the Ethics Committee of the Jules Bordet Institute and is in accordance with the Declaration of Helsinki. Exclusion criteria were pregnancy, history of allergic reaction to iodinated contrast media, renal insufficiency and history of pancreatic disease. Patients with altered pancreatic imaging (abnormal volume, morphology and/or composition, or focal lesions) were not excluded.
All patients were scanned in a Siemens Somatom® Force 192-slice scanner (Munich, Germany). The perfusion CT examination was performed in the interval between unenhanced and portal phases. To define a correct delay for the perfusion CT, a test phase was performed after injecting 10 mL of nonionic contrast medium (Iomeron 400), followed by a 21 mL bolus of saline solution after an 8 s delay. For this test phase, a region of interest (ROI) was set on the distal thoracic aorta and 15 images were acquired (1 every 2 s, rotation time: 0.25 s, 40 mAs, 100Kvp), so that a curve of aortic enhancement could be obtained (software DynEva®, Siemens). The time required to achieve peak aortic enhancement was used to define the delay for the perfusion CT. Next, 50 mL of nonionic contrast medium (Iomeron 400) were injected through an 18-gauge catheter into an antecubital vein at a flow rate of 4.0 mL/s, followed by a 21 mL chaser bolus of saline solution. Eighty kilovolt peak (kVp) voltage was used for the CT tube. The dynamic imaging sequence consisted of 31 acquisitions of 0.25-second duration (rotation time) at an interval of 1.5 s (cycle time), resulting in a total examination time of 45.45 s. The perfusion sequence covered a craniocaudal width of 24 cm (collimation of 48 × 1.2 mm). A portal phase was acquired with a delay of 70 s after 70 mL of contrast media was injected intravenously at the end of the perfusion CT (Table 1).
The images were analyzed by three radiologists (J.L.E. – reader 1, T.S.G. - reader 2, M.V. – reader 3) with 25, 16 and six years of experience in abdominal imaging, respectively. Each reader performed two reading sessions with at least a 24 h interval. The image data were processed on a workstation (Syngo.via®, Siemens) with commercial perfusion CT software (CT Body Perfusion, Siemens) based on the maximum slope model. Motion correction was performed by using a commercial non-rigid motion correction algorithm (Syngo®.via Body Perfusion, Siemens) to improve anatomical alignment. The following parameters were measured: blood flow (measured in mL/100 mL/min), blood volume (measured in mL/100 mL), time to peak (measured in seconds), and mean transit time (measured in seconds). Arterial input was measured by automatically placed ROI in the abdominal aorta. To obtain perfusion CT parameters, each radiologist manually drew three non-superposable circular ROI (between 1.0 and 2.0 cm²) in the head, three in the body and three in the tail of the pancreas to measure these parameters, avoiding visible vessels and ducts. The mean ROI value for each parameter in each part of the pancreas was considered for analysis. The parameters for the whole pancreas were calculated as the sum of the values of the pancreatic head, body and tail divided by three. An example of perfusion CT image processing is shown in Figs 1–3.
Statistical analysis
The statistical packages utilized were R-project and SPSS (version 18). Mean and standard deviation were used to describe the analyzed variables. Intraobserver agreement was evaluated by the Bland-Altman method and Student’s t-test for paired samples. Linear mixed model analysis was performed to determine interobserver reliability, considering session and reader effects. The significance level was set at 0.05. Intraclass correlation coefficients ICCs were also calculated to analyze both intra- and interobserver agreement, interpreted by using the following categories: 0.00–0.10 = virtually none; 0.11–0.40 = slight; 0.41–0.60 = fair; 0.61–0.80 = moderate and 0.81–1.00 = substantial agreement37.
Approval, accordance and informed consent statement
The study was approved by the Ethics Committee of the Jules Bordet Institute (Brussels, Belgium) and is in accordance with the Declaration of Helsinki. Informed consent was obtained from all participants that were prospectively included.
Results
A total of 39 patients [men: n = 21 (53.8%)] were included, with a mean age of 64 years. Seventeen patients had type 2 diabetes mellitus (DM2) and 22 did not. Two patients were excluded due to technical problems, which lead to difficulties in measuring pancreatic perfusion parameters (large ascites and improper contrast media injection).
The Bland-Altman analysis showed no significant intraobserver variability for readers 1 or 2 regarding BF, BV, MTT and TTP; for reader 3, there was significant variability for BF in the pancreatic tail and the whole pancreas (tail - mean difference: 11.49 mL/100 mL/min ± 27.6, 95% limit of agreement: −42.6 ± 65.5, p = 0.016; whole pancreas - mean difference: 6.04 mL/100 mL/min ± 13.0, 95% limit of agreement: −19.4 ± 31.5, p = 0.008) and for MTT in the pancreatic tail and whole pancreas (tail - mean difference: −0.47 s ± 1.2, 95% limit of agreement: −2.92 ± 2.0, p = 0.027; whole pancreas - mean difference: −0.26 s ± 0.8, 95% limit of agreement: −1.7 ± 1.2, p = 0.048). The ICCs showed an overall moderate to substantial agreement for the intraobserver measurements, with the exception of MTT in all pancreatic regions (Table 2). Bland-Altman plots graphics with interobserver variability between readers 1 and 2 in BF, BV, TTP and MTT in the whole pancreas are shown in Fig. 4. Table 3 summarizes the pancreatic perfusion measurements for each session and reader, as well as their respective effects and the interaction effect between session and reader. There were session effects on BF in the pancreatic body (mean difference: 4.5 mL/100 mL/min ± 2.20, p = 0.048) and on MTT in the pancreatic tail (mean difference: 0.28 s ± 0.11, p = 0.021) and the whole pancreas (mean difference: 0.18 s ± 0.06, p = 0.007). There were reader effects on BV in pancreatic head, pancreatic body and the whole pancreas. No session effects were found on BV or TTP and no reader effects were found on BF, MTT or TTP. There were no interaction effects between session and reader for any perfusion parameter. ICCs for the interobserver measurements on pancreatic perfusion CT parameters showed an overall substantial agreement, with the exception of MTT in the body, tail and whole pancreas, where it was only fair (Table 4).
Discussion
Our results showed good overall intraobserver agreement for pancreatic perfusion CT parameters, except for BF in the pancreatic tail for reader 3, who had the least experience, and for MTT in all regions of the pancreas for the three readers. No session effects were found on BV or TTP, and there were no reader effects on BF or MTT. TTP values were not significantly different between readers or reading sessions. However, there were session effects on BF in the pancreatic body and on MTT in the pancreatic tail and the whole pancreas. This is probably because the measurements in the body and the tail of the pancreas are more difficult to obtain due to the smaller thickness and the greater variation of the morphology of the pancreas in these regions. Thus, some variability may occur between different sessions of the same reader. Reader effects were found on BV in the pancreatic head and body and the whole pancreas. The measurements obtained by our less experienced (reader 3), seemed different than those obtained by our two most experienced readers. Possibly this is because no training sessions have been held prior to the measurements, suggesting that training sessions for inexperienced readers should be performed. Of note, BF in the head, BV in the tail, and TTP in all pancreatic regions showed good intraobserver correlation and no significant reader or session effects, which supports the use of these parameters in pancreatic perfusion CT.
Measurement reproducibility and accuracy are of particular interest in radiology, since important clinical decisions are often based on CT measurements38,39. Accordingly to McErlean et al.40, “lesion measurements on images should be accurate, reproducible, and performed in a standardized fashion with low rates of intra- and interobserver variability”. Li et al.35 reported an interobserver correlation over 0.9 for BF and BV in normal pancreas. Xie et al.36 also observed substantial agreement (0.85). Our study evaluated intraobserver agreement by two methods: Bland-Altman and mixed model analysis; interobserver correlation was evaluated by mixed model analysis, considering session and reader effects. It is important to emphasize that we obtained perfusion measurements in pancreas without lesions, which may not reflect clinical practice where perfusion CT may be used to assess focal pancreatic lesions. This issue could be addressed in forthcoming studies”.
This study has some limitations. Our results are based on the readings of only three radiologists. This small number of observers may not truly represent the community of radiologists. The measures obtained by our least experienced reader (reader 3) showed poor intraobserver agreement for BV and MTT in the pancreatic tail and the whole pancreas in the Bland-Altman analysis, suggesting that a preliminary training session, which was not performed in this study, could improve intraobserver results. However, no differences attributable to a single reader were found in mixed model analysis. Second, we only applied maximal slope model to calculate perfusion parameters. There are many other methods that can be used for this purpose, including deconvolution model and dual-input single compartment, and these algorithms are not interchangeable. Although there is no consensus about the best method, maximal slope model is used in the majority of the studies on CT perfusion. Third, some reading sessions of the same patient were performed by a reader with a 24 hour interval. In such a short interval, the intraobserver results can be affected by an observer’s recognition of an image (memory bias).
In conclusion, our data support the use of pancreatic perfusion CT by radiologists of different levels of experience. BF in the head, BV in the tail, and TTP in all pancreatic regions seem to be the best parameters to ensure a reasonably reliable reproducibility for pancreatic perfusion CT.
Data Availability
All data generated or analyzed during this study are included in this published article.
References
Doi, R. et al. Simultaneous measurement of hepatic arterial and portal venous flows by transit time ultrasonic volume flowmetry. Surgery, Gynecology & Obstetrics. 167(1), 65–9 (1988).
Kleber, G. et al. Hepatic arterial flow volume and reserve in patients with cirrhosis: use of intra-arterial Doppler and adenosine infusion. Gastroenterology. 116(4), 906–14 (1999).
Blomley, M. J. et al. Liver perfusion studied with ultrafast CT. Journal of computer assisted tomography. 19(3), 424–33 (1995).
Materne, R. et al. Assessment of hepatic perfusion parameters with dynamic MRI. Magnetic Resonance in Medicine. 47(1), 135–42 (2002).
Hirshberg, B. et al. Pancreatic perfusion of healthy individuals and type 1 diabetic patients as assessed by magnetic resonance perfusion imaging. Diabetologia. 52(8), 1561–5 (2009).
Bali, M. A. et al. Pancreatic perfusion: noninvasive quantitative assessment with dynamic contrast-enhanced MR imaging without and with secretin stimulation in healthy volunteers–initial results. Radiology. 247(1), 115–21 (2008).
Schraml, C., Schwenzer, N. F., Martirosian, P., Claussen, C. D. & Schick, F. Perfusion imaging of the pancreas using an arterial spin labeling technique. Journal of Magnetic Resonance Imaging 28(6), 1459–65 (2008).
Park, M. S. et al. Perfusion CT: noninvasive surrogate marker for stratification of pancreatic cancer response to concurrent chemo- and radiation therapy. Radiology. 250(1), 110–7 (2009).
Miles, K. A. Perfusion CT for the assessment of tumour vascularity: which protocol? The British Journal of Radiology. 76 Spec No 1:S36–42 (2003).
Kambadakone, A. R., Sharma, A., Catalano, O. A., Hahn, P. F. & Sahani, D. V. Protocol modifications for CT perfusion (CTp) examinations of abdomen-pelvic tumors: impact on radiation dose and data processing time. European Radiology. 21(6), 1293–300 (2011).
Kandel, S. et al. Whole-organ perfusion of the pancreas using dynamic volume CT in patients with primary pancreas carcinoma: acquisition technique, post-processing and initial results. European Radiology. 19(11), 2641–6 (2009).
Ng, C. S. et al. Reproducibility of Perfusion Parameters in Dynamic Contrast-Enhanced MRI of Lung and Liver Tumors: Effect on Estimates of Patient Sample Size in Clinical Trials and on Individual Patient Responses. American Journal of Roentgenology. 194(2), W134–W140 (2010).
Campbell, B. C. et al. Comparison of Computed Tomography Perfusion and Magnetic Resonance Imaging Perfusion-Diffusion Mismatch in Ischemic Stroke. Stroke. 43, 2648–2653 (2012).
Miles, K. A., Hayball, M. P. & Dixon, A. K. Measurement of human pancreatic perfusion using dynamic computed tomography with perfusion imaging. The British Journal of Radiology. 68(809), 471–5 (1995).
Cui, B., Zhao, C., He, J. & Zhang, X. Whole-organ CT perfusion imaging of the pancreas in patients with type 2 Diabetes. Chinese Journal of Medical Imaging Technology. 27(8), 1629–32 (2011).
Bize, P. E., Platon, A., Becker, C. D. & Poletti, P. A. Perfusion measurement in acute pancreatitis using dynamic perfusion MDCT. American Journal of Roentgenology. 186(1), 114–8 (2006).
D’Onofrio, M. et al. Perfusion CT can predict tumoral grading of pancreatic adenocarcinoma. European Journal of Radiology. 82(2), 227–33 (2013).
Delrue, L. et al. Tissue perfusion in pathologies of the pancreas: assessment using 128-slice computed tomography. Abdominal imaging. 37(4), 595–601 (2012).
Klauss, M. et al. Computed tomography perfusion analysis of pancreatic carcinoma. Journal of Computer Assisted Tomography. 36(2), 237–42 (2012).
Klauss, M. et al. l. Dual-energy perfusion-CT of pancreatic adenocarcinoma. European Journal of Radiology. 82(2), 208–14 (2013).
Tsushima, Y. & Kusano, S. Age-dependent decline in parenchymal perfusion in the normal human pancreas: measurement by dynamic computed tomography. Pancreas. 17(2), 148–52 (1998).
Abe, H. et al. Quantitative tissue blood flow evaluation of pancreatic tumor: comparison between xenon CT technique and perfusion CT technique based on deconvolution analysis. Radiation Medicine. 23(5), 364–70 (2005).
d’Assignies, G. et al. Pancreatic endocrine tumors: tumor blood flow assessed with perfusion CT reflects angiogenesis and correlates with prognostic factors. Radiology. 250(2), 407–16 (2009).
Watanabe, T. et al. Relationship between serum angiopoietin-2 level and perfusion CT parameters in severe acute pancreatitis. The American Journal of Gastroenterology. 106(10), 1859–61 (2011).
Kanda, T. et al. Perfusion measurement of the whole upper abdomen of patients with and without liver diseases: initial experience with 320-detector row CT. European Journal of Radiology. 81(10), 2470–5 (2012).
Motosugi, U. et al. Multi-organ perfusion CT in the abdomen using a 320-detector row CT scanner: preliminary results of perfusion changes in the liver, spleen, and pancreas of cirrhotic patients. European Journal of Radiology. 81(10), 2533–7 (2012).
Yao, J. C. et al. Perfusion computed tomography as functional biomarker in randomized run-in study of bevacizumab and everolimus in well-differentiated neuroendocrine tumors. Pancreas. 44(2), 190–7 (2015).
Mower, W. R. Evaluating bias and variability in diagnostic test reports. Annals of Emergency Medicine. 33(1), 85–91 (1999).
Lederle, F. A. et al. Variability in measurement of abdominal aortic aneurysms. Abdominal Aortic Aneurysm Detection and Management Veterans Administration Cooperative Study Group. Journal of Vascular Surgery. 21(6), 945–52 (1995).
Oxnard, G. R. et al. Variability of lung tumor measurements on repeat computed tomography scans taken within 15 minutes. Journal of Clinical Oncology. 29(23), 3114–9 (2011).
Thiesse, P. et al. Response rate accuracy in oncology trials: reasons for interobserver variability. Groupe Francais d’Immunotherapie of the Federation Nationale des Centres de Lutte Contre le Cancer. Journal of Clinical Oncology. 15(12), 3507–14 (1997).
Wormanns, D., Diederich, S., Lentschig, M. G., Winter, F. & Heindel, W. Spiral CT of pulmonary nodules: interobserver variation in assessment of lesion size. European Radiology. 10(5), 710–3 (2000).
Zhao, B. et al. Pulmonary metastases: effect of CT section thickness on measurement-initial experience. Radiology. 234(3), 934–9 (2005).
Bankier, A. A., Levine, D., Halpern, E. F. & Kressel, H. Y. Consensus interpretation in imaging research: is there a better way? Radiology. 257(1), 14–7 (2010).
Li, H. O. et al. Low-dose whole organ CT perfusion of the pancreas: preliminary study. Abdominal Imaging. 39(1), 40–7 (2014).
Xie, Q. et al. Whole-organ CT perfusion of the pancreas: impact of iterative reconstruction on image quality, perfusion parameters and radiation dose in 256-slice CT-preliminary findings. PloS One. 8(11), e80468 (2013).
Bretas, E. A. S. et al. Is liver perfusion CT reproducible? A study on intra and interobserver agreement of normal hepatic haemodynamic parameters obtained with two different software packages. British Journal of Radiology. 90, 20170214 (2017).
Edge, S. B. & Compton, C. C. The American Joint Committee on Cancer: the 7th edition of the AJCC cancer staging manual and the future of TNM. Annals of Surgical Oncology. 17(6), 1471–4 (2010).
Eisenhauer, E. A. et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur J Cancer. 45(2), 228–47 (2009).
McErlean, A. et al. Intra- and interobserver variability in CT measurements in oncology. Radiology. 269(2), 451–9 (2013).
Author information
Authors and Affiliations
Contributions
Conceptualization: T.S.G., J.E. and C.B.L. Investigation: T.S.G., J.E. and M.V. Methodology: T.S.G., V.N.H. and C.B.L. Project administration: T.S.G. and C.B.L. Supervision: C.B.L. Writing: T.S.G., V.N.H., C.B.L. Validation: T.S.G., J.E., M.V., V.N.H. and C.B.L.
Corresponding author
Ethics declarations
Competing Interests
The authors declare no competing interests.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Garcia, T.S., Engelholm, JL., Vouche, M. et al. Intra- and interobserver reproducibility of pancreatic perfusion by computed tomography. Sci Rep 9, 6043 (2019). https://doi.org/10.1038/s41598-019-42519-w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-019-42519-w
- Springer Nature Limited
This article is cited by
-
Assessing the inter-observer and intra-observer reliability of radiographic measurements for size-specific dose estimates
BMC Medical Imaging (2024)
-
Advanced imaging techniques for chronic pancreatitis
Abdominal Radiology (2020)