Abstract
As digital imaging technology advances, gigapixel or terapixel super resolution microscopic images become available. We have built a real time virtual microscopy technique for interactive analysis of super high resolution microscopic images over internet on standard desktops, laptops or mobile devices. The presented virtual microscopy technique is demonstrated to perform as fast as using a microscopy locally without any delay to assess gigapixel ultra high resolution image data through wired or wireless internet by a Tablet or a standard PC. More importantly, the presented technology enables analysis of super high resolution microscopic image across sites and time and allows multi-person analysis at the same time, which greatly speed up data analysis process and reduces miscommunication among scientists and doctors. A web site has been created for illustration purposes. (http://www-o.ntust.edu.tw/~cweiwang/VirtualMicroscopy).
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Introduction
Super high-resolution digital microscopic images are necessary for accurate diagnosis and analysis in many applications such as neurobiology and histopathology and it is expected to play a revolutionary role to describe high-resolution anatomy of large biological specimens. Recent computer vision studies have been working on capturing and reconstructing large-scale images. Saalfeld et al.1 introduced an automatic registration method for stitching tiled serial section microscopic stacks together. There have been some works on interactive display of large-scale image. Kopf et al.2 introduced a method for capturing and interactively displayed gigapixel images, which was released as Microsoft HDview3. Leica Aperio WebScope4,5 has been released as a web based and platform independent digital slide viewer. However, the limitation on the bandwidth to a server over wired or wireless internet, including 3G or 4G mobile telecommunication technologies, makes the retrieval of an extremely large scale image, even compressed, very time consuming6 and existing techniques such as Microsoft HDview and Leica Aperio WebScope suffer from significant time delay and image blurriness (see Fig. 1, supplementary video 1 and supplementary video 2). In addition, none of the aforementioned approaches enables real time interaction over internet as using a microscope.
High-resolution image database is extremely large in size, making them expensive to transmit and process and difficult to scan and search for specific contents. Due to the enormous size of microscopic sections, it is difficult to handle the images efficiently7. In evaluation of two state of the art systems3,4, all suffer from unsatisfactory time delay. Although whole-slide digital imaging has been receiving increased attention in anatomic pathology, due to technology limitation, it is still not widely used in a diagnostic setting8. Challenges in high-resolution whole slide images include not only making the image quality good enough but also the interactive visualization speed fast enough to be comparable to the experience of using a microscope9.
In this work, we have built a real time virtual microscopy technique for interactive analysis of super high resolution microscopic images over internet. The presented virtual microscopy technique is demonstrated to perform as fast as using a microscopy locally without any delay. More importantly, the presented technology enables analysis of super high resolution microscopic image across regions and time and allows multi-person analysis at the same time, which greatly speed up data analysis process and reduces miscommunication among scientists and doctors. In comparison with the state of the art methods, including Microsoft HDview3 and Leica Aperio WebScope4, the presented technique is significantly faster (p < 0.001 according to Tukey’s HSD and LSD tests). A virtual microscopy demonstration on a standard PC through wired internet is given in the supplementary video 3 and a demonstration on a Tablet through wireless internet is given in the supplementary video 4.
Results
Two state of the art systems, including Microsoft HDview3 and Leica Aperio WebScope4, were adopted as benchmark techniques for comparison. As it is impossible to obtain the same images to test in each benchmark system, images in the benchmark systems with the closest dimension were chosen to test and the size of the super resolution images in the individual benchmark systems is 3.7 GB (the largest image in Microsoft HDview Demo Website) and 12.78 GB (Leica Aperio WebScope), respectively. In evaluation of the proposed method, a full phase lung tissue section was digitalized at 40× magnification using Leica Aperio Scanscope CS2 and the size of the resulting super resolution microscopic image is 12.8 GB with image dimension 84570 × 54248.
In evaluation, four quantitative measurements were set up by adopting two kinds of data assessing tests and two different client site settings. Regarding the data assessing tests, one data assessing test is in Z direction by zooming in the image four times and the other data assessing test is in X direction by dragging the image from right to left four times. The two client site settings include a standard PC through wired internet (CPU: Intel Core i7-2600 3.4 GHz, RAM: 12 GB, OS: Windows 7 64-bit) and a Tablet through wireless 3G telecommunication (Samsung Note 10.1, CPU: Exynos4412 1.4 GHz, RAM: 2 GB, OS: Android 4.1.1). For the server site of the presented system, instead of using a super computer, a standard PC (CPU: Intel Core i7-2600 3.4 GHz, RAM: 12 GB) is used as the hardware platform to test the performance of the presented method. Statistical analysis was performed using SPSS software10 and the quantitative results were analyzed with the Tukey’s Honestly Significant Difference test (Tukey’s HSD) and the Fisher’s Least Square Difference test (LSD) with the significance level 0.001.
Z-direction test using PC with wired internet
Table 1 presents the rendering time by Z-direction test using a PC with wired internet and a statistical analysis using the Tukey’s HSD and the LSD tests is given in Table 2. The experimental results show that the proposed real time method achieves 0.392 seconds averaged rendering time and significantly outperforms the existing methods on Tukey’s HSD and LSD tests (P ≤ 0.001). On average, it costs Microsoft HDview and Leica Aperio WebScope 36.897 and 7.978 seconds, respectively, showing that the existing techniques perform poorly and suffer from serious time delay. Overall, the proposed method performs 94.13 times faster than Microsoft HDview and 20.35 times faster than Leica Aperio WebScope on average. A box plot of the quantitative evaluation result is provided in Fig. 2, showing that the proposed method works constantly well than two existing methods overall.
X-direction test using PC with wired internet
Table 3 presents the rendering time by X-direction test using PC with wired internet. A statistical analysis using the Tukey’s HSD and the LSD tests is given in Table 4. The experimental results show that the proposed real time method achieves 0.363 seconds averaged rendering time and significantly outperforms the existing methods on Tukey’s HSD and LSD tests (P ≤ 0.001). On average, it costs Microsoft HDview and Leica Aperio WebScope 32.673 and 5.317 seconds, respectively, showing that the existing techniques perform poorly and suffer from serious time delay. Overall, the proposed method performs 90.01 times faster than Microsoft HDview and 14.67 times faster than Leica Aperio WebScope on average. A box plot of the quantitative evaluation result is provided in Fig. 3, showing that the proposed method works constantly well than the benchmark methods overall.
For the third and fourth quantitative evaluation, a Tablet with wireless internet is adopted as the hardware platform. As Microsoft HDview does not support the Android system, Microsoft HDview can not function on the Tablet (Samsung Note 10.1) and therefore comparison is performed between the presented system and the Leica Aperio WebScope.
Z-direction test using a Tablet with wireless internet
Table 5 presents the rendering time by Z-direction test using a Tablet with wireless internet and a statistical analysis using the Tukey’s HSD and the LSD tests is given in Table 6. The experimental results show that the proposed method achieves 1.39 seconds averaged rendering time and significantly outperforms the benchmark method on Tukey’s HSD and LSD tests (P ≤ 0.001). On average, it costs Leica Aperio WebScope 13.85 seconds, showing that the existing technique performs poorly and suffers from serious time delay. Overall, the proposed method performs 9.93 times faster than Leica Aperio WebScope on average. A box plot of the quantitative evaluation result is provided in Fig. 4, showing that the proposed method works constantly well than Leica Aperio WebScope overall.
X-direction test using a Tablet with wireless internet
Table 7 presents the rendering time by X-direction test using a Tablet with wireless internet and a statistical analysis using the Tukey’s HSD and the LSD tests is given in Table 8. The experimental results show that the proposed method achieves 0.88 seconds averaged rendering time and significantly outperforms the benchmark method on Tukey’s HSD and LSD tests (P ≤ 0.001). On average, it costs Leica Aperio WebScope 6.72 seconds, showing that the existing technique performs poorly and suffers from serious time delay. Overall, the proposed method performs 7.6 times faster than Leica Aperio WebScope on average. A box plot of the quantitative evaluation result is provided in Fig. 5, showing that the proposed method works constantly well than Leica Aperio WebScope overall.
Based on the experimental results, Table 9 further compares the ratios of the averaged rendering time by the proposed method to the benchmark techniques. In comparison to Microsoft HDview, the presented method is 94.13 times faster in the Z-direction test and 90.01 times faster in the X-direction test using a standard PC. On the other hand, in comparison to Leica Aperio WebScope, using a standard PC as the client site platform, the presented work is 20.35 times faster in the Z-direction test and 14.67 times faster in the X-direction test. Using a tablet with wireless internet, the presented work is 9.933 times faster in the Z-direction test and 7.603 times faster in the X-direction test than Leica Aperio WebScope.
Discussion
As digital imaging technology advances, gigapixel or terapixel super resolution microscopic images become more and more common. However, due to the enormous size of these data, data analysis across sites or time is difficult because real time interactive accessing super resolution microscopic images over internet is challenging. Existing state of the art techniques such as Microsoft HDview and Leica Aperio WebScope suffer from significant time delay. We have developed a virtual microscopy method enabling real time interactive analysis of large-scale microscopic images through wireless or wired internet on standard desktops, laptops and mobile devices. VirtualMicroscopy enables analysis of super high resolution microscopic image across sites and time and allows multi-person analysis at the same time, which greatly speed up data analysis process and reduces miscommunication among scientists and doctors. A web site has been created for illustration purposes. (http://www-o.ntust.edu.tw/cweiwang/VirtualMicroscopy).
Methods
Different from the commonly adopted pyramid data structure11,12,13,14,15,16 for large-scale images, a highly efficient quad-tree based data storage and accessing strategy is built in VirtualMicroscopy, enabling interactively accessing ultra-high resolution gigapixel or terapixel microscopic images, accessing gigapixel or terapixel data over internet in real time and interaction over internet with limited bandwidth in real time. In the common pyramid data structure11,12,13,14,15,16, a quad-tree model17,18 is used as the fundamental data structure. A quad-tree models a two-dimensional region by recursively dividing it into quadrants in the multi-resolution image tiers d. These quadrants are referred to as the leaves of the parent tile unit. Therefore, each parent tile unit has up to four leaf tile units. However, this model is a waste of time when seeking a unit locating in deep tiers. The search time of a tile unit in tier d requires ld, where l is set as four (quad-tree based), as seen in Fig. 6(a). Thus, we have designed a fast and efficient data structure for large data applications. The seek time of the proposed strategy is significantly reduced. In comparison to the existing data structure costing ld times, the seeking time of the presented architecture is at most K as seen in Fig. 6(b), where K = 28. For example, if the system is searching a unit in tier 11, d = 11, existing methods will take 411 times. In comparison, the presented technique only takes at most 28 times. The proposed data model for large data processing is presented as follows.
Quickly data accessing strategy
The core idea behind the proposed framework is to access the regions of interest (ROI) directly and the neighboring tile units are saved in the neighboring physical address. Two key elements for quickly data accessing are the index table strategy and specific storage strategy.
Index table strategy
The main advantage of the index table over other data structures is speed. This advantage is more apparent when the number of tile units is large. Each tile unit is mapped into specific index code in the range 1 to γ, where γ denotes the total amount of tile units. The mapping is called an index function, which ideally should be simple to compute and should ensure that any two distinct tile units get different index codes. Then, the index codes are used to quickly locate the tile units without having to access each tier in the storage architecture.
The numbering of the tile name code is zero-based, therefore the smallest tier is represented by one tile unit that is at most ST × ST pixels with the tile code “0-0-0” and the index code “1”. Then, the codes for the index individual tiles can be generated as follows.
Name code: z − y − x, where z equals d, x is the number of the tile unit from left to right and y is the number of the tile unit from top to bottom.
Specific storage strategy
The goal of the specific storage strategy is that the neighboring tile units are stored in the specific set. First, the size of each set is given as an integer K, the integer K should ideally be a power of two since it is easier to access in the computer architecture. Then, the second step is the specific storage ordering. Each tile unit is stored in hard drive according to the index code. The neighboring tile units of a tile unit will be stored in the same small set (K = 28), therefore the seeking time for accessing neighboring tile units can be reduced to K.
Figure 6 shows the comparison of the proposed data structure and the existing data structure for ultra-high resolution images. For example, accessing a tile unit, which index code is 500, requires go through to tier five in the existing data structure; however, the proposed data structure only requires go through one tier for accessing the unit. According to the proposed specific storage strategy, the neighboring tile units will be stored in the same small set. On the contrary, a neighboring tile unit in the existing data structure may be sought in a larger set (i.e. with over than 210 units). It means that it is wasting of time for seeking fundamental data units, which are stored in the deep tier. The reason is that the size of a set in the proposed data structure is limited by a small number K (K = 28 is used in the experiments.). Therefore, the seeking time in a set can be reduced at most to K. In contrast to the existing data structure, the size of a set for deep tiers is larger and it requires significantly more seeking time. Table 10 presents the comparison of the proposed data structure and the existing pyramid data structure11, which is commonly used to deliver high-resolution images over the Web. In the proposed encoding and decoding algorithm, the proposed framework achieves great performance in the seeking time O(k). However, the seeking time for displaying a tile unit in the tier d requires O(ld), where the depth d ∈ [0, D − 1] in the existing data structure.
Implementation
A demo website has been created for illustrating the presented Virtual Microscopy and Fig. 7 shows the web interface of the proposed framework. Demo website: http://www-o.ntust.edu.tw/~cweiwang/VirtualMicroscopy http://www-o.ntust.edu.tw/cweiwang/.
Additional Information
How to cite this article: Wang, C.-W. et al. VirtualMicroscopy: ultra-fast interactive microscopy of gigapixel/terapixel images over internet. Sci. Rep. 5, 14069; doi: 10.1038/srep14069 (2015).
References
Saalfeld, S., Cardona, A., Hartenstein, V. & Tomancak, P. As-rigid-as-possible mosaicking and serial section registration of large ssTEM datasets. Bioinformatics 26(12), i57–i63 (2010).
Kopf, J., Uyttendaele, M., Deussen, O. & Cohen, M. Capturing and Viewing Gigapixel Images. ACM Trans. Graph. 26(3), 93:1–93:10 (2007).
Microsoft: Microsoft HD view. Available at: http://research.microsoft.com/en-us/um/redmond/groups/ivm/HDView/. (Accessed: 21st March 2015).
Aperio: WebScope Demo Site. Available at: http://images2.aperio.com/BigTIFF/BreastCancer.tif/view.apml. (Accessed: 21st March 2015).
The American Society of Dermatopathology: WebScope Quick Reference Guide. Available at: http://www.asdp.org/exam/upload_files/moc/WebScopeQuickReferenceGuide.pdf. (Accessed: 21st March 2015).
Basu, A., Minghong, P., Cheng, I. & Bates, M. Distributed retrieval of wavelet images using bandwidth monitoring. Proc. IAPR/IEEE Int. Conf. Pattern Recognition 604–607 (2002).
Jeong, W., Schneider, J., Turney, S., Faulkner-Jones, B., Meyer, D., Westermann, R., Reid, R., Lichtman, J. & Pfister, H. Interactive Histology of Large-Scale Biomedical Image Stacks. IEEE Trans. Vis. Comput. Graph. 16(6), 1386–1395 (2010).
Ho, J., Parwani, A. V., Jukic, D. M., Yagi, Y., Anthony, L. & Gilbertson, J. R. Use of whole slide imaging in surgical pathology quality assurance: design and pilot validation studies. Hum. Pathol. 37(3), 322–331 (2006).
Gilbertson, J. & Yagi, Y. Histology, imaging and new diagnostic work-flows in pathology. Diagn. Pathol. 3(Suppl), S14 (2008).
SPSS Inc. SPSS for Windows, Version 17.0. 2008. Chicago: SPSS Inc.
Rashwan, M. A., Elsherif, M. S. & Elsayad, A. M. Pyramid data structures for on-line image progressive transmission. in Proceedings of the 36th MIDWEST. SYMP. CIRCUIT. 1(1), 103–106 (1993).
Sullivan, G. J. & Baker, R. Efficient quadtree coding of images and video. IEEE Trans. Image Process. 3(3), 327–331 (2002).
Trotts, I., Mikula, S. & Jones, E. G. Interactive Visualization of Multiresolution Image Stacks in 3D. Neuroimage 35(3), 1038–1043 (2007).
Panusopone, K., Sarika, K. & Cheevasuvit, F. Progressive image transmission of variable block-size coding. IEEE Trans. Consum. Electron. 40(4), 836–841 (1994).
Catalyurek, U., Beynon, M. D., Chang, C., Kurc, T., Sussman, A. & Saltz, J. The Virtual Microscope. University of Maryland CS-TR-4405 (2002).
Guedalia, J. L. Method and System for Providing On-line Interactivity over a Server-client Network. Patent US5968120 A (1999).
Finkel, R. & Bentley, J. L. Quad Trees: A Data Structure for Retrieval on Composite Keys Acta Inform. 4(1), 19 (1974).
Samet, H. Storing a Collection of Polygons Using Quadtrees ACM Trans. Graph. 4(3), 182–222 (1985).
Acknowledgements
This work was supported by the Ministry of Science and Technology, Taiwan, under Young Scholar Research Grant (MOST-101-2628-E-011-006-MY3) and by the Ministry of Education, Taiwan under Young Scholar Research Grant to Prof Ching-Wei Wang.
Author information
Authors and Affiliations
Contributions
C.-W. designed the study and developed the method; C.-W. and C.-T. wrote the paper; C.-M. implemented the software and conducted the experiments.
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Electronic supplementary material
Rights and permissions
This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/
About this article
Cite this article
Wang, CW., Huang, CT. & Hung, CM. VirtualMicroscopy: ultra-fast interactive microscopy of gigapixel/terapixel images over internet. Sci Rep 5, 14069 (2015). https://doi.org/10.1038/srep14069
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/srep14069
- Springer Nature Limited