Abstract
The imaging rate of structured illumination microscopy (SIM) reached 188 Hz recently. As the exposure time decreases, the camera detects fewer virtual photons, while the noise level remains the same. As a result, the signal-to-noise ratio (SNR) decreases sharply. Furthermore, the SNR decreases further because of photobleaching and phototoxicity. This decreased quality of SIM raw data may lead to surprising artifacts with various causes, which may confuse a new user of SIM microscopy. We summarize three significant possible sources of severe artifacts in reconstructed super-resolution (SR) images. Ultrafast motion of a biological sample or an uneven illumination pattern is the most difficult to be identified. The estimated parameter could also be incorrect, leading to artifact of regular patterns. Furthermore, reconstruction with the Wiener method generates stochastic artifacts due to the amplification of noise during the deconvolution process. To deal with these problems, we have established a protocol to reconstruct ultrafast SIM raw data obtained in low SNR conditions. First, we checked the quality of the raw data with the ImageJ plugin SIMcheck before reconstruction. Then, a modified parameter estimation method was used to improve the precision of the parameters. Finally, an iterative algorithm was used for SIM reconstruction under low signal-to-noise ratio conditions. This procedure effectively suppressed the artifacts in the super-resolution images reconstructed from raw data of low signal-to-noise ratio.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Super-resolution (SR) fluorescence microscopy is a powerful tool to obtain higher spatial resolution in biological imaging. Many techniques, such as photoactivated localization microscopy (PALM) (Betzig et al. 2006; Hess et al. 2006), stochastic optical reconstruction microscopy (STORM) (Huang et al. 2008; Rust et al. 2006), stimulated emission depletion (STED) (Klar and Hell 1999), super-resolution optical fluctuation imaging (SOFI) (Dertinger et al. 2009), and structured illumination microscopy (SIM) (Gustafsson 2000; Gustafsson et al. 2008), have been proposed. Among current techniques, SIM is often used in the living cell imaging process because of its fast speed and need of illumination of low intensity. Compared with STED, PALM/STORM methods that focus on modulation in the spatial domain, SIM manipulates in the frequency domain to boost resolution (Dan et al. 2014). Thus, the required sample preparation and imaging process are the same as for wide-field fluorescence microscopy. Consequently, any fluorescent sample imaged using wide-field microscopy is compatible with SIM. Moreover, the temporal resolution is as important as the spatial resolution in living cell studies. Many SR techniques attain high spatial resolution by sacrificing the temporal resolution, such as PALM/STORM and SOFI (Kapanidis and Strick 2009). On the other hand, SIM is a notable for its fast speed in imaging live cells (Kner et al. 2009).
Because of these advantages, SIM has been broadly combined with many imaging modes, such as total-internal reflection fluorescence microscopy (TIRF–SIM) (Li et al. 2015), light sheet microscopy (Chen et al. 2014), and surface plasmon microscopy (Wei and Liu 2010). There are also a series of developments in atypical SIM, for example, image scanning microscopy (ISM) (Schulz et al. 2013) and SIM with speckle pattern (blind-SIM) (Mudry et al. 2012). To reduce the artifacts in the reconstructed results, there have been many developments in SIM microscopy, including the strategic and practical protocol of sample preparation and system calibration (Demmerle et al. 2017), more accurate parameter estimation methods for SIM (Chu et al. 2014; Huang et al. 2018; Shroff et al. 2009; Wicker 2013; Wicker et al. 2013; Zhou et al. 2016), SIM reconstruction based on Bayesian estimation (Orieux et al. 2012) or with Richardson–Lucy deconvolution method for Poisson noise (Perez et al. 2016), and some open-source reconstruction programs (Křížek et al. 2016; Lal et al. 2016; Müller et al. 2016). Iterative reconstruction methods have also been proposed to achieve better artifact-suppression during the reconstruction of low SNR data (Chu et al. 2014; Huang et al. 2018).
By restricting the excitation to only a small fraction of the cellular volume (Li et al. 2015), TIRF–SIM achieves SR with reduced phototoxicity and out-of-focus background. However, the faster imaging rate and shorter exposure time render the signal-to-noise ratio of raw TIRF–SIM data to become extremely low. In addition, low-dose or low-efficiency fluorochromes can also be an important cause of a low signal-to-noise ratio. We find that parameter estimation and Wiener deconvolution, which are commonly used, cannot work well in low signal-to-noise ratio conditions (Gustafsson et al. 2008; Lal et al. 2016; Müller et al. 2016). Furthermore, uneven field illumination and low modulation may also affect the reconstruction of raw data. During the data reconstruction process, researchers, especially novice users, may not recognize these factors that may introduce magnified artifacts. Thus, we need to check the quality of raw data and use a modified parameter estimation and deconvolution method during the reconstruction process to reduce artifacts in the reconstruction results.
We used SIMcheck to judge the quality of raw data and employed an effective algorithm for parameter estimation and deconvolution that was specifically developed for the low SNR condition (Ball et al. 2015). Then, we listed the advantages of each method in the following section and proposed a full reconstruction protocol for reconstruction under the low SNR condition. Compared with the commonly used method, this protocol can more efficiently minimize the artifacts in reconstructed SR images.
Summarized procedure
-
(1)
Capture the ultrafast SIM raw data.
-
(2)
Compute the MIV and MCNR value of raw images with the SIMcheck software in ImageJ plugin [TIMING] ~10 s
[CRITICAL STEP] The sample may have a fast movement or an uneven illumination pattern, and a non-white area in MIV result could indicate the precise location of these flaw, which will cause serious artifacts in the reconstructed SR images. These areas of reconstructed SR images should be recognized as invalid areas corresponding to the non-white areas in MIV result. The value of MCNR could determine the deconvolution method in the reconstruction process.
-
(3)
Estimate the parameters with a modified method [TIMING] ~60 s
A modified method could guarantee the precision of parameter estimation even the SNR of SIM raw data is extremely low.
-
(4)
A deconvolution method was chosen corresponding to the MCNR value of SIMcheck.
[CRITICAL STEP] If the value exceeded 4, the Wiener filter in Fair-SIM or RL deconvolution is sufficient. In contrast, the iterative method (TV and Hessian) should be considered if the MCNR is below 4.
Theory and reconstruction for TIRF–SIM
Image theory
Structured illumination microscopy (SIM) is a wide-field SR imaging method that measures the details in the frequency domain, using the Moiré pattern (Gustafsson 2000). The patterns transform the high-frequency components to low-frequency information during the exposure process. The patterns can be described as a sinusoidal function and are discretized by a spatial light modulator (SLM). Iθ,φ is the sinusoidal intensity pattern for illumination and is described by the following equation:
where r ≡ (x, y) is the two-dimensional space position vector, I0 is the illumination intensity, cθ is the modulation depth, φθ is the starting phase, and pθ represents the pattern wave vector in the illumination pattern orientation θ. Let g(r) represents the fluorescence density distribution of the sample; then, the observed emission distribution Dθ,φ(r) can be described as follows:
where h(r) is the point spread function of the SIM set-up and ⊗ is the convolution operator. The Fourier transform of the observed raw data distribution Dθ,φ(r) is given by the following equation:
where H(k) is the optical transfer function (OTF) and Iθ,φ(k) is the Fourier transform of the illumination sinusoidal intensity pattern in Eq. 1. Dθ,φ(k) can be represented as follows:
where g(k \(\pm\,\varvec{p}_{\theta }\)) is the \(\pm \varvec{p}_{\theta }\) frequency shift of g(k) and contains information components that are normally outside of the OTF H(k). To separate g(k \(+\,\varvec{p}_{\theta }\)), g(k \(-\,\varvec{p}_{\theta }\)), and g(k), three different phases, φ (1) θ , φ (2) θ , and φ (3) θ , respectively, are applied to the illumination pattern as substitutes for φθ in Eq. 1. Then, the raw data distributions \(D_{{\theta ,\varphi^{\left( 1 \right)} }} ({\varvec{k}})\), \(D_{{\theta ,\varphi^{\left( 2 \right)} }} ({\varvec{k}})\), and \(D_{{\theta ,\varphi^{\left( 3 \right)} }} ({\varvec{k}})\) are obtained, respectively, with three different phases, φ (1) θ , φ (2) θ , and φ (3) θ , with the illumination pattern orientation θ. The phases φ (1) θ , φ (2) θ , and φ (3) θ are unknown; however, the phase differences φ (2) θ − φ (1) θ = 120° and φ (3) θ − φ (2) θ = 120° are determined in a typical case, and then the different phases in one orientation can be described as follows:
where the matrix M is
After the sample data distributions \(D_{{\theta_{1} ,\varphi^{\left( 1 \right)} }} ({\varvec{k}})\), \(D_{{\theta_{1} ,\varphi^{\left( 2 \right)} }} \left( {\varvec{k}} \right)\), and \(D_{{\theta_{1} ,\varphi^{\left( 3 \right)} }} \left( {\varvec{k}} \right)\) from the image are obtained in the first orientation θ1, the process is repeated twice for two illumination orientations, θ2 and θ3, to cover the whole SR-frequency domain. The nine frames of raw TIRF–SIM data are gathered after the repetition of the three orientations and will be used together to reconstruct one frame of an SIM SR image via the following procedure.
Reconstruction
The conventional procedure for reconstruction can be divided into two important steps. First, we estimate the three unknown parameter values: the pattern wave vector pθ, the start phase φ (1) θ , and the modulation depth cθ. Second, the raw data are reconstructed with these parameters via the Wiener filter.
Parameter estimation
In the first step, we use the three raw data distributions: \(D_{{\theta_{1} ,\varphi^{\left( 1 \right)} }} \left( {\varvec{k}} \right)\), \(D_{{\theta_{1} ,\varphi^{\left( 2 \right)} }} \left( {\varvec{k}} \right)\) , and \(D_{{\theta_{1} ,\varphi^{\left( 3 \right)} }} \left( {\varvec{k}} \right)\) that have the same orientation θ1. These three raw data distributions are a linear combination of the different frequency distributions g(k) and g(k \(\pm\,\varvec{p}_{{\theta_{1} }}\)), including the low-pass optical transfer function H(k). Taking the phase differences φ (2) θ − φ (1) θ = 120° and φ (3) θ − φ (2) θ = 120°, Eq. 5 can be rewritten as follows:
The frequency-shifted component \(g\left( {{\varvec{k}} \pm {\varvec{p}}_{{\theta_{1} }} } \right)\) is the high-frequency counterpart of g(k), which means \(g\left( {{\varvec{k}} \pm {\varvec{p}}_{{\theta_{1} }} } \right)\) has common information in the sections that overlap with g(k). We can calculate the cross-correlation of the overlap sections to estimate the pattern wave vector \({\varvec{p}}_{{\theta_{1} }}\). The cross-correlation of \({\varvec{C}}_{1} \left( {\varvec{k}} \right) = g\left( {\varvec{k}} \right) \cdot H\left( {\varvec{k}} \right) \cdot H\left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} } \right)\) with the \({\varvec{p^{\prime}}}_{{\theta_{1} }}\) shifted variant \({\mathbf{C}}_{2} \left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} } \right) = g\left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} - {\varvec{p}}_{{\theta_{1} }} } \right) \cdot H\left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} } \right) \cdot H\left( {\varvec{k}} \right) \cdot \frac{{ - c_{{\theta_{1} }} }}{2}{\text{e}}^{{ - i\varphi_{{\theta_{1} }}^{\left( 1 \right)} }}\) is the maximum when \({\varvec{p^{\prime}}}_{{\theta_{1} }}\) equals \({\varvec{p}}_{{\theta_{1} }}\) at any given modulation depth cθ and starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\).
After a subpixel accuracy estimation of the pattern wave vector \({\varvec{p}}_{{\theta_{1} }}\), the starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and modulation depths \(c_{{\theta_{1} }}\) are estimated by linear regression of the overlap at different coordinate k in the frequency domain. Similarly, the parameter estimation process for the other two orientations, θ2 and θ3, is the same as for the orientation θ1.
Reconstruction with the Wiener filter
Once the parameters are estimated, we can separate the nine SR-frequency components iθ,m from the known parameters \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and \(c_{{\theta_{1} }}\):
where mth= −1, 0, +1 denotes the order of the bands. Then, we shift iθ,m to the proper place Sθ,m with the known wave vector \({\varvec{p^{\prime}}}_{{\theta_{1} }}\):
In the above equation, Sθ,m is the frequency band of the shifted components corresponding to the θ direction and the mth order, θ = 1–3 is the pattern orientation, and mth= −1, 0, +1. The Wiener filter can combine the different components into a SR image through a weighted average process during the deconvolution, as indicated in the following equation:
In the above equation, H is the optical transfer function, * is the conjugate transpose, k is the frequency domain coordinates, pθ is the pattern wave vector of the θ pattern orientation, α is the Wiener parameter, and A(k) is an apodization function used to eliminate artifacts (Gustafsson et al. 2008).
Ultrafast SIM
Artifact analysis
The reconstruction results for ultrafast structured illumination microscopy (ultrafast SIM) are disturbed by artifacts that seriously affected the data analysis. Thus, the causes of these artifacts need to be analyzed and solved individually. In summary, the artifacts originate mainly from two sources: optical imperfection and noise disturbance in the imaging process.
Optical imperfection causes a regular artifact that can be found in the reconstructed images. However, these artifacts can confuse new users of SIM microscopy. SIMcheck, a plugin for ImageJ, was developed to check a series of important criteria of the raw data and reconstruction results to find the origins of these artifacts (Ball et al. 2015).
Noise includes readout noise, dark current and shot noise. These types of noise can be regarded as white noise, which is frequency dependent in the frequency domain and unique random in the time domain. These kinds of noise lead to the raw data being distinctly disturbed with a mix of Poisson and Gaussian noise, especially for ultrafast exposure times.
This mixing noise in the spatial domain will render precise estimations of the pattern wave vector more difficult during the reconstruction of ultrafast SIM data. If the pattern wave vector \({\varvec{p}}_{{\theta_{1} }}\) is not estimated accurately, the subsequent estimation of the starting phase φmd and the modulation depth c will deviate severely due to the error \({\varvec{p}}_{{\theta_{1} }}\). Then, the 0 and ±1 spatial frequency components will not be perfectly separated and will be shifted to an incorrect location with an error vector \({\varvec{p}}_{{\theta_{1} }}\) in Fourier space, leading to regular artifacts in the reconstructed SR images. These artifacts, which are caused by the deviation of the initial parameter determination, are obviously distinguished in the SIM reconstructed images. An error in the pattern wave vectors will lead to a patterned artifact, such as a snowflake shape, an error in the initial phase will lead to an asymmetric ring artifact, and an error in the modulation contrast will lead to a change in the smoothness or shape of the high-frequency signal (Huang et al. 2018). In addition, this mixing noise will be amplified by the classic Wiener deconvolution method. Thus, the reconstructed SR images have disturbing artifacts that obscure the structure of biological sample.
Motion artifacts or uneven patterns
Before reconstructing raw images of ultrafast SIM, we propose that the following pivotal criteria provided by the SIMcheck software should be checked. A motion and illumination variation (MIV) feature in SIMcheck obtains normalized intensity of the phase-shifted images acquired for each plane for the different illumination patterns and highlights uneven field illumination or movement during acquisition with a colored subregion map. Modulation contrast is another important parameter of the raw data quality, as it critically affects the amount of the frequency-shifted information that can be reassigned during the reconstruction process. The process of calculating the modulation contrast-to-noise (MCN) ratio within the image and translating this information to a heat map provides an average modulation contrast-to-noise ratio (MCNR) as a criterion for evaluating image quality.
Modified parameter estimation
Under the high-NA TIRF–SIM configuration, overlapping regions of different spectra decrease. Furthermore, even fewer excited photons are captured per exposure during the ultrafast exposure time. The low SNR raw data in the spatial domain will render precise estimations of the parameters more difficult; thus, the modified parameter estimation method needs to be considered. The filter method and the averaged method can be simply applied in the conventional reconstruction process under this low SNR condition (Chu et al. 2014; Huang et al. 2018). In addition, iterative and non-iterative methods have been proposed to estimate the parameters even when the pattern ratio in an experimental setup is unknown (Wicker 2013; Wicker et al. 2013). An image recombination transform algorithm can obtain the initial phase of fringe with high precision even with high background and weak modulation depth (Zhou et al. 2016).
Reconstruction software
A number of software exists for users to reconstruct SR images. Fair-SIM, an open-source plugin for ImageJ, contains extensive functions for the reconstruction process. This plugin can process the TIRF–SIM, two-dimensional SIM and three-dimensional SIM data captured from a homemade setup or a commercial SIM microscope, such as OMX and Zeiss. With the parameter determination approximated by the OTF, Fair-SIM can estimate the parameters \({\varvec{p}}_{{\theta_{1} }}\), \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and \(c_{{\theta_{1} }}\) rapidly. Then, SR images can be reconstructed continuously with the Wiener deconvolution or Richardson Lucy deconvolution depending on the noise level.
A two-filter step with Richardson Lucy deconvolution (RL deconvolution) method exists for effective parameter estimation (Perez et al. 2016). If the raw data are corrupted with Poisson noise or an out-of-focus background, this two-filter steps method can effectively move a fluctuating reconstruction spectrum artifact caused by an out-of-focus background or a discontinuous artifact caused by Poisson noise.
The iterative method has also been applied in the reconstruction of SR images, such as TV and Hessian deconvolution (Chu et al. 2014). These iterative methods were based on Gaussian noise and used a priori knowledge, such as the continuity of the biological sample and the PSF, to establish an optimization function. These iterative methods can restrain the artifacts more effectively than the above methods but require more computation.
A protocol for parameter estimation and reconstruction
We proposed a procedure to choose a suitable reconstruction method. First, we used SIMcheck to compute the MCNR and MIV values. A non-white MIV area indicates possible blurring of the SR images due to motion artifacts or an uneven pattern, and this error was resolved by increasing the imaging frame rate or checking the illumination pattern. Then, a modified parameter estimation method was used to improve the estimation precision (refer to the section of “Modified parameter estimation”). Finally, a deconvolution method corresponding to the MCNR value of SIMcheck was chosen. If the value exceeded 4, the Wiener filter in Fair-SIM or RL deconvolution is sufficient. In contrast, the iterative method (TV and Hessian) should be considered if the MCNR is below 4. The TV deconvolution method is appropriate for a flaky structure, and the Hessian–SIM method is applied for situations with flakiness or a line structure. Furthermore, if the biological sample moves slower than 65 nm per frame, time-axis continuity can be used in the Hessian method to better minimize the artifacts in the reconstruction (Huang et al. 2018).
Experiments
We tested our reconstruction protocol on a synthetic sample that combined different structures and low signal-to-noise ratio actin with a 0.5-ms exposure time.
The synthetic sample used for the simulations is shown in Fig. 1A, with a maximum intensity of 109 a.u.. To imitate the actual exposure process, the phantom was illuminated with the pattern presented in Eq. 1. The simulated excitation wavelength λex was 488 nm, the emission wavelength λem was 515 nm, and the numerical aperture (NA) was 1.7. We then added Poisson distributed noise to the illuminated images. We also added a background value of 99 a.u. to all pixels, in combination with Gaussian noise with a standard deviation from 1 to 20 a.u. At the same time, we also considered that there may be fast movements or an uneven illumination pattern, which will cause serious artifacts in the reconstructed SR images, during the raw data-capture process. We illuminated a shifting structure with three different pattern phases and orientations to simulate a rapid movement during the SIM raw data-capture process. We then followed the above protocol with the SIMcheck software to calculate the MIV of the SIM raw images and the reconstruction of SR images from the simulation data.
In addition, we also applied two different parameter estimation methods to the synthetic images of different noise levels to evaluate the bias of each method compared with the ground truth. Then, we used the non-iterative Wiener method provided by Fair-SIM and the Hessian method separately to reconstruct the noisy synthetic image. Finally, we used the following evaluation criteria to judge the quality of the reconstruction images.
The biological sample actin was exposed for 0.5 ms, and data were taken continually seven times before the subsequent phase, and orientation information were collected. We averaged the seven frames with the same phase and orientation into one frame, which had a higher SNR and could be regarded as the ground truth. We also extracted one frame from every seven frames to represent ultrafast raw data with a 0.5-ms exposure time. The excitation wavelength λex was 488 nm, the emission wavelength λem was 515 nm, and the numerical aperture NA was 1.7. We applied the conventional reconstruction process and the proposed protocol to these raw data and compared the artifacts of the different reconstruction methods with the following evaluation criteria.
Evaluation criteria
To compare the performances of different methods, we adopted the following criteria: the peak signal-to-noise ratio (PSNR), and the structural similarity (SSIM) (Wang et al. 2004).
PSNR is defined as
where μmax is the maximum possible pixel value of the image and MSE is the mean-squared error between the reconstructed image and the original image. A higher PSNR means less difference between the reconstructed image and the original image.
SSIM is defined as
where a and b are two windows of size 11 × 11 pixels in the same position of two images. μa and μb are the averages of window a and b, respectively. σ 2 a and σ 2 b are the variances of window a and b, respectively. σxy is the covariance between the two windows. C1 and C2 were chosen as C1 = (0.01μmax)2 and C2 = (0.03μmax)2, respectively. A higher SSIM means that the region of these two images is more similar.
Simulation experiment
For reconstructions of the SR images, the results will contain a large number of artifacts scattered randomly throughout the SR images. Under the high signal-to-noise ratio condition, the signal of reconstructed SR images is relatively strong, and the artifacts are not obvious in the reconstructed SR images. However, when the signal-to-noise ratio is reduced, the signal is relatively weaker, and the artifacts are more pronounced. The continuous signals of the reconstructed SR images become discontinuous because of the artifacts, which is the primary problem that we designed this protocol to solve. Therefore, we simulated data obtained with a very low photon number and applied Poisson and Gaussian noises with different standard deviations, which represented the ultrafast SIM raw data acquired with different exposure times. In addition, there are other several underlying problems, such as an uneven illumination pattern or fast movement of the biological structure during the acquisition process, which lead to severe artifacts in reconstructed SR images.
We first used the structured pattern to illuminate nine different images to simulate the capture process of fast-moving biological samples. Then, we use SIMcheck to calculate the MIV result of the raw images before reconstructing the SIM SR images. If the color of whole MIV result was white, we confirmed that there was no problem that may cause serious artifacts in the SIM raw images. Otherwise, a non-white area of MIV means that motion artifacts exist in the reconstructed SR images (Fig. 1).
By comparing the nine-frame raw data in Fig. 1B, we can see that the reconstructed SR image will produce motion artifacts (Fig. 1D) when there are fast-moving structures in the nine frames of SIM raw data. Additionally, the MIV result will also produce a non-white area at the corresponding position (Fig. 1C). Therefore, SIMcheck is a necessary step to pre-check SIM raw data before reconstructing SR images.
After pre-checking the raw data, the next procedure is to estimate the pattern wave vector p, the starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and the modulation depths \(c_{{\theta_{1} }}\) from the SIM raw data. We first estimated the pattern wave vectors of these series of simulation data with different noise levels using the Fair-SIM software. However, the conventional method had errors in the estimated results when the standard deviation was larger than 10 (the third column in Table 1). Thus, we used the modified parameter estimation method by averaging the raw data corresponding to their patterns and phases (Huang et al. 2018) and then estimated the pattern wave vectors of the averaged raw data using the Fair-SIM software (the fourth column in Table 1).
As we can see from the table, a negligible error (0.14%) in the pattern wave vector was obtained with the conventional parameter estimation method when the standard deviation was below than 10, which could hardly generate artifacts caused by the pattern wave vector error (Huang et al. 2018). However, as the standard deviation of Gaussian noise increased (simulation of the lower SNR of raw images with shorter exposure times), the conventional parameter estimation method differed more from the ground truth parameter. When the standard deviation of Gaussian noise was greater than 10, an error of 46 pixels was generated compared with the ground truth. This error in the pattern wave vector can generate a serious error in the estimated starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and modulation depths \(c_{{\theta_{1} }}\), which could produce periodic artifacts in the reconstructed SR images. These artifacts were much more significant and severe because these SR images contained not only artifacts generated by noise but also artifacts due to the deviations of the parameters from the ground truth. We can see that artifacts due to errors in parameter estimation have a great impact on the reconstructed SR images, even completely obscuring the structures and details in the reconstructed SR image (Fig. 2).
In contrast, the pattern wave vectors estimated using the average method (the fourth column in Table 1) always had a miniscule deviation (0.02%) from the ground truth (the second column in Table 1), even when the standard deviation of Gaussian noise was 20 (not shown in Table 1).
Under ultrashort exposures, the noise source of raw images may be dominated by Poisson but not Gaussian noise. Thus, we simulated raw images that had max pixel value of 29–44 a.u. and were corrupted with Poisson noise. We used the Fair-SIM to estimate this series of noisy raw data with custom and averaged method, and the results are presented as shown in Table 2.
The pattern wave vectors estimated using the average method (the fourth column in Table 2) always had a miniscule deviation (0.02%) from the ground truth (the second column in Table 2). As the pixel value decreased (simulation of the raw images with shorter exposure time), the conventional parameter estimation method (the third column in Table 2) differed more from the ground truth parameter (the second column in Table 2), when the Max pixel value of raw data was smaller than 34, 50 pixels of error was generated compared with the ground truth. Notably, the estimation results of the third and fourth columns of Tables 1 and 2 were all calculated by the same algorithm provided by the Fair-SIM software, and the only difference was that the modified method simply averages the raw data according to the phase and direction of the pattern wave vectors. The averaged parameter estimation method only needs to average the SIM raw images according to their pattern and phase along the time axis; accurate results could then be obtained even with the same conventional parameter estimation method. The average method is simple and effective and has huge promising potential for combination with other algorithms (Chu et al. 2014; Shroff et al. 2009; Wicker 2013; Wicker et al. 2013; Zhou et al. 2016).
To distinguish the artifacts caused by the erroneous parameters from artifacts caused by the Wiener deconvolution, we used simulation data with Gaussian noise with a standard deviation of 2, which could cause a clear difference between these two artifacts in the reconstructed SR images. We artificially changed the pattern wave vector to a 2% error from the ground truth and then used the following three methods to reconstruct the low SNR raw data: (1) an error pattern wave vector and the Wiener deconvolution method; (2) modified parameter estimation and the Wiener deconvolution method; and (3) modified parameter estimation and the iterative Hessian algorithm. Then, the reconstructed SR images reconstructed by these three methods were compared with the ground-truth image using the above evaluation criteria.
As seen in Fig. 2, the Wiener method reconstructed SR image with an error parameter has the most significant artifacts and the lowest evaluation criteria compared with the ground truth (Fig. 2B). The reason is that SR image reconstructed by this method contains not only artifacts generated by noise but also artifacts due to the deviations in the parameter estimation. Comparing Fig. 2B with 2C, we found that artifacts due to parameter estimation errors have a greater impact on the reconstructed SR images. We cannot completely obscure the structures and details in the reconstructed SR image. Furthermore, we also found that artifacts were also generated in the SR images even with a small error in the parameter estimation, which came from the amplification of the noise by the Wiener deconvolution (Fig. 2C). As shown in the yellow box in Fig. 2C, continuous lines in the ground truth may also be intermittent due to the existence of artifacts. Therefore, an improved parameter estimation method is needed to obtain the correct parameters during the reconstruction, and modified deconvolution algorithms are needed to achieve a stronger suppression effect of artifacts, which is shown in Fig. 2D with the use of the iterative Hessian algorithm. The evaluation criteria (Fig. 2E) also indicated the proposed protocol attained the highest value compared with that of the other two methods.
Ultrafast exposed actin with SIM
The number of photons received by the sCMOS camera was further reduced in the ultrafast exposed raw data of actin. Under this condition, the intensity of the noise will affect the structured illumination pattern and actin structure in the SIM raw data. When estimating the parameters of SIM raw data using conventional methods and performing reconstruction with the Wiener deconvolution, severe artifacts are generated in the reconstructed SR results. We then used the following two protocols to reconstruct this low SNR biological sample: (1) the Wiener deconvolution method with the averaged parameter estimation method and (2) the modified parameter estimation and iterative Hessian reconstruction algorithm.
As shown in Fig. 3, even with the correct parameter estimation, with the averaged method, artifacts are also generated in the reconstructed SR images because of the amplification of noise by the Wiener deconvolution (Fig. 3B). Therefore, an improved parameter estimation method is needed to obtain the correct parameters during the reconstruction, and better deconvolution algorithms to suppress artifacts are needed. The use of iterative Hessian algorithms can better reduce artifacts, as shown with Fig. 2C and the SSIM criteria (Fig. 3D).
Discussion
Compared with the classic reconstruction method, the proposed protocol can estimate the parameters accurately and reconstruct SIM SR images with minimized artifacts. As experimentally demonstrated, this protocol has the dual benefits of avoiding the periodic artifacts caused by parameter errors and suppressing the artifacts caused by the underdetermined problem of the deconvolution method. Predictably, when the time resolution of the SIM microscope is further increased and the exposure time is further reduced, the proposed protocol can still suppress artifacts.
Among several parameter estimation methods, the filter method and the average method are simple improvements of the conventional parameter estimation methods but can effectively solve the problem of the bias in parameter estimation. The filter method needs to set a threshold range and may fail once the signal-to-noise ratio drops to a certain extent. The averaging method utilizes the principle that the parameters of the SIM microscope are maintained for a long period of time during the imaging process, and the N × 9 original images are averaged over time according to their phases and angles into N averaged images, which can convert the Gaussian noise to 1/√N compared with that in the original images. The iterative and non-iterative parameter estimation methods proposed by Wicker do not depend on prior information about the phase difference of the illumination vector, which is necessary when the phase difference in each direction is not certain. The image recombination transform algorithm can obtain the initial phase of fringe with high precision, even with high background and weak modulation depth.
The RL method is originally used to constrain the artifacts in the reconstructed SR image caused by the Poisson noise in the original images. However, the RL algorithm has a drawback: as the number of iterations increases, the reconstructed result does not converge. The iterative TV and Hessian algorithms converge to results with fewer artifacts but have their own advantages and disadvantages. The TV method utilizes first-order partial differentiation without decreasing the resolution of SR images but results in block structures and staircase effects in the reconstructed SR results. Thus, the TV method is suitable for biological samples that contain massive dense structures. The Hessian algorithm utilizes second-order partial differentiation to make the image more natural and is more effective in removing artifact. Moreover, the Hessian algorithm also does not depend on specific structures. Dense structures, microtubules or protein scaffolds can be applied. However, the unsuitable parameters of the Hessian algorithm result in a reduction in the image resolution. Therefore, choosing the right parameters is very important, and the recommended parameters settings are provided in a reference paper (Huang et al. 2018).
However, under the condition of a high signal-to-noise ratio, the conventional reconstruction method can well reconstruct the SR image without artifacts. Under such circumstance, using of advanced iterative deconvolution algorithms does not help to reduce artifacts further. In fact, the proposed protocol may require a more complicated procedure and more calculation time. In our opinion, it is a better choice to use the conventional reconstruction method provided by Fair-SIM under conditions of a long exposure time or high signal-to-noise ratio.
Conclusion
An ultrafast imaging rate and shorter exposure time make the signal-to-noise ratio of raw TIRF–SIM data extremely low. The noise in the raw data increases, resulting in erroneous parameter estimation and artifacts within the deconvolution results. Moreover, the raw data may be illuminated by an uneven pattern. These issues cannot be solved by a junior user of SIM microscopy. Considering this condition, we summarized the parameter estimation algorithms and deconvolution methods proposed for low signal-to-noise ratio raw data in this study. We also proposed a reconstruction protocol, combining some effective algorithms in a reconstruction procedure to minimize the artifacts of SR images. The experiment showed that the SR images reconstructed from the proposed protocol outperformed that of the commonly used method, as indicated by the evaluation criteria. We also utilized the proposed protocol to process biological images that were captured with 0.5 ms exposure times, which demonstrated fewer artifacts in reconstructed SR images than those reconstructed by the conventional Wiener method.
The given protocol enables a further reduction in the exposure time and an improvement in the time resolution. We expect it to be widely used in long-term SR imaging of live cells in the future.
References
Ball G, Demmerle J, Kaufmann R, Davis I, Dobbie IM, Schermelleh L (2015) SIMcheck: a toolbox for successful super-resolution structured illumination microscopy. Sci Rep 5:15915
Betzig E, Patterson GH, Sougrat R, Lindwasser OW, Olenych S, Bonifacino JS, Davidson MW, Lippincott-Schwartz J, Hess HF (2006) Imaging intracellular fluorescent proteins at nanometer resolution. Science 313:1642–1645
Chen B-C, Legant WR, Wang K, Shao L, Milkie DE, Davidson MW, Janetopoulos C, Wu XS, Hammer JA, Liu Z (2014) Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346:1257998
Chu K, McMillan PJ, Smith ZJ, Yin J, Atkins J, Goodwin P, Wachsmann-Hogiu S, Lane S (2014) Image reconstruction for structured-illumination microscopy with low signal level. Opt Express 22:8687–8702
Dan D, Yao B, Lei M (2014) Structured illumination microscopy for super-resolution and optical sectioning. Chin Sci Bull 59:1291–1307
Demmerle J, Innocent C, North AJ, Ball G, Muller M, Miron E, Matsuda A, Dobbie IM, Markaki Y, Schermelleh L (2017) Strategic and practical guidelines for successful structured illumination microscopy. Nat Protoc 12:988–1010
Dertinger T, Colyer R, Iyer G, Weiss S, Enderlein J (2009) Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI). Proc Natl Acad Sci USA 106:22287–22292
Gustafsson MG (2000) Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J Microsc 198:82–87
Gustafsson MG, Shao L, Carlton PM, Wang CR, Golubovskaya IN, Cande WZ, Agard DA, Sedat JW (2008) Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J 94:4957–4970
Hess ST, Girirajan TP, Mason MD (2006) Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys J 91:4258–4272
Huang B, Wang W, Bates M, Zhuang X (2008) Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319:810–813
Huang X, Fan J, Li L, Liu H, Wu R, Wu Y, Wei L, Mao H, Lal A, Xi P, Tang L, Zhang Y, Liu Y, Tan S, Chen L (2018) Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat Biotechnol 36:451–459
Kapanidis AN, Strick T (2009) Biology, one molecule at a time. Trends Biochem Sci 34:234–243
Klar TA, Hell SW (1999) Subdiffraction resolution in far-field fluorescence microscopy. Opt Lett 24:954–956
Kner P, Chhun BB, Griffis ER, Winoto L, Gustafsson MG (2009) Super-resolution video microscopy of live cells by structured illumination. Nat Methods 6:339–342
Křížek P, Lukeš T, Ovesný M, Fliegel K, Hagen GM (2016) SIMToolbox: a MATLAB toolbox for structured illumination fluorescence microscopy. Bioinformatics 32:318–320
Lal A, Shan C, Xi P (2016) Structured illumination microscopy image reconstruction algorithm. IEEE J Sel Top Quantum Electron 22:1–14
Li D, Shao L, Chen B-C, Zhang X, Zhang M, Moses B, Milkie DE, Beach JR, Hammer JA, Pasham M (2015) Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science 349:aab3500
Mudry E, Belkebir K, Girard J, Savatier J, Le Moal E, Nicoletti C, Allain M, Sentenac A (2012) Structured illumination microscopy using unknown speckle patterns. Nat Photonics 6:312–315
Müller M, Mönkemöller V, Hennig S, Hübner W, Huser T (2016) Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nat Commun 7:10980
Orieux F, Sepulveda E, Loriette V, Dubertret B, Olivo-Marin J-C (2012) Bayesian estimation for optimized structured illumination microscopy. IEEE Trans Image Process 21:601–614
Perez V, Chang B-J, Stelzer EHK (2016) Optimal 2D-SIM reconstruction by two filtering steps with Richardson-Lucy deconvolution. Sci Rep 6:37149
Rust MJ, Bates M, Zhuang X (2006) Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat Methods 3:793–796
Schulz O, Pieper C, Clever M, Pfaff J, Ruhlandt A, Kehlenbach RH, Wouters FS, Großhans J, Bunt G, Enderlein J (2013) Resolution doubling in fluorescence microscopy with confocal spinning-disk image scanning microscopy. Proc Natl Acad Sci USA 110:21000–21005
Shroff SA, Fienup JR, Williams DR (2009) Phase-shift estimation in sinusoidally illuminated images for lateral superresolution. J Opt Soc Am A 26:413–424
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612
Wei F, Liu Z (2010) Plasmonic structured illumination microscopy. Nano Lett 10:2531–2536
Wicker K (2013) Non-iterative determination of pattern phase in structured illumination microscopy using auto-correlations in Fourier space. Opt Express 21:24692–24701
Wicker K, Mandula O, Best G, Fiolka R, Heintzmann R (2013) Phase optimisation for structured illumination microscopy. Opt Express 21:2032–2049
Zhou X, Lei M, Dan D, Yao B, Yang Y, Qian J, Chen G, Bianco PR (2016) Image recombination transform algorithm for superresolution structured illumination microscopy. J Biomed Opt 21:096009
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
Junchao Fan, Xiaoshuai Huang, Liuju Li, Shan Tan, and Liangyi Chen declare that they have no conflict of interest.
Human and animal rights and informed consent
This article does not contain any studies with human or animal subjects performed by the any of the authors.
Additional information
Junchao Fan, Xiaoshuai Huang and Liuju Li contributed equally to this work.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Fan, J., Huang, X., Li, L. et al. A protocol for structured illumination microscopy with minimal reconstruction artifacts. Biophys Rep 5, 80–90 (2019). https://doi.org/10.1007/s41048-019-0081-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41048-019-0081-7