Abstract
High-throughput microscopy is vital for screening applications, where three-dimensional (3D) cellular models play a key role. However, due to defocus susceptibility, current 3D high-throughput microscopes require axial scanning, which lowers throughput and increases photobleaching and photodamage. Point spread function (PSF) engineering is an optical method that enables various 3D imaging capabilities, yet it has not been implemented in high-throughput microscopy due to the cumbersome optical extension it typically requires. Here we demonstrate compact PSF engineering in the objective lens, which allows us to enhance the imaging depth of field and, combined with deep learning, recover 3D information using single snapshots. Beyond the applications shown here, this work showcases the usefulness of high-throughput microscopy in obtaining training data for deep learning-based algorithms, applicable to a variety of microscopy modalities.
Similar content being viewed by others
Introduction
High-throughput (HTP) microscopy is instrumental in applications such as drug development and screening1,2, study of cell processes3,4,5, and treatment efficacy estimation6 as these require testing different biological conditions in large quantities for statistical sufficiency. Furthermore, when incorporated with an incubator, these instruments allow the observation of multiple samples over prolonged observation windows (days to even weeks1,7).
Recently, anti-cancer drug screening and toxicology studies have widely adopted 3D multi-cellular tumor spheroids as a potential middle ground bridging the gap between 2D cell cultures and animal testing8. However, by large, ensemble analysis of these biological models with HTP microscopes is done over thin 2D slices, limited by imaging at a single focal plane.
The naïve solution for acquiring 3D information is to axially scan each sample at multiple focal planes by acquiring a z-stack9. However, this increases acquisition time and requires computationally demanding post-processing of 3D data. While sophisticated optical solutions such as high-throughput light sheet microscopy10,11, or a focus-tunable lens12 do provide axial information by axial scanning, acquisition of 3D information remains time-consuming and is therefore often sacrificed for the sake of shorter experiment and analysis duration.
Point-spread function (PSF) engineering is a powerful microscopy technique that enables the extraction of typically unavailable information from biological specimens. In PSF engineering, the PSF, namely, the image that a point source generates on the camera, is modified using supplementary optical elements to encode information. This encoding, usually combined with computational image decoding, can enable 3D imaging13,14,15,16,17,18,19, multispectral imaging19,20,21, extended depth of field (EDOF) imaging22,23,24, and more.
Most often, PSF engineering is implemented by extending the optical path of the microscope to gain access to the Fourier plane of the optical system, where wavefront shaping is performed25. This optical extension, in the form of a 4-f system, comprises of multiple elements, making it large and complex to align. In some microscope arrangements, size considerations are imperative, e.g., in HTP microscopes where the entire imaging apparatus moves laterally while scanning many samples per experiment or when the microscope is placed inside an incubator. These considerations render PSF engineering by a 4-f extension practically impossible in such cases.
Here, we introduce a compact PSF-engineering modality implemented inside an Incucyte® S3 Live-Cell Analysis System (Sartorius BioAnalytical Instruments, Inc., Bohemia, NY). In our design, we position a phase-modulating element in the back focal plane (BFP) of the objective lens at its exit. This allows us to engineer the microscope’s PSF and demonstrate applications such as single-shot z-projection and, additionally, combined with a deep learning analysis, snapshot 3D reconstruction of biological structures at high throughput and robustness to misfocus.
This work presents our implementation of compact PSF engineering in HTP microscopy. We showcase two cornerstone PSFs from the field of PSF engineering—an EDOF PSF23,26 and the Tetrapod PSF15. The EDOF PSF maximizes the on-axis intensity over an extended axial range to provide an extended acquisition depth without the need for post-processing. The Tetrapod PSF is optimal for the 3D localization of individual emitters over a given axial range, thus providing 3D information via post-processing. We describe and analyze both PSFs experimentally, discuss the tradeoffs they exhibit, and through them, shed light on the opportunities in applying PSF engineering to HTP microscopy.
Results
A microscope objective lens can be modified directly at its BFP, as has been demonstrated using amplitude masks27. In this plane, shift-independent changes to the PSF can be made by modulating the wavefront. In this work, we take advantage of the robustness of infinity-corrected objectives to the precise axial position of the phase mask to perform photon-efficient PSF engineering. In our implementation, the correct axial placement of the mask was validated by observing minimal field-dependent aberrations of the PSF (Supplementary Note 3).
We used two alternative modalities for mask placement at the exit of the objective lens. In the first, an annular mount was threaded into the objective turret, holding the phase mask in place just below the objective (Fig. 1A). In the second, a 3D-printed mask mount was attached to the bottom of the objective (Fig. 1B, Supplementary Movie 1).
Direct snapshot extended depth of field imaging
Many HTP microscopy applications are limited by the native depth of field (DOF) of the system, while often a z-projection of the sample is desired rather than an image of a single z-plane28. Another DOF-related problem is the challenge of quickly focusing on each sample in an experiment, as they often have different planes of best focus. Both challenges can be addressed using an EDOF PSF.
Here, we implement EDOF imaging via PSF engineering. Our implementation does not require a deconvolution step, which alleviates any need for post-processing. The EDOF PSF provides a sharp image over an extended DOF by exhibiting greater robustness to misfocus than the standard PSF. Our implementation maximizes light throughput by using transparent phase elements, instead of incorporating amplitude modulation (e.g., NA reduction, see Supplementary Note 5).
Various EDOF methods that do not require an image deconvolution step have been previously investigated22,23,26,29. Here, we demonstrate two EDOF PSFs by a phase element placed at the exit of the objective. One element is composed by a photolithographically-fabricated phase mask, designed to maximize the depth of field of the PSF. The design process consisted of retrieving the BFP phase30 that best approximates a constant Gaussian PSF, where photons are maintained in a tight single spot across an extended axial range29. The second EDOF element is a homemade multilayered glass element based on the principle of Abrahamsson et al.26. In short, EDOF is obtained by separating the BFP into several annular regions, which do not coherently interfere with each other. This way, the PSF comprises an incoherent sum of the PSFs produced by each of the regions. Additional description of the element is found in Supplementary Note 2 and in Chen et al.24.
We tested the EDOF PSF with a synthetic sample of volumetrically scattered fluorescent microspheres fixed in gel and with fluorescently tagged nuclei of live spheroids. First, we evaluated the EDOF performance by imaging dense fluorescent microspheres suspended in gel. The bead density was ~22,000 \({\rm {m{m}}}^{-3}\), resulting in ~2000 localizations per frame with the standard PSF, using ThunderStorm31.
The standard PSF broadens quickly with defocus. As a result, the average localization precision is compromised. Moreover, PSF overlap becomes problematic in a dense environment, which further impairs localization quality. The EDOF PSF is far more robust to this issue, as the PSF remains compact over a long axial range before rapidly dispersing. We evaluated the EDOF contribution by finding the PSF width as a function of misfocus (Fig. 2B), as well as counting the number of localizations per width, presented in the histograms in Fig. 2C. By analyzing bead localizations within the entire volume, the EDOF mask exhibits an empirical improvement of the DOF by a factor of 1.9 (Supplementary Note 4). As a result, twice as many localizations were obtained with the EDOF PSF, with an average width (standard deviation of the Gaussian fit) of 1.39 μm. This is compared to 1.74 μm for the standard PSF (Fig. 2C). Often, sources that are laterally proximal but axially separated may not be imaged without suffering from an overlap of the defocused PSFs, unless the EDOF PSF is imposed. This is demonstrated in Fig. 2D.
Next, we performed EDOF imaging of fluorescently labeled nuclei in FaDu cell spheroids. Here, we used a photolithography-fabricated phase mask30. The results are presented in Fig. 3, showing the EDOF effect. Due to the characteristic spherical shape of the spheroid, each image exhibits a radial band of “best focus” around the spheroid center. This effect is more distinctive with the standard PSF, where the nuclei inside this band are very sharp, but the PSF divergence outside the band is swift.
The direct benefit of the EDOF PSF for biological sample imaging is that details can be observed over an increased depth, reducing the number of frames required to properly image a thick sample. Another advantage is relevant to high-throughput scanning experiments, where sample height at the imaging positions (e.g., the wells of a well plate) might vary. Today, this problem is addressed by a step of focus-finding prior to every image, which is time-consuming and may be destructive to the sample due to photobleaching. EDOF can relax the focus finding constraint as it tolerates some misfocus thanks to a decreased PSF sensitivity to axial change. The zoom-ins of Fig. 3 show the superiority of the EDOF PSF, as in a single image the object is sharply observable both at the center (spheroid bottom) and the side, compared to the single image with the standard PSF. We have also shown that further improvement can be attained through post-processing. We took advantage of the axially slow PSF change to apply Lucy–Richardson deconvolution32,33 over the spheroid image, which significantly improves the results. More information can be found in Supplementary Note 1. An additional visualization is provided in Supplementary Movie 2.
Snapshot three-dimensional imaging with CellSnap
In recent years, various methods have addressed the need for 3D imaging with enhanced temporal resolution, including lensless imaging34, light-field imaging35, PSF engineering36, and more. Compared with PSF engineering, lensless and light-field imaging typically entail a degradation in lateral resolution, especially mask-only systems (such as a diffuser and a bare sensor), which have no magnifying optics and are thus limited to low effective numerical apertures (NA), although resolution, as well as reconstruction speed, can be significantly improved using appropriate post-processing37,38. Specifically, in the field of HTP microscopy, light-sheet imaging with microfabricated culture chips10 was recently proposed as an attractive candidate; however, this technique requires dedicated culture chips with embedded optical elements. Therefore, currently, there’s an unmet need for a compact, scannable, easy-to-setup, and cost-effective HTP system that can rapidly capture and analyze 3D data.
Here we implement HTP 3D imaging using the Tetrapod PSF15 (Fig. 4A, B), complemented with a deep neural network post-processing algorithm dubbed CellSnap. By combining the power of PSF engineering and deep learning, we demonstrate the possibility of 3D cell segmentation using a single snapshot, offering an order of magnitude faster acquisition compared to a traditional z-scan. Moreover, our approach is also computationally efficient, as our post-processing is highly parallelizable on a GPU and involves analyzing 2D images as opposed to 3D data cubes using existing methods39,40.
To showcase our approach, we imaged live spheroids with fluorescently stained nuclei of the outer-shell cells, cultured in a 96-well plate (one per well, Supplementary Fig. 6). Training data consisted of pairs of z-stacks: one with the standard PSF and one using the Tetrapod PSF. We have taken advantage of the HTP capabilities of the microscope, enabling simple acquisition of large training datasets featuring hundreds of spheroids. Notably, such experiment-based training is superior to simulation-based training which exhibits limited consistency with experimental images when dealing with complex shapes and patterns. Z-stacks extending 400 μm were acquired—a necessary redundancy in range to counter variability in the axial position of the spheroids due to a global micro-tilt of the plate, as well as different in-well positioning of each spheroid (Supplementary Fig. 6). The variability in sample axial positions is inherent to HTP microscopes and serves as a further motivator for 3D imaging. Implications of this are discussed in detail in the Supplementary Information.
Next, we turn to describe our deep learning-based post-processing algorithm, dubbed CellSnap. To devise a neural network that handles a range of focal settings, we break the task into two parts. First, we train a focus finder that, given a snapshot taken with the Tetrapod PSF (shown in Fig. 4A, B), can estimate the focal plane at which it was measured (Fig. 4C). Afterward, we train a conditional 3D segmentation model (Fig. 4F), which, given a focus setting and a Tetrapod snapshot, can recover a canonical 3D segmentation of cell nuclei in the form of a binary occupancy grid, where occupied voxels belong to cells. For architecture details and learning hyper-parameters (see Supplementary Notes 8, 9).
The dataset for training CellSnap consisted of 592 spheroid “views” out of which 532 were used for training and 60 were used for validation (Supplementary Note 6). The model was then tested on another 20 spheroids from wells not seen during training/validation.
Training CellSnap consisted of two sub-tasks: training a focus finder and training a conditional 3D segmentation model, with each element in our dataset consisting of a pair of matched and aligned z-stacks acquired by scanning a spheroid twice: once with the Tetrapod PSF (Fig. 4C) and once with the standard PSF (Fig. 4E). First, the focus finder was trained on Tetrapod z-stacks alone (Fig. 4C). At each training step, a random Tetrapod snapshot was sampled, and the focus finder was trained to estimate its focal plane using the mean squared error (MSE) loss. The estimated focus was fairly accurate with an error standard deviation of ~4 μm (Fig. 4D), motivating us to train the conditional 3D segmentation model to output a voxel grid with an axial resolution of 4 μm.
For training the conditional 3D segmentation model, we used Cellpose40 to segment the standard PSF stack and post-processed the result to produce a binary 3D segmentation label where occupied voxels belong to cells (Fig. 4E). The resulting stack-pairs for this phase of training were a z-stack of Tetrapod snapshots at different focal planes (serves as possible inputs), and the accompanying 3D cell segmentation at a canonical axial position (serves as desired output). At this stage, the focus finder parameters were kept fixed, and only the 3D segmentation model was trained using the Dice loss (Fig. 4F).
After both components of CellSnap are trained, at test time, we can feed a single Tetrapod snapshot at an arbitrary focal plane and obtain as output the 3D binary segmentation of cell nuclei. The results were also further post-processed using watershed splitting41 to recover individual cell instances (Fig. 4G).
Figure 5 shows a representative example of CellSnap applicability. As a baseline, the results are compared to the application of Cellpose to a 3D z-scan of the standard PSF (Fig. 5A–C). Using a single snapshot with the Tetrapod PSF, CellSnap achieves 3D cell segmentation (Fig. 5D–F) at comparable quality to the baseline, with a slight performance deterioration at higher depths (edges in the image) where cells are densely packed (Fig. 5H, I). Importantly, compared to the baseline, CellSnap was an order of magnitude faster both in acquisition (10×) and in post-processing (8.3×) (Fig. 5G and Supplementary Note 14).
When comparing the number of recovered cells as a rough estimate, CellSnap was able to roughly match the performance of Cellpose (Fig. 5J), with the results being highly stable across a broad range of focus settings (Fig. 5K), proving robustness to misfocus. In some cases, CellSnap was even able to recover cells missing from the segmentation label derived with Cellpose (Fig. 5B, E, H insets (i)). This is likely due to the robustness of deep networks to small amounts of label noise in training42, as most training patches represent the correct mapping between 2D pixels and 3D cells (Supplementary Note 9). Nonetheless, such predictions were still mistakenly classified as false positives due to imperfect labels in the quantitative analysis described next (Supplementary Note 12).
We used two different metrics to quantify performance (Fig. 5L, M): the Dice coefficient and the “average precision”39,40. The Dice coefficient quantifies the per-voxel accuracy of our binary 3D segmentation, while the average precision highlights the performance on the individual cell level (Supplementary Note 11). The labels derived with Cellpose were mainly reliable at the lower part of the spheroids. Hence our quantitative analysis was restricted to the middle 150 × 150 pixels of the image (~120 μm axially), roughly capturing the bottom third of the spheroid. The reported quantitative metrics are the average of the entire test set over a large focus range of 140 μm. As a baseline for the quantitative analysis, we calculated the mean Dice coefficient and average precision for two random 3D segmentations in the test set. CellSnap provides strong segmentation results, highly correlated with the underlying 3D cells (see Supplementary Movies 3, 4). Supplementary Fig. 11 presents additional segmentation examples.
Finally, it is worth noting that even without PSF engineering, CellSnap alone can recover 3D information from standard snapshot data, although the performance is limited to a narrow focus range, significantly smaller than the range afforded by the Tetrapod PSF (Supplementary Note 13).
Three-dimensional tracking with DeepSTORM3D
The main advantage of snapshot 3D imaging is the fast acquisition of volumetric information, which is particularly important for dynamic scenes. We demonstrate this advantage by using our high throughput microscopy for the application of nanoparticle tracking analysis (NTA)43. In NTA, particles diffusing in a suspension are tracked and their diffusion coefficient is extracted, which can provide information such as the diffusers’ size distribution. Typical NTA studies make use of the standard PSF and perform tracking in 2D. Using PSF engineering can significantly enhance the information throughput of an NTA experiment by adding an extra tracking dimension over an extended z range. Our system can be seamlessly deployed for this task, combined with off-the-shelf software to handle data post-processing. To demonstrate this, we applied the Tetrapod PSF to the task of 3D single particle tracking of fluorescent microspheres diffusing in a water-glycerol mixture and analyzed the resulting time-lapse with DeepSTORM3D14. The results are presented in Fig. 6 (see also Supplementary Movie 5). Compact PSF engineering combined with DeepSTORM3D offers the ability to capture precise dynamics in 3D, a valuable capability that is rarely explored in NTA studies. Compared to a post-processing-only approach, combining DeepSTORM3D with the Tetrapod PSF increases the overall number of recovered tracks and the mean track length and substantially improves the localization precision in z (see Supplementary Figs. 12 and 13).
Discussion
In this work, we introduced a simple, elegant way to implement PSF engineering in HTP microscopes. The approach opens the door to a variety of applications. We experimentally demonstrated two of the most common PSF engineering applications in an HTP microscope on synthetic and live samples. First, we presented EDOF imaging, i.e., a photon-efficient optical z-projection. The demonstrated implementations of EDOF could be suitable both for imaging and for performing localization microscopy using the Gaussian approximation of the PSF. The post-processing-free nature of the EDOF application is important because complicated analysis is often the bottleneck of elaborate microscopy techniques. Second, we demonstrated snapshot 3D cell segmentation using the Tetrapod PSF with a custom deep-learning architecture dubbed CellSnap. Exploiting this synergy between engineered optics and powerful computation enabled us to achieve comparable results to existing approaches while being an order of magnitude faster both in acquisition and in data post-processing. Notably, our proposed hardware modification is cost-effective, easy-to-setup, and highly customizable, paving the way to rapid and reliable 3D HTP microscopy. As another application of snapshot 3D imaging, we showcased the proposed hardware prototype with the Tetrapod PSF for NTA. Compact PSF engineering combined with DeepSTORM3D captured precise dynamics in 3D, improving z precision and increasing the depth range.
Incorporating PSF engineering in an optical system to increase the information throughput is subject to trade-offs. Most notable is the engagement with the optics, which in our work is done with minimal effort as an easily applied/removed add-on. Other drawbacks may include the requirement for post-processing, and resolution compromise, as we have presented in this work. Our proposed hardware-software combination triggers many possible questions regarding its capabilities and limitations. For example, on the optics side, how optimal is the Tetrapod PSF for imaging multiple cells rather than single point sources? It is likely that a content-aware PSF design44, optimizing the “cell spread function”, could provide a substantial boost in performance, as previously shown in dense multi-emitter imaging18. Similarly, on the post-processing side, how sensitive is our performance to the choice of the CNN architecture, output representation, loss function, and overall training schedule of CellSnap? Currently, it is unclear how each of these affects our bottom-line performance. However, the contributions presented in this work are orthogonal to efforts optimizing instance separation of crowded cells39,40,45,46, and promoting model adaptivity to conditioning information47,48 such as the focus setting. Hence, we envision that CellSnap performance can be boosted by adapting these advancements to our existing solution.
Finally, in this work we focused on the task of 3D cell segmentation as segmenting nuclei in microscopy images is typically the first step of any quantitative analysis performing phenotypic measurements on the single cell level9. However, our system can be tweaked on the post-processing end to address other interesting tasks encountered in HTP microscopy. For example, some applications might require the intensity profile of individual cells (e.g., when monitoring the activity of specific neurons). Other applications could be satisfied with a sample summarizing statistic, e.g., the number of dead cells after an external intervention. Clearly, our approach can be implemented with a variety of phase masks to fulfill any applicational requirement, including, e.g., spectral19,21 or polarization49 modulation. This work suggests that HTP microscopes can be made computational as facilitated by compact PSF engineering and deep learning, augmenting them with enhanced capabilities in a similar fashion to traditional microscopes, and possibly even more so considering the ease of obtaining large quantities of training data using HTP acquisition.
Methods
Phase-mask mounts
Modality 1
A thin aluminum annular mount was fabricated with CNC, having an M25×0.75 external thread corresponding to the microscope objective turret. Onto the mount, a phase mask was glued from its corners using UV-cured optical adhesive (NOA 68, Norland). The aperture of the mount, at a diameter of 12 mm, corresponds to the size of the objective BFP and the mask. The mount with the mask was then screwed into the turret below the objective.
Modality 2
A mask mount was attached to the bottom of the objective itself rather than the turret. This was done by exploiting an RMS-M25 threading adapter native to the objective. We added a ring above the threading adapter, resulting in it protruding 2 mm below the objective. Having done so, a 3D printed mount could be fit into the protruding internal RMS threading. The mount contains a centered aperture and space for the mask to be glued onto it, coinciding with the aperture. The advantage of the second modality is that it is easier to insert and remove the mount and that the mount does not occupy threaded turret space.
Optical system and photolithographic phase mask design
The optical system consists of the native imaging system of Incucyte® S3 Live-Cell Analysis System, comprising an objective and a tube lens of 104 mm focal length. In all experiments besides NTA we used the 10X objective, with an NA of 0.3, on which we mounted phase elements. For NTA, the 20X with NA of 0.45 was used.
Tetrapod phase-mask optimization
To achieve precise 3D localization, we aim to minimize the Cramér–Rao lower bound (CRLB) as a proxy for the system’s localization precision. The PSFs, once optimized through this process, facilitate precise 3D position estimation of individual emitters at position \(\varTheta=({x}_{0},{y}_{0},{z}_{0})\) based on noisy measurements (to simplify, we assume the system is governed by shot noise—for a complete model see ref. 24).
We denote \({{{{{\rm{PSF}}}}}}(x,y,M;\varTheta )\) as the model PSF of an emitter at camera coordinates \(\left(x,y\right)\), with a phase mask \(M\). Assuming Poisson statistics, the measured image \(I(x,y)\) is given by
where \(B(x,y)\) is additive background noise, and \({{{{{\rm{Poiss}}}}}}\left(\cdot \right)\) is a Poisson noise operator corresponding to the shot-noise model. The log-likelihood function \(L(\varTheta )\) for the measurement in Eq. (1) is given by
where \({N}_{x,}{N}_{y}\) are the number of image pixels per dimension, and \(C(I(x,y))\) is a function of the measurements that is independent of \(\varTheta\).
Given the likelihood function \(L(\varTheta )\), the Fisher Information matrix \(Q(\varTheta )\) is defined as
Substituting the log-likelihood from Eq. (2), we get
And the CRLB for estimating \({\varTheta }_{i}\in ({x}_{0},{y}_{0},{z}_{0})\) is defined as
In our implementation, \({{\rm {CRL}{B}}}_{i}(\varTheta ; {M})\) is sampled at 100 equal intervals in the axial range, giving our optimization cost function:
We also simplify the per-pixel background term \(B(x,y)\) to a single scalar and scale the PSFs to match realistic signal counts encountered in SMLM imaging. Notably, different from the original proposition of a CRLB-optimized phase mask15, we optimized the CRLB using a per-pixel approach rather than constraining the solution to a subspace of Zernike polynomials, as has been previously suggested30. This was particularly important to navigate the wide variety of possible solutions.
EDOF phase-mask optimization
The PSF simulation model in the EDOF design was similar to the description above, although for EDOF, the cost function was different. Here, instead of minimizing the CRLB of 3D localization precision, the cost function used was the mean square difference of the PSF from a narrow 2D Gaussian profile. Here, a non-changing Gaussian profile was set as the target PSF (the algorithm is relatively robust to the exact size of this Gaussian), at equal axial intervals spanning the 60 µm axial design range.
Photolithographic phase mask fabrication
The optimized phase profiles were converted to depth maps to be etched in quartz, considering the emission wavelength of the design (in our work, each mask corresponds to one of the two emission channels available in the HTP microscope). Using the depth maps, the phase masks were etched using photolithography (Holo/Or, Israel).
Incoherence-based EDOF masks
The optical element comprised of stacked 0.17 mm-thick microscope coverslips (Merienfeld Cat. # 0107052), with holes cut using a UV laser cutter. The coverslip thickness provided sufficient optical path difference per layer, i.e., with the refractive-index difference between the element material and air of \(\Delta n \, \approx \, 0.5\), the path difference is about 85 μm, while the typical fluorescence coherence length is on the order of 10 μm. We measured the fluorescence coherence length using an interferometer, and the results are consistent with emission spectrum-based calculation (Supplementary Note 2). The element comprises of 3 glass layers, resulting in 4 non-interfering (mutually incoherent) zones in the BFP. The proper diameters were calculated24,26 such that each coherent zone produces the same DOF—given an objective aperture of 12 mm, the diameters of the glass layers are [6, 8.5, 10.4] mm. We used a laser-cutting machine (Epilog Zing 16) to cut holes in the coverslips and an outer frame to maintain the concentricity of the holes. After the coverslips were cut and cleaned, they were stacked inside a custom-designed 3D-printed mount that attaches to the bottom of the objective lens (Fig. 1C).
Spheroid preparation
FaDu cells (courtesy of the Shamay lab, Technion) were grown in DMEM high glucose w/o l-glutamine and w/o sodium pyruvate (Sartorius, Israel), supplemented with 0.02% fetal bovine serum (Biological Industries, Israel), 0.01% l-glutamine and 0.01% penicillin-streptomycin solution (Biological Industries, Israel), in 25 cm2 flask at 37 °C, 5% CO2, until full coverage.
For spheroid formation, the cells were detached using 1 ml Trypsin-EDTA solution, transferred into a 15 ml falcon with 9 ml fresh medium, and centrifuged for 2 min, 893×g. The supernatant was removed, and the cells were resuspended in 10 ml fresh medium. The cell solution was diluted to 15,000 cell/ml, transferred into Corning® Costar® Ultra-Low Attachment 96-well plate (Merck), 200 µl per well (3000 cells), and incubated at 37 °C, 5% CO2 for 4–7 days until spheroid formation.
For live cell nucleus labeling and imaging, each spheroid was stained by adding 1 µl of 0.15 mM SYTO®24 Green Fluorescent Nucleic Acid Stain (diluted with medium) (Invitrogen) into each well and incubated for 1 hour on a rocker at RT.
Bead sample preparation and imaging
We used Mini-PROTEAN gel casting stand (BioRad) with 0.75 mm short glass to prepare acrylamide gel with beads by vertical solidification to allow beads to sparsely spread out in 3 dimensions. Gel solution with beads was prepared by gently mixing 1.2 ml double distilled water, 750 µl 40% acrylamide/bis-acrylamide solution (Sigma, A7802), 80 µl 50% glycerol, 3 µl bead solution (1 µm, fluoro-max dyed green aqueous fluorescent particles (Thermo Fisher Scientific, G0100)), 1.5 µl TEMED, and 10 µl 10% ammonium per sulfate. The gel solution was immediately pipetted into the gel casting stand and gently topped with about 1 ml of double-distilled water to level the gel. Once solidified, the gel was kept in water in the dark at 4 °C. Imaging comprised of a z-stack spanning 300 µm, at 1 µm step. Each FOV was imaged once with the standard PSF and once with EDOF, realized by a lithography-fabricated phase mask. Exposure time was 1 ms per frame for both PSFs.
Nanoparticle tracking analysis sample preparation
Fluorescent microspheres (FF806, Invitrogen) were diluted in DDW to a concentration of \({10}^{-6}\). A mixture of 20% glycerol (by volume) and water was prepared. The bead solution was further diluted with 20% glycerol/water(v/v) to \({10}^{-7}\). To avoid sample evaporation we used 25 µl Gene Frame sticker (Thermo Fisher, AB0576) on a coverslip, added 25 µl bead solution and sealed the sample with another coverslip.
Imaging—General
Imaging was performed using customized acquisition software on the Incucyte® S3 Live-Cell Analysis System equipped with a standard Green/Red fluorescence module (Green Excitation 440–480 nm, Green Emission 504–544 nm; Red Excitation 565–605 nm, Red Emission 625–705 nm). All references to the “standard PSF” address the native PSF of the Incucyte microscope. Comparisons between engineered and standard PSFs were obtained with similar objectives (either ×10 or ×20 original Incucyte objective) in all experiments.
Spheroid imaging—EDOF
Each well of a 96-well plate was prepared with a single spheroid inside (see “spheroid preparation”). Imaging comprised of two z-stacks per well—one with the standard PSF for ground-truth data, followed by an objective comprising an EDOF phase element (layered glasses - incoherence-based EDOF). Z-stacks comprised 60 frames with axial steps of 5 µm. Exposure times were 0.49 ms for the standard PSF and 1.22 ms for the EDOF PSF, determined by a 2× base exposure for the EDOF as well as an additional 23% exposure time to compensate for signal reduction from photobleaching per z-stack.
Spheroid imaging—Tetrapod
Each well of a 96-well plate was prepared with a single spheroid inside (see “spheroid preparation”). Imaging comprised of two z-stacks per well—one with the standard PSF for ground-truth data, and one with the Tetrapod PSF. Z-stacks comprised 100 frames with axial steps of 4 μm. Each well plate was scanned 3 times, where it was gently shaken between scans. By shaking the well plate, the spheroids had changed their positioning, providing a random new view, and we regarded them as new spheroids altogether. In total, data from four different plates was used. Per scan, fluorescence intensity dropped by 40% due to photobleaching, thus we increased exposure by the same factor with every repeated scan of a spheroid. Base exposure for the unmodified objective was 0.5 and 1.5 ms for the Tetrapod objective.
Nanoparticle tracking analysis imaging
Imaging was conducted using Incucyte 20X objective, and a Tetrapod phase mask designed for the objective and the red emission channel. Imaging was performed with an exposure time of 400 and 2000 ms delay between images.
Statistics and reproducibility
Experimental z-stacks for x–z projections (Figs. 2A, 3B) were acquired once using a bright, isolated fluorescent bead on a coverslip. The same bead was used for both standard and EDOF data.
Data of dense beads in gel (Fig. 2) was acquired using a single sample. Five different z-stacks were acquired (Different FOVs). We chose one FOV where we visually observed an ROI that showcases multiple beads that are laterally close but axially apart, evidently showing the benefit of EDOF (Displayed in Fig. 2D). Only that FOV was analyzed using ThunderStorm, and statistics were done over 623k (1232k) localizations of σ < 5 μm of the standard (EDOF) PSF.
Spheroid EDOF data (Fig. 3C) was acquired over 96 spheroids in a 96-well plate. A single spheroid was chosen to visually showcase the advantage of our method. No statistical or quantitative analysis was performed on this data.
Quantitative analysis of the EDOF performance was done using a single z-stack acquisition of dense beads in gel (Fig. 2), comprising of thousands of localizations, as specified in the text.
Sample size in 3D reconstruction experiments with CellSnap (Figs. 4 and 5J–M) consisted of n = 20 cellular spheroids constructed from three biologically independent samples examined over four independent experiments with a 96-well plate. As detailed in Supplementary Note 6, our analysis excluded data from wells with saturated/insufficient signals and where cells did not form spheroids. No statistical method was used to predetermine the sample size.
Nanoparticle tracking data was acquired for two different diffusing bead samples (Fig. 6). The same sample was imaged twice, once with the Tetrapod PSF and once with a standard imaging objective. The FOV, in Fig. 6, was chosen to highlight the enhanced-depth capabilities afforded by the Tetrapod PSF. Similar results were obtained for another four nonoverlapping FOVs from the same sample. Additionally, in Supplementary Fig. 13, we include the results of a similar experiment with a denser bead sample to highlight the applicability of our technique at higher emitter densities.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Data availability
The data supporting the findings of this study is publicly available at Zenodo under accession code 1092812250: https://zenodo.org/records/10928122. Source data are provided with this paper.
Code availability
Code is made publicly available at https://github.com/EliasNehme/HTPmicroscopy. The code release used to produce the results in this study is also publicly available at Zenodo under accession code 1093803551: https://zenodo.org/records/10938035.
References
Li, Y. & Kumacheva, E. Hydrogel microenvironments for cancer spheroid growth and drug screening. Sci. Adv. 4, eaas8998 (2018).
Bickle, M. The beautiful cell: high-content screening in drug discovery. Anal. Bioanal. Chem. 398, 219–226 (2010).
Zhou, X. & Wong, S. T. C. Informatics challenges of high-throughput microscopy. IEEE Signal Process. Mag. 23, 63–72 (2006).
Hanson, K. M. & Finkelstein, J. N. An accessible and high-throughput strategy of continuously monitoring apoptosis by fluorescent detection of caspase activation. Anal. Biochem. 564–565, 96–101 (2019).
Pegoraro, G. & Misteli, T. High-throughput imaging for the discovery of cellular mechanisms of disease. Trends Genet. 33, 604–615 (2017).
Murphy, D. A., Cheng, H., Yang, T., Yan, X. & Adjei, I. M. Reversing hypoxia with PLGA-encapsulated manganese dioxide nanoparticles improves natural killer cell response to tumor spheroids. Mol. Pharm. 18, 2935–2946 (2021).
Åkerfelt, M. et al. Automated tracking of tumor-stroma morphology in microtissues identifies functional targets within the tumor microenvironment for therapeutic intervention. Oncotarget 6, 30035–30056 (2015).
Zanoni, M., Pignatta, S., Arienti, C., Bonafè, M. & Tesei, A. Anticancer drug discovery using multicellular tumor spheroid models. Expert Opin. Drug Discov. 14, 289–301 (2019).
Desmaison, A. et al. Impact of physical confinement on nuclei geometry and cell division dynamics in 3D spheroids. Sci. Rep. 8, 8785 (2018).
Beghin, A. et al. Automated high-speed 3D imaging of organoid cultures with multi-scale phenotypic quantification. Nat. Methods 19, 881–892 (2022).
Zhu, T. et al. Large-scale high-throughput 3D culture, imaging, and analysis of cell spheroids using microchip-enhanced light-sheet microscopy. Biomed. Opt. Express 14, 1659 (2023).
Alonso, J. R., Silva, A., Fernández, A. & Arocena, M. Computational multifocus fluorescence microscopy for three-dimensional visualization of multicellular tumor spheroids. J. Biomed. Opt. 27, 066501 (2022).
Greengard, A., Schechner, Y. Y. & Piestun, R. Depth from diffracted rotation. Opt. Lett. 31, 181 (2006).
Huang, B., Wang, W., Bates, M. & Zhuang, X. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science (1979) 319, 810–813 (2008).
Shechtman, Y., Sahl, S. J., Backer, A. S. & Moerner, W. E. Optimal point spread function design for 3D imaging. Phys. Rev. Lett. 113, 133902 (2014).
Pavani, S. R. P. et al. Three-dimensional single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. Proc. Natl. Acad. Sci. 106, 2995–2999 (2009).
Weiss, L. E. et al. Three-dimensional localization microscopy in live flowing cells. Nat. Nanotechnol. 15, 500–506 (2020).
Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat. Methods 17, 734–740 (2020).
Opatovski, N. et al. Multiplexed PSF engineering for three-dimensional multicolor particle tracking. Nano Lett. 21, 5888–5895 (2021).
Shechtman, Y., Weiss, L. E., Backer, A. S., Lee, M. Y. & Moerner, W. E. Multicolour localization microscopy by point-spread-function engineering. Nat. Photonics 10, 590–594 (2016).
Hershko, E., Weiss, L. E., Michaeli, T. & Shechtman, Y. Multicolor localization microscopy and point-spread-function engineering by deep learning. Opt. Express https://doi.org/10.1364/oe.27.006158 (2019).
Ben-Eliezer, E., Marom, E., Konforti, N. & Zalevsky, Z. Experimental realization of an imaging system with an extended depth of field. Appl. Opt. 44, 2792 (2005).
Nehme, E. et al. Learning optimal wavefront shaping for multi-channel imaging. IEEE Trans. Pattern Anal. Mach. Intell. 43, 2179–2192 (2021).
Chen, B. et al. Extended depth of focus multiphoton microscopy via incoherent pulse splitting. Biomed. Opt. Express 11, 3830 (2020).
Backer, A. S. & Moerner, W. E. Extending single-molecule microscopy using optical Fourier processing. J. Phys. Chem. B 118, 8313–8329 (2014).
Abrahamsson, S., Usawa, S. & Gustafsson, M. A new approach to extended focus for high-speed high-resolution biological microscopy. In: Proc. SPIE 6090, Three-Dimensional and Multidimensional Microscopy: Image Acquisition and Processing XIII, 60900N (eds Conchello, J.-A. et al.) (2006).
Rieu, M. et al. Parallel, linear, and subnanometric 3D tracking of microparticles with stereo darkfield interferometry. Sci. Adv. 7, eabe3902 (2021).
Liu, C., Lewin Mejia, D., Chiang, B., Luker, K. E. & Luker, G. D. Hybrid collagen alginate hydrogel as a platform for 3D tumor spheroid invasion. Acta Biomater. 75, 213–225 (2018).
Cao, R. et al. Optical-resolution photoacoustic microscopy with a needle-shaped beam. Nat. Photonics 17, 89–95 (2023).
Ferdman, B. et al. VIPR: vectorial implementation of phase retrieval for fast and accurate microscopic pixel-wise pupil estimation. Opt. Express 28, 10179 (2020).
Ovesný, M., Křížek, P., Borkovec, J., Švindrych, Z. & Hagen, G. M. ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).
Richardson, W. H. Bayesian-based iterative method of image restoration*. J. Opt. Soc. Am. 62, 55 (1972).
Lucy, L. B. An iterative technique for the rectification of observed distributions. Astron J. 79, 745 (1974).
Antipa, N. et al. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018).
Skocek, O. et al. High-speed volumetric imaging of neuronal activity in freely moving rodents. Nat. Methods 15, 429–432 (2018).
Yanny, K. et al. Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy. Light Sci. Appl. 9, 171 (2020).
Lu, Z. et al. Phase-space deconvolution for light field microscopy. Opt. Express 27, 18131 (2019).
Wang, Z. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat. Methods 18, 551–556 (2021).
Weigert, M., Schmidt, U., Haase, R., Sugawara, K. & Myers, G. Star-convex polyhedra for 3D object detection and segmentation in microscopy. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) 3655–3662 (IEEE, 2020).
Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100–106 (2021).
Ollion, J., Cochennec, J., Loll, F., Escudé, C. & Boudier, T. TANGO: a generic tool for high-throughput 3D image analysis for studying nuclear organization. Bioinformatics 29, 1840–1841 (2013).
Natarajan, N., Dhillon, I. S., Ravikumar, P. & Tewari, A. Learning with noisy labels. In Advances in Neural Information Processing Systems 26 (eds Burges, C. J. et al.) (Curran Associates, 2013).
Dragovic, R. A. et al. Sizing and phenotyping of cellular vesicles using nanoparticle tracking analysis. Nanomedicine 7, 780–788 (2011).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).
Pachitariu, M. & Stringer, C. Cellpose 2.0: how to train your own model. Nat. Methods 19, 1634–1641 (2022).
Lalit, M., Tomancak, P. & Jug, F. EmbedSeg: embedding-based instance segmentation for biomedical microscopy data. Med. Image Anal. 81, 102523 (2022).
Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 34 (eds Larochelle, H. et al.) (Curran Associates, 2020).
Yosef, E., Elmalem, S. & Giryes, R. Video reconstruction from a single motion blurred image using learned dynamic phase coding. Sci Rep 13, 13625 (2023).
Oldenbourg, R. Polarized light microscopy: principles and practice. Cold Spring Harb. Protoc. 2013, pdb.top078600 (2013).
Opatovski, N., Nehme, E. et al. Depth-enhanced High-throughput Microscopy by Compact PSF Engineering—Data Repository https://zenodo.org/records/10928122 (2024).
Opatovski, N, Nehme, E. et al. Depth-enhanced High-throughput Microscopy By Compact PSF Engineering—Code Repository https://zenodo.org/records/10938035 (2024).
Acknowledgements
We’d like to thank Dana Meron Azagury from Yosef Shamay’s lab for assisting with spheroid preparation and Amit Meller’s lab for using their laser cutting machine (Zing 16, Epilog). This research was supported in part by Sartorius Stedim Biotech: N.O., N.Z., R.K.O.; by the Israel Innovation Authority, Technological Infrastructure Division, Applied Research in Academia; by the ISRAEL SCIENCE FOUNDATION (grant No. 450/18); by funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 802567-ERC-Five-Dimensional Localization Microscopy for Sub-Cellular Dynamics: Y.S., N.O., O.A., B.F., E.N., I.B.; Additionally, Y.S. is supported by the Zuckerman Foundation and by the Donald D. Harrington fellowship.
Author information
Authors and Affiliations
Contributions
N.O., R.K.O., B.F., and Y.S. conceived the experimental setup; E.N. conceived, designed, and implemented CellSnap; N.O., R.K.O., and N.Z. performed the experiments with technical assistance by P.K.; N.Z., I.B., and O.A. prepared the biological samples; B.F. designed the phase masks with contributions from E.N.; N.Z. ran deconvolution based post-processing over EDOF samples; E.N. analyzed the three-dimensional nanoparticle tracking data; N.O., E.N., and Y.S. did the writing.
Corresponding author
Ethics declarations
Competing interests
Y.S., B.F., N.O, E.N, and R.K.O. have published a patent (WO2022259243A1) relating to the wavefront shaping aspect described in this work. P.K. is employed by Sartorius Stedim North America, Inc. The remaining authors declare no competing interests.
Peer review
Peer review information
Nature Communications thanks Yichen Ding, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Source data
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Opatovski, N., Nehme, E., Zoref, N. et al. Depth-enhanced high-throughput microscopy by compact PSF engineering. Nat Commun 15, 4861 (2024). https://doi.org/10.1038/s41467-024-48502-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467-024-48502-y
- Springer Nature Limited