2.1.

Multifocus Image Acquisition and Field-of-View Correction

The scheme of the custom-built fluorescence microscope is shown in Fig. 1. The main components include a camera sensor (Thorlabs CCD 8051-C USB 3.0, $3296\times 2472\text{pixels}$ resolution, $5.5\mu \mathrm{m}$ pixel pitch), an EFTL (Optotune EL-16-40-TC-VIS-5D-C), a LED with an emission peak at 385 nm and FWHM of 12 nm (Thorlabs LED M385LP1), a dichroic mirror (reflection 375 to 393 nm, transmission 414 to 450 nm) and a water immersion microscope objective lens (Olympus UMPLFLN $20\times$, numerical aperture $\mathrm{NA}=0.5$, focal length $f_{\mathrm{MO}}=9.00\mathrm{mm}$, working distance $\mathrm{WD}=3.5\mathrm{mm}$).

Fig. 1

Scheme of the multifocus custom built fluorescence microscope. LED: Light emission diode as excitation source with center wavelength at 385 nm, L: illumination lens system, DM: dichroic mirror, MO: microscope objective, EFTL: electrically focus-tunable lens. MCTS: multicellular tumor spheroid stained with DAPI (emission peak centered at 461 nm).

The biological sample used in this work is from human prostatic carcinoma cell line (LNCaP)²⁷ and was cultivated to form MCTS by means of the hanging drop method in which cells are suspended in droplets of medium where they develop into coherent 3D aggregates and are readily accessed for analysis.²⁸ Cells were then stained with DAPI with a broadband excitation centered at 358 nm and emission at 461 nm. An extra filter centered at 457 nm and bandwidth 22 nm was placed before the sensor in order to enhance the contrast of the images.

Parts of the sample to be captured in-focus are those placed at the conjugate image plane of the sensor, which is shifted from the working distance plane of the microscope objective by an amount $z$ given as

In-focus parts of the sample are obtained at the sensor with lateral magnification $M$ given as

The multifocus stack was acquired for a set of currents in the EFTL $j_k,k=1,\dots ,N$ between 265 and 125 mA in steps of $-10\mathrm{mA}$. The $N=15$ image stack is shown in Fig. 2. Note that field of view is not constant along with the stack of images and needs to be corrected before the synthesis of novel viewpoints from multifocus stack is performed.²⁹

Fig. 2

(a)–(o) 15 multifocus images acquired ($z$-stack) for currents in the EFTL between 265 and 125 mA in steps of $-10\mathrm{mA}$. See Video 1 for a visualization of the stack (Video 1, MP4, 0.2 MB [URL: https://doi.org/10.1117/1.JBO.27.6.066501.1]).

Since in the present work the EFTL is positioned in the set-up in a way that total intensity remains constant between the acquired images (illumination path is not affected by the change in focus of the EFTL) this allows us to use conservation of energy (i.e., integral of intensity values in a given image of the stack should be constant) to implement registration between images of the stack (note that in other works including an EFTL,²⁴ illumination intensity changes between the acquired images due to the position of the EFTL in the set-up, so conservation of radiant energy does not hold and registration needs to be performed following different approaches). Energy is evaluated for a given reference image (in our case $k=1$ image) and the rest of the images in the stack are rescaled to give the same value. Then the captured visual information is reorganized through a Fourier domain postprocessing approach which does not require depth-map estimation or segmentation of in-focus regions. Image reconstruction is accomplished considering only, besides an effective parameter, the current through the EFTL for each image in the stack.

2.2.

Image Formation Model and Novel Viewpoint Synthesis

Once the multifocus image stack is acquired, postcapture processing algorithms enable the synthesis of images with novel viewpoints of the scene.²⁵ Let $i_k$ be the intensity distribution of the $k$’th image of a stack of $N$ images. [For color images in $RGB$ space $i_k=(i_k^R,i_k^G,i_k^B)$, $k=1,\dots ,N$]. The $i_k$ image taken with current $j_k$ through the EFTL can be described, neglecting noise, and chromatic aberrations, by the following equation:

Let us consider the Fourier transform (FT) of the set of $N$ coupled Eq. (4), and arrange them in vector form as

If $H(u,v)$ is invertible, then the solution to the linear system given by Eq. (8) is $\overrightarrow{F}(u,v)=H^{-1}(u,v)\overrightarrow{I}(u,v)$, but if $H(u,v)$ is not invertible (as for the DC frequency components), then a solution to the system may be found through the Moore–Penrose pseudoinverse $H^{†}$.³⁰ The Moore–Penrose pseudoinverse provides the set of vectors that minimize the Euclidean norm $\Vert H(u,v)\overrightarrow{F}(u,v)-\overrightarrow{I}(u,v)\Vert$ in the least squares sense. Thus, the minimal norm vector is given as

The reconstruction of an arbitrary horizontal viewpoint of the scene is accomplished by simulating the displacement $b_x$ of a pinhole camera in the horizontal direction with respect to the center of the original pupil (similarly for a $b_y$ displacement in the vertical direction). The horizontal disparity $d_k$ between the images of a given point of the in-focus component $f_k$ as seen by the sensor of a centered pinhole camera and a pinhole camera displaced to the left is given as

(aside from a constant factor independent of $k$ and related to the magnification at zero current). Then, in the piecewise planar approximation of the 3D scene, to obtain a shifted viewpoint $s_{b_x}(x,y)$, each focus slice $f_k(x,y)$ should be shifted in an amount according to the disparity associated with the $j_k$ current through EFTL and the baseline displacement $(b_x)$ of the camera

In particular, $s_0(x,y)$ recovers the image as captured with a pinhole camera in the center of the original circular pupil (i.e., extended-DoF or all-in-focus image reconstruction of the scene²³).

By means of the FT shift theorem, which states that translation in the space domain introduces a linear phase shift in the frequency domain,³¹ and by using Eq. (10) for $\overrightarrow{F}(u,v)$, we obtain the FT of Eq. (12):

Let us now consider the baseline displacement $b_x$ as a fraction ${\beta }_x$ ($|{\beta }_x|\le 1$) of the pupil $R$ (since displacements outside of the aperture have no physical meaning)

Finally, by Fourier inverse transform of Eq. (15) we obtain the new scene perspective as seen from a pinhole camera, translated a fraction ${\beta }_x$ to the left of the center of the original circular pupil²⁵

In order to achieve visualization with full parallax, it is straightforward to extend Eq. (15) to the case of vertical motion simulation

The proposed method is then able to reconstruct the extended DoF and allows the visualization of the reconstructed scene from different perspectives without previous segmentation of the focused regions from the images in the stack. However, it is possible to retrieve the depth map by combining this method with other schemes.³²

3.1.

Extend Depth-of-Field and Viewpoint Synthesis

Reconstruction of the extended DoF or all-in-focus image corresponds to ${\beta }_x=0$, (i.e., as seen with a centered pinhole camera) and is shown in Fig. 3. Note how, unlike original images of the stack in Fig. 2, individual cell nuclei from different depths of the aggregate can be clearly seen in a single image.

Fig. 3

Extended DoF for a virtual centered pinhole view (${\beta }_x=0$, ${\beta }_y=0$); see Video 2 for a complete set of novel viewpoints corresponding to $-0.25\le {\beta }_x\le 0.25$ and $-0.25\le {\beta }_y\le 0.25$ (Video 2, MP4, 1.0 MB [URL: https://doi.org/10.1117/1.JBO.27.6.066501.2]).).

If we instead consider arbitrary fractional displacements ${\beta }_x$, ${\beta }_y$, the corresponding viewpoints can be synthesized from Eqs. (16) and (18), respectively (combination of horizontal and vertical viewpoints is straightforward). A complete set of novel viewpoints for $-0.25\le {\beta }_x\le 0.25$, $-0.25\le {\beta }_y\le 0.25$ is available in Video 2.

3.2.

Stereoscopic Pairs for 3D Visualization

Binocular vision is based on the fact that 3D objects are perceived from two different perspectives due to the horizontal separation between our left and right eyes. As a result, the left and right images of a 3D scene in our retinas are slightly different. This retinal disparity between the images provides the observer with information about the relative distances and depth structure of 3D objects. Both perspectives of the same 3D scene are fused by the brain to give the perception of depth.^33,34

In a similar way, a pair of stereoscopic images can be generated by considering a virtual stereocamera³⁵ formed by a left pinhole camera displaced to the left of the center of the original pupil, $b_x=B/2$, and a right pinhole camera displaced to the right of the center of the original pupil, $b_x=-B/2$, where the separation $B$ between the left and right virtual pinhole cameras is known as the baseline. Since points of view from outside of the aperture have no physical meaning, $B\le 2R$.

Then, it is straightforward to reconstruct the left and right views according to

Fig. 4

Stereoscopic pair for cross eyed visualization. (a) Reconstructed and (b) perspective images [images corresponding to pinhole virtually displaced to the right and to the left, respectively].

3.3.

Performance Assessment

Quantitative comparison can be performed with the help of a synthetic multifocus stack since, unlike the real stack, a ground-truth reference for each point of view of interest can be constructed. Figure 5(a) shows the synthetic 3D scene representing three rings of fluorescent beads with each ring lying on a different plane at distances $z_i,i=\mathrm{1,2},3$. Images of the stack corresponding to the system focusing on each of these planes (for currents $j_i,i=\mathrm{1,2},3$) are shown in Fig. 5(${\mathrm{b}}_{1-3}$), respectively.

Fig. 5

Synthetic stack of fluorescent beads and reconstruction of viewpoints. (a) 3D scene. (${\mathrm{b}}_{1-3}$) Images of the stack corresponding to $R_0\approx 12.7{\mathrm{mA}}^{-1}$ and the system focusing for currents through the EFTL 50, 33.3, 0 mA, respectively. (${\mathrm{c}}_{1-3}$) Ground-truth for fractional displacements ${\beta }_x=-0.5,0,0.5$, respectively [vertical dashed guideline passing through a bead in the central viewpoint (${\mathrm{b}}_2$) added to visualize more clearly the displacements]. (${\mathrm{d}}_{1-3}$) Reconstructed viewpoints for fractional displacements ${\beta }_x=-0.5,0,0.5$, respectively.

Figure 5(${\mathrm{c}}_{1-3}$) shows the ground truth for the scene as viewed from a pinhole camera displaced to the left, center, and right, for relative displacements ${\beta }_x=0.5,0,-0.5$, respectively. The multifocus stack of Fig. 5(${\mathrm{b}}_{1-3}$) is used to render the same viewpoints and the results obtained by means of Eq. (16) are presented in Fig. 5(${\mathrm{d}}_{1-3}$), respectively. Table 1 shows the mean square error resulting from the comparison (of luminances) against the ground truth for each relative displacement, showing a very good agreement between the reconstructed viewpoint and its corresponding ground truth.

Table 1

MSE values from comparison against ground-truth for different horizontal relative displacements βx.