RECONSTRUCTING DEPTH OF FIELD BY SOFTWARE PROCESSING
The solution relies on a software algorithm capable of reconstructing a full image based on a stack of multiple individual pictures. There are two popular image processing techniques based on image stacks: the first is called image stacking and consists of using multiple image of an object taken in the same conditions. Image stacking is effectively working on limitations dues to the imaging system (and especially digital camera sensors) by combining all the separate images into a single image with reduced noise, defects, and increased resolution. This reconstruction technique, very popular for astronomical photography, should not be confused with the second image processing technique called focus stacking. Focus stacking uses a stack of images taken in different focus conditions in order to reconstruct a complete image of the object.
Both commercial and free open sourced software incorporate or were specifically developed to apply image stacking and/or focus stacking on a set of images. A summary of commonly available software is listed in Table 1 along with the operating system on which it can be installed, as well as the type of license for its usage (free or commercial). For the purpose of this paper, focus stacking was tested on each of the listed software to compare the quality of the end result image. However, keep in mind that the majority of this software is designed for use by photographers using the "macro" mode of their camera. While this is compatible for use with opaque paleontological and geological objects, translucent palynomorphs may be only partially reconstructed as two overlapping areas (e.g., the proximal pole and the distal pole of a spore) can appear in focus at a different level. For this reason, among all of the available software listed in
Table 1, only CombineZM (Hadley 2006) will be discussed herein. This particular software was adapted to work on translucent objects and give significantly better results for palynological use.
The algorithm behind the "magic"
Several methods can be adopted for the image processing required to complete depth of field reconstruction. The method used by CombineZM consists of identifying zones within each frame that appear to be in focus. These zones are subsequently merged while out of focus areas are ignored. Identification of zones in focus is determined by using the amount of details visible in a frame, such as edges and dots: when parts of the object are out of focus, details will appear blurred or will not be seen at all. Details on a picture are usually recognizable as zones with well-defined edges and strong contrast, a characteristic that tends to disappear if out of focus. In a digitized image, this translates into the amount of variation that can be observed among neighboring pixels. To illustrate this,
Figure 2 shows a checkerboard pattern imaged in the focus plane (1) and out of focus (2). Indication that the checkerboard pattern A is actually in focus is indicated by the well-defined edges of each black and white square (representing a maximum variation of 100% (white) to 0% (black) on the edge of each squares). The checkerboard pattern in B depicts a much more gradual variation in the value of each adjacent pixel's brightness. The algorithm used in CombineZM uses this principle and quantifies the variations in the values of adjacent pixels, then sum these quantities thus giving each pixel a score. The areas around the pixels with the highest scores are then merged to form the composite image.
Method of reconstruction
CombineZM provides a set of two predefined routines (macros), each using a different method for reconstructing depth of field. These methods can be customized by editing the macros, (e.g. the sharpness and/or contrast of the resultant images can be adjusted to attain better results depending of the equipment used). Correct operation of the program requires the original pictures to be normalized; the reconstructed image consists of an overlay of pieces of the individual original frames, all the individual frames must be normalized to the same value of contrast, brightness and colour. Also, the object of interest must be scaled at the same size ratio and the same position on each frame in order to overlap correctly. CombineZM can perform all of these adjustments automatically using the top frame of the stack as the reference for adjusting all other pictures. Thus it is necessary to ensure that the brightness/contrast ratio as well as the colour balance is adequate on this first frame. Once normalized, two processing macros are proposed:
- Do stack
This macro selects areas that are identified to be in focus and uses them to reconstruct the object by stacking them on top of each other. Adjacent patches from different frames are merged into a composite image. However, this macro does not merge details that reside on different frames at the same location. Due to this limitation, details on the front, within, and on the back of transparent objects may not appear in the final reconstructed image.
- Do weighted average
This macro uses the weighted average of corresponding pixels in the stack. The weighting factor used is the score produced by summing the difference between each pixel and its neighbours on the same frame. As translucent objects such as palynomorphs create a challenge for the "Do Stack" macro above (having parts of the object that can overlap), this algorithm may be more suitable in dome cases. This macro overcomes the problem outlined earlier, however, its drawback is that because several frames may contribute to the image at each location there may be some loss of sharpness, some fogging, and the merging of objects that in reality are far apart along the optical axis.
Guidelines for photography
Regarding the operations conducted during software processing, a number of guidelines need to be followed during acquisition of the individual frames of the stack. Particular attention should be made to use the same settings for image acquisition on each frame. Even if the CombineZM software is able to correct disparities between frames, it is always preferable to keep them at a minimum. The camera should be set to a manual mode in which both white balance and exposure time can be kept identical for all pictures. Additionally, attention should be paid to avoid possible motion of the subject or of the optical setup (especially if the camera is mounted on the top of a long optical tube subject to vibration) between frames. Remote shutter release (electronic or cable) and mirror lockup (for single-lens reflex cameras (SLRs)) can prevent these vibrations.
The number of optical section pictures required for a particular stack is not critical, as long as at least each frame contains the focus interval to be reconstructed in the stacked image. However, there is no need to take pictures at close intervals, or to take duplicated pictures, as they do not add extra data and may alter the final image. Ideally, pictures should be ordered starting at the uppermost frame (processed first by CombineZM), to the lowermost frame. With the advances of digital photography, digital cameras, or even high-end SLRs, are commonly mounted directly on microscopes, and provide sensor resolutions in the tens of megapixels range. While creating large, ten-megapixel images may be useful for certain situations, processing a series of such images requires a tremendous amount of memory and processing power on the host computer running CombineZM. Not only will the processing time exponentially increase, but the quality of the output decreases as camera resolution increases (close examination of these images usually reveals that edges and details are less sharp at the pixel level). It is preferred to downsample such large images to a more acceptable value of five, two, or even one megapixel. Lowering the resolution not only speeds up the processing, but also artificially improves the identification of details, as the reduced number of pixel artificially increase the difference between neighboring pixels located at the edge of each detail. Remember that at a standard printing resolution of 300 dpi, a five megapixel picture of a palynomorph occupying the entire field of view of the camera represents an actual size of 22 x 16.5 cm, which surpasses the size requirements for publication, so downsampling images should not be a concern.
In addition to the CombineZM software, an extra plug-in called CZBatch was developed as a way to automatically apply a predefined set of operations to a group of stacks. Each group of frames representing a stack needs to be collected in an individual folder prior to running CZBatch. While the batch processing plug-in can dramatically speed up the processing of large numbers of stacks of pictures, it is recommended to carefully control frames that are chosen as input for each stack.
Resulting stacked pictures can be post-processed prior to assembling a palynological plate. Because of the stacking technique, all areas of the palynomorph now appear in focus, and the margins are sharp and clear, which allows for easy removal of unwanted background around each palynomorph. Several palynomorphs can then be assembled on a single plate with minimum lost space and on a more desirable, totally white background. On occasions the stacking process may introduce undesirable artifacts such as halos next to some edges, smearing of reflected and refracted highlights, trails of dots left when a single "bad pixel" or "dust spot" on the camera sensor appears in a different place on each frame after resizing and alignment, and erroneous patches often caused by noise in the original frames. Such artifacts can be easily edited manually in an image editing program.