1.

Introduction

All range images begin with a series of range measurements, and the quality of the range image depends on the quality of each of those measurements. The quality of a range measurement depends on measurement uncertainty and measurement resolution; however, spatial uncertainty is also strongly affected by environmental factors, such as return signal intensity and relationship to a measurement’s immediate neighbors. One or more of these factors can be expressed as a metric representing the deviation of some quality attribute associated with the measurement from a predefined standard. Spatial measurement quality represents the degree of confidence one can place in how accurately a measurement represents the position of a real surface in the environment. Laser range scanners can also provide intensity information that may be used in representing the surface so quality attributes relating to return signal intensity are useful. In this paper, contemporary approaches to evaluating measurement quality attributes are reviewed, including measurement uncertainty, return signal intensity, range, sampling density, and relationship to neighboring points. This review focuses particularly on measurement quality metrics for ground-based laser range scanners that can be adapted for automated systems. Within this context, measurement quality metrics provide a way to direct and terminate automated scanning procedures.

Perceptual quality metrics can be either objectively or subjectively defined;¹ however, only objective quality metrics are useful for automating data acquisition. For this reason, only objective quality metrics are considered here. Objective quality metrics can be further classified as referenced or unreferenced.¹ Unreferenced quality metrics use no benchmark; thus, automated processes can only evaluate the change in a quality attribute in response to some action. Referenced quality metrics can be evaluated in the same manner as unreferenced quality metrics, but the size of the deviation of an attribute from a reference can be used to evaluate whether the measurement should be either retained or ignored. For this reason, referenced quality metrics are preferred for automated systems in which thousands, or even millions, of measurements may be obtained.

Referenced range measurement quality metrics quantify the relationship of a quality attribute to some previously established benchmark or reference. These metrics can then be used to either compare methods or systems, or they can be used in an iterative process to maximize some qualitative attribute of a range image.² Quality metrics appear most often in the guise of a weighting parameter when merging measurements or data sets. Two important components of a referenced quality metric are a clearly defined quality benchmark against which to compare the current state of the range image, and a quality scale to indicate the degree to which the range measurement quality attribute deviates from the benchmark.

In this paper, metrics for quantifying the quality of measurements are reviewed. For purposes of discussion, these metrics have been classified as measurement uncertainty based, signal intensity based, range based, and neighborhood based. As will be demonstrated, considerable work remains to ensure that the quality of measurements and points used to construct virtual models is effectively and comprehensively defined.

2.

Spatial Resolution

The spatial resolution of a laser range scanner measurement is dependant on the size of the laser spot that illuminates the surface at the point the measurement is obtained. For pulsed laser systems, the spatial resolution is also dependant on the pulse length of the system. The spatial resolution can be divided into range resolution and angular resolution. Angular resolution is the minimum angular distance between features such that they can be resolved as separate features. Range resolution is the minimum distance between angularly resolved features such that they can be distinguished as separate features.³ The angular resolution of a laser range scanner is defined by the Rayleigh criterion⁴ and represents the size of the smallest feature that can be angularly resolved.^{3, 5}

The laser projects a spot into the surface being scanned, and the region in which the surface intersects the laser spot is referred to as the beam footprint. Features within the footprint contribute to the return signal intensity, which is used to obtain the spatial measurement that approximates the position of a portion of the surface.⁶ The area covered by the beam footprint is generally not measured by laser range scanners; thus, it is approximated by a model of the area of the laser spot that illuminates the area. Ideally, the laser spot area should be the same as the beam footprint area; however, environmental factors, such as spatial discontinuities⁷ or dense fog,⁸ can result in the beam footprint deviating from that predicted by the laser spot model. Moreover, if the surface normal is assumed to be oriented along the line of sight in the laser spot model, then surface angulation can result in a discrepancy between actual and predicted beam footprint areas. Quality metrics provide a way to predict by how much the beam footprint of a measurement might deviate from that predicted by the laser spot model.

The spatial resolution of a measurement can be represented by the instantaneous resolution, which assumes the footprint is stationary at the time the measurement is acquired, or the effective resolution, which takes into account the procession of the footprint over the surface during the acquisition process. When the term resolution is used in this paper, unless otherwise stated, it refers to the instantaneous resolution. The term footprint and laser spot are also used interchangeably in this paper, although the terms are strictly equivalent only when the surface is continuous within the laser spot.

3.

Measurement Uncertainty-Based Metrics

Measurement uncertainty is the most common attribute used to assess measurement quality. Range measurement uncertainty is generally modeled as an independent zero-mean Gaussian process added to the quantity returned by the range sensor; that is,

The uncertainty associated with the range sensor is referred to here as the radial error and is one attribute that can be used to evaluate measurement quality. Rotational or translational position are referred to here generically as positional error and represent two more attributes which can be employed to evaluate the quality of a measurement. Figure 1 shows one example of a triangulation laser range scanner system in which the angular position of the laser spot on a surface in the environment is controlled by two rotating mirrors. Similar dual-axis optical scanning configurations are used in time-of-flight (TOF) systems and other laser range systems by combining orthogonal galvanometers, rotating mirrors, or motors. As a result, the geometrical model and measurement uncertainties can be generalized to a variety of laser range scanning systems.

Fig. 1

Example of a fixed-viewpoint laser range scanner employing dual-axis galvanometer-controlled rotating mirrors [modified from Fig. 6(b) of Ref. 10].

3.3.

Radial Uncertainty

Range measurement uncertainty depends on how the interaction of the laser with the surface is measured. In TOF systems, the range is determined by the time between the pulse being generated and being detected. In triangulation systems, the range measurement depends on the position of the signal peak on a photodetector array. In both cases, a significant portion of the range measurement uncertainty is the ambiguity of the location of the signal peak.

Range uncertainty is typically assumed constant for TOF scanners, as shown in Fig. 2 . Specifically,

Eq. 2

$${\sigma }_R=\frac{c}{2}{\sigma }_{\tau },$$

where ${\sigma }_R$ is the range measurement error, $c$ is the speed of light, and ${\sigma }_{\tau }$ is the time measurement error. The last term represents the uncertainty in the temporal location of the signal peak. This is found by

Eq. 3

$${\sigma }_{\tau }=\sqrt{\frac{T_r^2}{\mathrm{SNR}}},$$

where $T_r$ is the pulse rise time and SNR is the signal-to-noise ratio.^{15, 16} The range measurement error is determined by the signal bandwidth,¹⁵ amplitude of the return signal,¹⁷ thermal drift,^{17, 18} crosstalk between the transmitter and receiver,¹⁸ timing jitter,¹⁹ and nonuniformities and changes in the returning signal shape.^{15, 18, 19} For example, different surface materials can change the shape of the return signal resulting in significantly different error distributions.¹⁰ Moreover, feedback within the sensor can result in a measurement being affected by the previous measurement, violating the assumption that there is no correlation among range measurements.

Fig. 2

Common range uncertainties for Amplitude-Modulation Continuous Wave (AM), Frequency-Modulation Continuous Wave (FMCW), Time-of-Flight (TOF) and triangulation scanners up to $100\text{-}\mathrm{m}$ effective range (referred to in this figure as volume). The range measurement uncertainties of all but the triangulation scanners are considered constant with respect to range. [modified from Fig. 2(b) of Ref. 16].

Laser motion while the signal is being emitted is negligible for pulse TOF systems because the pulse duration is so short, but it can affect continuously modulated laser systems. This motion can distort the return signal and introduce ambiguity into the true measurement. Consider, as well, that the range measurement equation is given by

Eq. 4

$$R=\frac{c\tau }{2},$$

where $\tau$ is the propagation delay.^{16, 20, 21} This assumes that the TOF between the laser and the surface is equal to the TOF between the surface and the sensor. This may not always be the case, especially if the true origin of the laser pulse varies as a function of the mirror angles.

The peak uncertainty of a triangulation scanner is typically dominated by speckle noise.²² This can be modeled as

Eq. 5

$${\sigma }_a=\frac{\lambda f}{D\cos \left(\beta \right)\sqrt{2\pi }},$$

where $\lambda$ is the laser wavelength, $f$ is the focal length of the receiving lens, $D$ is the diameter of the collecting lens, and $\beta$ is the Scheimpflugg angle of the photodetector array.²³ Speckle noise arises when speckle elements on the surface illuminated by the laser spot are large when compared to the wavelength of the laser light.^{22, 24} Under this assumption, each speckle element becomes a point emitter with respect to the photodetector array. Interference patterns are generated when each speckle element reflects light from the laser onto the photodetector array,^{22, 25} as shown in Fig. 3 . There, they constructively and destructively interact to form a speckle image on the photodetector array.^{22, 24, 25}

Fig. 3

Speckle noise arises from the interference of a series of diffraction patterns, each generated by a speckle element. (modified from Fig. 2.11 of Ref. 25).

Speckle noise is generally countered by integrating a single measurement over several intensity samples as the laser spot is moved over the surface being scanned.²⁶ Figure 4 illustrates the reduction in speckle after integration. This is complicated by the need to minimize aliasing by ensuring that the measurements are, where possible, separated by a distance less than the radius of the laser spot (Fig. 4).⁷ Similarly, the range uncertainty in an amplitude-modulation continuous-wave scan can be decreased by increasing the sampling rate.²⁷

Fig. 4

Speckle noise is reduced by integrating the measurement over several sampling intervals. (modified from Fig. 3 of Ref. 23).

3.4.

Environmental Effects

The mechanical effects described in Sec. 3.3 can be included in a model of expected range and rotational uncertainty; however, many environmental factors, summarized in Table 1 , can cause the true measurement uncertainty to deviate from the model. For example, measurement uncertainty can increase with increasing incidence angle,^{28, 29, 30, 31} a reduction in surface reflectivity,^{10, 32} and an increase in ambient lighting.^{6, 33}

Table 1

Environmental factors affecting measurement uncertainty.

Error Source	Effect
Range	Range uncertainty generally increases with range
Angle of Incidence	Range uncertainty increases with increased angle of incidence
Surface Material	Translucent non-homogeneous materials increase range uncertainty
Surface Complexity	Surface discontinuities introduce range errors
Reflectivity	Range uncertainty increases with a decrease in reflectivity
Ambient Lighting	Range uncertainty increases with an increase in ambient lighting

Equation ⁵ assumes that the size of the spot projected onto the photodetector array has not been distorted by occlusion, surface orientation, or other environmental effects. Figure 5 shows the effect of laser spot distortion arising from a surface discontinuity.⁷ In this case, the discontinuity occludes part of the laser spot so that the spot centroid no longer coincides with the signal peak. This introduces an error into the horizontal location of the signal peak, denoted here as $\Delta x$ . This results in a range error $\Delta z$ , which is compounded by the surface orientation with respect to the direction of the laser. The deviation of the surface normal from the laser path is denoted here as $\gamma$ . Sudden changes in surface height are not uncommon and represent a reduction in measurement quality that is not captured by model-based measurement uncertainty.

Fig. 5

A range discontinuity results in a shift $\left(\Delta x\right)$ in the position of the centroid in a triangulation laser range scanner. This results in a range error $\Delta z$ . [modified from Fig. 7(a) of Ref. 7].

Different surface materials can also affect the accuracy of range measurements. Figure 6 demonstrates the effect of a partially translucent material, such as marble, in which the laser may penetrate part way into the surface before sufficient light is reflected to estimate the distance to the surface.⁷ In this case, the range measurement does not represent the surface of the material, and the actual range measurement obtained depends on the reflective and refractive qualities of the material. According to Beraldin ,³⁴ translucent surfaces like marble change the shape of the laser spot on the photodetector array of a triangulation scanner, resulting in the range estimate being in error. As well, the nonhomogeneity of the material increases the range measurement uncertainty.¹⁶ Translucent nonhomogeneous materials can also feature a greater measurement uncertainty as well as a bias that increases with the distance between the scanner and the surface.³⁵

Fig. 6

Range errors can result from the laser penetrating the surface of the material being scanned. (modified from Fig. 6 of Ref. 7).

Surface complexity is not limited to variations in the height and frequency of surface structures; transitions between areas of different surface reflectivity can affect the accuracy of a range measurement,^{7, 36} as illustrated in Fig. 7 . Different materials with different reflectivity properties can also generate very different range measurement uncertainties.¹⁰ The change in reflectivity for different portions of the laser spot results in a shift in the signal peak that introduces an error into both the range measurement and return signal intensity, a topic discussed in Sec. 4. Moreover, a reduction in surface reflectivity can result in an increase in range measurement uncertainty.³³

Fig. 7

Transitions between regions of different surface reflectivity can affect the accuracy of the range measurement (Fig. 1 of Ref. 36).

Increasing the surface orientation with respect to the line of sight of the scanner can result in an elongation of the laser spot, which increases peak detection uncertainty.³¹ This problem is most pronounced when the length of the baseline is significant with respect to the distance to the surface, as is the case with triangulation laser range scanners, even when operating in the far field. Moreover, increased surface orientation with respect to the line of projection of the laser increases the spot size on the surface, resulting in more speckle elements contributing to the spot projected onto the photodetector array. Because the range uncertainty of triangulation laser range scanners is dependant on the surface orientation, model-based range uncertainty is not sufficient to represent the quality of a range measurement.

4.

Signal Intensity-Based Metrics

It was noted in Sec. 3.4 that a decrease in surface reflectivity can result in an increase in measurement uncertainty. Surface reflectivity can be assessed by examining how the intensity of the received signal varies from what would be expected for a surface of known reflectivity; however, signal intensity measurements can vary significantly as a result of such factors as range,³² high incidence angles,^{28, 31, 32} low reflectivity,^{10, 18} atmospheric attenuation,⁴⁴ sharp discontinuities,^{16, 45} and translucency of the material being scanned.^{16, 40} For example, the return signal intensity decreases with an increase in angle of incidence and decreases with an increase in distance between the scanner and the surface when the transmitted signal power remains constant.³² As a result, quality metrics provide a way to predict the extent to which the actual reflectivity of a surface might deviate from that predicted from the return signal intensity.

Figure 5 illustrated that surface discontinuities can result in range errors; however, a change in the shape of the signal intensity profile in a triangulation laser range scanner can also result in a reduction in return signal intensity. When the shape of the peak is sufficiently distorted, as is the case with mixed measurements, it becomes difficult to locate its centroid. Laser spots that cross edges can result in smeared or multiple return signals that result in ambiguous range measurements, what is referred to as mixed measurement error.^{32, 46} Mixed measurements are a result of receiving reflected energy from two surfaces within the laser spot and are often interpreted as a range measurement somewhere between the two surfaces.^{6, 33, 47} Hebert and Krotkov⁶ referred to the interdependence of measured range with signal intensity as range/intensity crosstalk. TOF systems calculate range by comparing the return signal to the transmitted signal, thus are more sensitive to signal intensity changes. Figure 8 shows that a discontinuity in surface reflectivity can also reduce the return signal intensity.⁷ As a result, quality metrics provide a way to predict the extent to which the spatial position of the measurement might be in error as a result of the return signal intensity deviating from that predicted using a model of the laser range scanner optics.

Fig. 8

Discontinuity in surface reflectivity results in a shift $\left(\Delta x\right)$ in the position of the centroid in a triangulation laser range scanner. This results in a change in return signal intensity. [modified from Fig. 7(b) of Ref. 7].

Some surfaces may be difficult, if not impossible, to scan because the return signal is diffusely scattered, what is referred to as volumetric scattering.^{46, 48} Surfaces that exhibit this property include glass, hair,⁴⁶ and grass.⁴⁸ Figure 6 illustrates that translucent materials can also reduce the strength of the return signal.⁷ Other surfaces are excessively absorbent so the signal is of insufficient intensity to obtain a range measurement, while other surfaces may be very highly reflective that the photodetector is saturated.⁴⁶ The absence of a return signal, referred to as a nonreturn measurement, can be a valuable piece of information but is almost always discarded.

Given a reference material, the change in return signal intensity can be modeled as a function of range. A shift in the return signal intensity from the model value can then be used as a metric of the quality of a measurement. Measurement spatial uncertainty is also affected by return signal intensity; thus, both, variables are important in assessing measurement quality, and neither are sufficient by themselves. Moreover, signal intensity shifts can indicate the presence of mixed pixels and surface material transitions, either of which may introduce errors into the range measurement. The challenge is in how to determine the cause of the intensity shift, given that only the spatial position and deviation in signal intensity from a model value are known. Deviations from model return intensity can arise from several different environmental conditions; therefore, return intensity, even when combined with spatial position and model spatial uncertainty, is not sufficient to completely represent the quality of a measurement. Table 2 summarizes the factors that affect return signal intensity.

Table 2

Environmental factors affecting return signal intensity.

5.

Range-Based Metrics

It was noted in Sec. 3.3 that measurement spatial uncertainty generally increases with increased range and, in Sec. 4, that return signal intensity generally decreases with increased range. The range measurement itself can be used to represent the quality of a measurement. For example, Sequeira ^{37, 38} and Fiocco ⁵⁰ each used the range portion of the measurement as part of their reliability metrics. Figure 9 graphically demonstrates how the quality of a measurement decreases as the distance between the scanner and the surface that generated the measurement increases.

Fig. 9

Assuming the scanner position is limited to within a metre or two of ground level, the lateral measurement uncertainty increases as the height of the structure increases due to the increase in the distance between the scanner and the surface.

In general, the farther a surface is from the scanner, the larger the area encompassed within the laser spot. The size of the spot projected onto a surface is represented by the beam width at the point of intersection. The beam width depends on the distribution of irradiance, which is often assumed to follow a Gaussian distribution. Specifically,

Fig. 10

Laser beam $1∕{\mathrm{e}}^2$ boundary. The far field is the region in which $\zeta ⪢{\zeta }_0$ .

The surface formed by $w\left(\zeta \right)$ represents the distance $r$ from the central axis at which the beam irradiance falls to $1∕e^2=0.135$ . As a result, the volume bounded by $w\left(\zeta \right)$ represents the region within which 86.5% of the beam irradiance is contained.^{51, 52, 53} The laser spot defined in this way represents the portion of the surface being scanned from which most of the laser irradiance is being reflected. As a result, the laser spot represents the smallest region that can be resolved by the laser range scanner.

The boundary of $w\left(\zeta \right)$ can be approximated by the hyperbolic equation,

Range can act as a proxy for the resolution of a measurement under the assumption that the focal length remains fixed and the surface is farther from the scanner than the beam waist. Under these conditions, measurements closer to the scanner can be considered to be of higher quality than those farther from the scanner. Although range is generally not referred to as an indicator of the quality of a measurement, this relationship is implied when the more distant of a pair of measurements is dropped as part of the registration process.

Fiocco ⁵⁰ defined a distance quality metric based on the minimum and maximum range limits. In practice, a scanner is bounded by the minimum and maximum effective range, defined by a variety of factors, including the laser power, beam spread, and photodetector sensitivity. Sequeira ^{37, 38} simply applied a weighting factor to the range measurement to obtain a quality metric.

Only long range scans (those for which $\zeta ⪢{\zeta }_0$ , referred to as far-field measurements⁵¹) are guaranteed to have a measurement resolution decrease with range. Medium range scanners may be used for surfaces that are at, or even less than, the distance to the beam waist. Surfaces that are closer than the beam waist have an inverse relationship between resolution and range, as shown in Fig. 10. In this case, measurement quality decreases with distance. As a result, Fiocco ’s and Sequeira ’s methods are only applicable to the situation for which they were designed; laser range scanners in which the surface is farther from the scanner than the beam waist. For medium-range scanning, the surface may be placed such that it coincides as much as possible with the beam waist. A more general-purpose resolution-based quality metric should be applicable to both long and medium range scanner data, as well as data from scanners with multiple focal lengths. The use of laser spot size in assessing measurement quality will be addressed in Sec. 6.

6.

Neighborhood-Based Metrics

Attributes such as surface orientation, or spatial or reflectivity discontinuities, cannot be determined from single measurements; they can only be inferred from groups of measurements located in close spatial proximity to each other. Spatially related measurements are referred to here as a neighborhood and are used to model a small portion of the surface being scanned to predict some aspect of that surface, such as its orientation. The class of neighborhood-based metrics encompasses all quality metrics defined by the neighborhood of a measurement.

Neighborhood-based quality metrics attempt to infer some aspect of a measurement by its relationship to its immediate neighbors. For purposes of discussion, a neighborhood is defined as a point $\widehat{\mathbf{p}}$ and the set of all points $P=\{{\widehat{\mathbf{p}}}_0,\dots {\widehat{\mathbf{p}}}_K\}$ considered to be the immediate neighbors of $\widehat{\mathbf{p}}$ by some commonly accepted criteria. It is assumed that this criterion is either the Euclidean or the rotational distance, although the discussion could apply to other distance metrics.

Two neighborhood-based quality metrics are considered: those based on interpoint distance and those based on vertex orientation with respect to the line of sight. The former is a measure of the density of the measurements in a neighborhood, which, in turn, indicates how finely the surface has been sampled. The latter is used to estimate the orientation of the surface at a spatial location of the measurement and is the most commonly used quality metric after measurement uncertainty.

Surface complexity can also be evaluated using edge-detection techniques. Specifically, spatial (illustrated in Fig. 5) and intensity (illustrated in Figs. 7 and 8) discontinuities result in range measurement errors so the quality of measurements corresponding to discontinuities are of lower quality than measurements arising from surfaces without discontinuities. Edge detection, applied to either spatial data, intensity data, or both, can be used to detect the presence of discontinuities, which are one type of surface complexity. A complete review of edge-detection techniques is, however, beyond the scope of this paper. For surveys on edge-detection techniques, see Argyle,⁵⁴ Davis,⁵⁵ Peli and Malah,⁵⁶ Ziou and Tabone,⁵⁷ Trichili ,⁵⁸ Xiao ,⁵⁹ and Basu.⁶⁰

6.2.

Orientation Metrics

A typical approach to generating the orientation of a measurement is to obtain a mesh model of the surface and use the normals of each of the mesh elements to estimate the normal of the surface at the measurement.^{22, 42, 69, 70, 71} Orientation is often represented by the surface normal, which is generally found by taking the average of the normals of all Delaunay facets that have this measurement as a vertex.^{42, 70} The exception is Hoppe ,⁷² who preferred to use the normal of a plane fit to the neighborhood of the measurement. The benchmark for the grazing angle attribute is the angle that generates the most accurate range measurement; that is, when the surface normal is oriented along the line between the surface and the scanner. Assuming the maximum grazing angle is one in which the surface normal is perpendicular to the line between the surface and the scanner, the scale of the grazing angle attribute is from 0 (best quality) to $\pi ∕2$ (worst quality) radians. This is often represented as the cosine of the grazing angle,^{30, 70} which has a range of 1 (best quality) to 0 (worst quality).

Often the deviation of the return signal intensity from the ideal Lambertian model is represented by the surface normal^{25, 42, 69, 70} or grazing angle.⁷³ The reasoning is that the signal intensity decreases with increasing surface orientation; thus, surface orientation can be used as a proxy for signal intensity. However, return signal intensity is affected by all the factors summarized in Table 2. Therefore, this assumption is true only in the absence of other factors, such as surface spatial complexity and changes in surface reflectivity. Surface orientation also affects the uncertainty of range measurements,⁷⁴ particularly for triangulation laser range scanners; thus, surface orientation as a metric can affect quality metrics for both spatial uncertainty and return signal intensity.

Fiocco ⁵⁰ used the deviation of the line of sight to the scanner from the surface normal as a quality metric. This metric took the form

Eq. 21

$$\phi =1-\frac{{\phi }_i}{90},$$

where ${\phi }_i$ is the surface orientation deviation in units of degrees. Turk and Levoy⁷⁰ used the cosine of the grazing angle to weight measurements prior to ICP registration. Soucy and Laurendeau³⁰ showed that the squared cosine of the grazing angle corresponds to the relative illuminance received by the photodetector. They used this metric to perform a weighted merge of measurements from different viewpoints such that

Eq. 22

$$\mathbf{x}=\sum _{i=1}^N{W}_i{\widehat{\mathbf{x}}}_i,$$

where

Eq. 23

$$W_i=\frac{{\cos }^2\left({\gamma }_i\right)}{{\sum }_{j=1}^N{\cos }^2\left({\gamma }_j\right)}$$

is the weighting factor associated with measurement ${\widehat{\mathbf{x}}}_i$ . The cosine of the grazing angle can be found using

Eq. 24

$$\cos \left({\gamma }_i\right)=\frac{{\vec{n}}_i^T{\widehat{\mathbf{x}}}_i}{{\widehat{R}}_i},$$

where ${\widehat{\mathbf{x}}}_i$ is a measurement located ${\widehat{R}}_i$ units from the viewpoint, and ${\mathbf{n}}_i$ is the normal to the surface at ${\widehat{\mathbf{x}}}_i$ . In this case, the coordinate system is assumed centered on the scanner viewpoint. Curless²² employed a similar approach to merging measurements that co-occupied the same voxel. Soucy and Laurendeau³⁰ demonstrated that the reflectivity of the surface was directly proportional to the square of Eq. ²⁴. Because measurement quality was expected to be directly proportional to the amount of light returned to the sensor, ${\cos }^2\left(\gamma \right)$ would better represent measurement quality than $\cos \left(\gamma \right)$ ; however, this was based on the assumption that the reflectivity change was primarily caused by high surface orientation. The relationship is less clear when the surface reflectivity is more complex.

Scott ⁷³ suggested that basing quality solely on the grazing angle of a measurement ignores the objective effects of high grazing angle in favor of a more subjective metric. Surface orientation, in particular, ignores factors that affect the shape and peak height of the intensity profile, such as surface reflectivity changes. Moreover, the surface normal is the average of the orientations along each Delaunay edge extending from a point. As a result, it is possible to have a wide range of vertex normals but a surface normal oriented along the line of sight. Finally, for systems in which the baseline is not insignificant with respect to range, the line of sight could be defined with respect to the photodetector, the laser, or the scanner origin, each yielding a different result. As a result, surface orientation is important but insufficient as a quality metric.

An alternative to grazing angle for representing surface orientation of a range image obtained using a raster scan pattern is the facet edge length ratio. In this case, the ratio of longest to shortest edge of a Delaunay facet is used to assess the quality of the facet and, by extension, its measurements. Sequeira used this approach to discard the facet if the ratio was too large.³⁷ Consider the image on the left in Fig. 11 , which represents a two-dimensional Delaunay triangulation of a range image; when seen in three dimensions, facets on a discontinuity are elongated with respect to their neighbors. The ratio between the longest and shortest edge should ideally be 1:1; that is, the triangles should be equilateral. As the surface orientation increases with respect to the line of sight from the scanner, the ratio between the longest and shortest edges increases. Specifically, given a facet $F_i$ with edges $E_i=\{e_{i,1},e_{i,2},e_{i,3}\}$ , the facet edge ratio $w_i$ can be found by

Eq. 25

$$w_i=\frac{\min E_i}{\max E_i}∊(0,1].$$

The weighting factor $w_i$ decreases toward zero as the disparity between the longest and shortest edges increases.

Fig. 11

Range discontinuities result in elongation of Delaunay facets when viewed in three dimensions (Modified from Fig. 4 of Ref. 38).

The facet ratio represents a quality metric in which the neighborhood is limited to the three measurements bounding the Delaunay facet. High-quality measurements would be those in which the facet ratio was close to 1, whereas those in which $w_i$ was very small would be considered to be low-quality measurements. Low-quality measurements would have elongated facets indicating steep surface slopes. A drawback of this method is that it is specifically designed to assess the quality of facets and can only be applied to measurements as a side benefit. Moreover, it is specifically designed to work with regularly spaced raster patterns. Nonraster patterns can feature large edge ratios, even if the surface is relatively flat, as illustrated in Fig. 12 . Arrangements in Figs. 12, 12, 12 contain facets with large facet ratios regardless of the range value associated with them. Although well suited to the purpose for which it was designed, facet ratio is not easily adapted to use as a general-purpose quality metric representing surface orientation. Fiocco’s method as well as the more popular grazing angle metric described by Eq. ²⁴ are better suited as general-purpose surface orientation quality metrics.

Fig. 12

Facet ratio is most effective for regularly-spaced data, such as (a) and (b). As measurement distribution becomes less regular, facets with large facet ratios can emerge regardless of their relative range measurements.

7.

Total Quality Metric

Quality metrics are generally combined to generate an overall measure of quality, referred to here as a total quality metric. Scott ² cited two common examples of how quality metrics could be combined: weighted summation and composite binary pass/fail. The weighted summation approach takes the form

The choice of how quality metrics are combined depends on the application and the relative weight placed on each of the quality metrics. The weighted summation approach allows the researcher to tailor the contribution of each of the quality metrics to the overall measurement quality without any one metric dominating the result. For example, Fiocco ⁵⁰ experimentally derived the weights for each sensor used in the experiment. They also standardized the weighting factors such that each sensor technology could be represented by a single weighting factor that modified each of the metric weights. Sequeira ^{37, 38} also used the weighted-sum approach but did not indicate how the weights were derived. The binary product approach is effective if the goal is to simply exceed some preset quality level.

8.

Unresolved Quality Issues

Several quality attributes are notably absent from contemporary, and even emerging, quality metrics. In particular, no quality metric has been developed to address the motion of the laser spot during the acquisition process. This is of particular interest in triangulation scanners where multiple sample intervals may be integrated to combat speckle noise. No quality metric has been defined to quantify the effect of measurement resolution. Even using range as a quality metric only addresses measurement resolution by proxy. In fact, neighborhood-based metrics do not consider the issue of measurement density or proximity that is less than the measurement resolution of the system. No metric has addressed the problem of measurement repeatability, most likely because it requires multiple range images of the same surface, which substantially increases scanning time. Finally, surface complexity is only imperfectly evaluated using surface orientation.

Measurement quality metrics are rarely combined into a total quality metric. As a result, operations such as measurement merge, range image registration, and deciding whether or not to delete a measurement are often based on inadequate information. For example, although a maximum likelihood merge of two measurements is statistically valid, the covariance matrix only partially describes the quality of the measurement. In fact, a measurement with relatively large covariance may be of substantially lower quality than a measurement with relatively small covariance when other factors, such as distortion of the signal peak and surface orientation, are taken into account. A more comprehensive approach to applying measurement quality to manipulating measurements is required.

Finally, nonreturn measurements are generally treated as having no qualitative value, thus are often ignored during data collection. This means that information about regions of the environment that cannot be scanned is lost. Future research should examine what can be learned about the environment being scanned from the absence of a return signal.

9.

Conclusions

Quality metrics have featured significantly in contemporary research; however, most quality metrics have been designed for specific application or specific algorithms, and are often used independently. Measurement uncertainty has been used extensively to represent measurement quality, but many environmental factors affect measurement uncertainty, making it insufficient as an independent quality metric.

The relationship of range and resolution to measurement quality depends on the beam width. Additional work is required to better define the relationship between measurement quality and resolution for midfield measurements, where parallax must be taken into account. Sampling density has also been featured in various forms as a quality metric, although most approaches are highly application specific. Absent from the literature is a more detailed analysis of how sampling density is related to measurement quality and how to quantify sampling density as a quality metric in a generalized fashion. Surface orientation has also been used extensively as a quality metric, although it is also insufficient as an independent quality metric. Reflectivity is affected by surface materials, orientation, and surface complexity; thus, this factor has been used to represent measurement quality. Given, however, that reflectivity is affected by multiple factors, it, too, is insufficient as an independent quality metric.

The current state of the art in quality metrics performs adequately in assessing the quality of measurements within the context of specific applications, but are often not readily generalizable. Few researchers combine quality metrics so that the strengths of one may offset the weakness of the other. This paper was a first step in assessing the relationship among the various quality metrics currently in use. More work is needed to develop a more comprehensive approach to measurement quality assessment.