1.1.

Near-Infrared Spectroscopy Validation

The regional tissue oxygenation, ${\mathrm{rStO}}_2$, constitutes in theory a weighted average of arterial, capillary, and venous blood saturation. ${\mathrm{rStO}}_2$ is determined by the arterial oxygen saturation (${\mathrm{SaO}}_2$), the volumes of the arterial $V_a$ and venous $V_v$ blood, the CBF, the cerebral metabolic rate (${\mathrm{CMRO}}_2$), and the oxygen capacity of the blood defined by the hemoglobin concentration (HB), and the oxygen-binding capacity of HB ($k$) (1.306 mL of oxygen/g of HB):⁵

Theoretically, a greater venous blood volume ($V_v$) compared with arterial blood volume ($V_a$) means that the ${\mathrm{rStO}}_2$ will be closer correlated to venous blood samples from the jugular bulb than to the hemoglobin oxygen saturation of arterial blood. As this volume ratio is not known—and may vary among tissues and over time—it follows that no perfect reference standard exists. In validation studies, the “raw” ${\mathrm{rStO}}_2$ estimate—or a reference regional cerebral tissue oxygenation (${\mathrm{refStO}}_2/\mathrm{ref}\_{\mathrm{rStO}}_2$) is calculated assuming a fixed ratio of venous to arterial blood—is compared with jugular venous saturation (${\mathrm{SjvO}}_2$) or superior vena cava saturation (${\mathrm{ScvO}}_2$).

1.2.

Near-Infrared Spectroscopy Instrumentation

It is well established that the estimate of the regional tissue oxygenation, ${\mathrm{rStO}}_2$, is dependent on the instrumentation, e.g., differences in light source and detector distances of the sensors and different mathematical algorithms. Different commercial instruments show systematic differences in absolute values.^6–8 It is, thus, uncertain how results with a specific NIRS oximeter can be compared or combined with results obtained by other instruments. Furthermore, the repeatability of the measurements has been found to be about 5%, meaning that repeated measurements could be about 14% points apart.^7,9,10

The combination of lack of validation and poor repeatability is an important issue when designing an RCT of implementing cerebral NIRS oximetry into clinical care. The questions on what ${\mathrm{rStO}}_2$ target range to aim for, what instrument to use, and what interventions to use when ${\mathrm{rStO}}_2$ is out-of-range all need to be addressed. It is problematic if these issues are not addressed when implementing NIRS oximetry outside the rigorous settings of an RCT and, thus, the door is open for introduction of yet another intervention into standard clinical care that is poorly backed by higher level of evidence.

In the present review, we summarize the available in vivo validation of cerebral NIRS oximetry to inform future in vivo validation strategies, in particular to establish a way forward in the assessment of NIRS instrumentation for future randomized trials.

2.1.

Inclusion Criteria

2.2.

Search Strategy

The initial search on PubMed was performed combining two category searches.

The NIRS category combined the results from MeSH term “near-infrared spectroscopy” and the following keywords: “infrared spectroscopy,” “NIRS,” “near-infrared spectroscopy,” “cerebral venous oxygenation,” ”near-infrared spectroscopic cerebral oxygenation,” “tissue oxygenation index,” “cerebral oximetry,” “cerebral oximeter,” and “near-infrared cerebral oximetry.” The venous blood sample category combined following keywords: “superior vena cava oxygen saturation,” “jugular venous oxygen saturation,” “central venous oxygen saturation,” “oxygenation saturation,” “venous jugular bulb saturation,” “mixed venous saturation,” “jugular bulb oximetry.”

2.3.

Subgroups

Subgroups were defined by (1) patient group (adult or child), (2) use of cerebral NIRS device, and (3) site of venous blood sample collection or location of the continuous venous saturation catheter.

Blood samples were divided into three groups according to: site of sample collection or use of a fixed arteriovenous ratio: (1) internal jugular vein and jugular bulb (${\mathrm{SjOV}}_2$), (2) superior vena cava (${\mathrm{ScvO}}_2$), and (3) reference regional cerebral tissue oxygenation ($\mathrm{ref}\_{\mathrm{rStO}}_2$).

2.4.

Data Extraction

The following information was, where possible, extracted from the included studies:

The MATLAB^® (MathWorks, Inc., Massachusetts) script Grabit ( http://www.mathworks.com/matlabcentral/fileexchange/7173-grabit) allows data extraction from an image file. In studies without Bland–Altman plots, we extracted data from scatter plots enabling estimation of MD, SD, and LOA.

2.5.

Statistical Analysis

To assess the agreement between ${\mathrm{rStO}}_2$ estimated by NIRS and by blood samples, weighted averages of the MDs and SDs from each study were calculated:

The bias between ${\mathrm{rStO}}_2$ estimated by NIRS and by blood samples was also assessed by the root mean square error, $A_{\mathrm{RMS}}$.

Furthermore, a mixed effects model for each instrument with blood sample saturation as independent, NIRS ${\mathrm{rStO}}_2$ as dependent, and study as random factor was created. The 95% confidence intervals of the regression coefficients were calculated by bootstrapping.

All statistical tests were done in R [R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria].

3.1.

Literature Search

The result of the initial literature search was 698 papers. The records were screened and 262 abstracts were assessed to select studies fulfilling our inclusion criteria. We identified 41 in vivo validation studies of NIRS and additional 15 studies were identified by manual search of cited references, resulting in 25 pediatric ($\text{age}<18\text{years}$) and 31 adult ($\text{age}>18\text{years}$) studies published from 1996 to 2015.

Twenty-two pediatric studies were analyzed after exclusion of three articles. These were excluded, because either a prototype NIRS device was used^11,12 or the NIRS device was not described.¹³ Furthermore, two studies^14,15 were not included in the calculation of weighted mean deviation (MD) and SD, because the type of INVOS (COVIDIEN, Mansfield, Massachusetts) device could not be determined. 28 adult studies were analyzed after the exclusion of three studies. Two studies were excluded because a prototype NIRS device was used¹⁶ and one was excluded because it was impossible to extract the data points.¹⁷

We found studies of INVOS (Medtronic, Minneapolis, Minnesota), NIRO (Hamamatsu Photonics, Hamamatsu city, Japan), EQUANOX Cerebral Oximetry System (Nonin Medical, Inc., Plymouth, Minnesota), FORESIGHT Cerebral Oximeters (CAS Medical Systems, Branford, Connecticut), and TRS-20 (Hamamatsu Photonics, Hamamatsu city, Japan).

3.2.

Pediatric Studies

The 22 studies included in total 776 children (range 5 to 100). The overall mean ${\mathrm{ScvO}}_2$ was 67.7% (SD 16.1%), ${\mathrm{SjvO}}_2$ 70.4% (SD 13.0%), and ${\mathrm{refStO}}_2$ 75.0% (SD 8.9%). The overall MD was $-1.2\%$ with LOA of $-12.5\%$ to 10.1% between NIRS ${\mathrm{rStO}}_2$ and ${\mathrm{refStO}}_2$, $-1.9\%$ with LOA $-17.8\%$ to 14.0%, and $-2.0\%$ with LOA of $-21.8\%$ to 17.6% between NIRS ${\mathrm{rStO}}_2$ and ${\mathrm{SjvO}}_2$ and ${\mathrm{ScvO}}_2$, respectively (Fig. 1). The regression coefficients of the regression lines between ${\mathrm{ScvO}}_2$ (Fig. 2), ${\mathrm{SjvO}}_2$ (Fig. 3), and ${\mathrm{refStO}}_2$ (Fig. 4) against ${\mathrm{rStO}}_{2(\mathrm{NIRS})}$ are shown in Table 1.

Fig. 1

Pediatric studies; MD and associated LOA between ${\mathrm{refStO}}_2$, ${\mathrm{SjvO}}_2$, ${\mathrm{ScvO}}_2$, and NIRS ${\mathrm{rStO}}_2$ within the different groups. Weighted MD and LOA are also shown.

Fig. 2

NIRS ${\mathrm{rStO}}_2$ versus central venous saturation (${\mathrm{ScvO}}_2$) in children. The horizontal line represents the overall mean ${\mathrm{rStO}}_2$; the vertical line the mean ${\mathrm{refStO}}_2$.

Fig. 3

NIRS ${\mathrm{rStO}}_2$ versus jugular venous saturation (${\mathrm{SjvO}}_2$) in children. The horizontal line represents the overall mean ${\mathrm{rStO}}_2$; the vertical line represents the mean ${\mathrm{refStO}}_2$.

Fig. 4

NIRS ${\mathrm{rStO}}_2$ versus reference saturation (${\mathrm{refStO}}_2$) in children. The horizontal line represents the overall mean ${\mathrm{rStO}}_2$; the vertical line represents the mean refStO.

Table 1

Pediatric studies, regression between ScvO2, SjvO2, refStO2, and rstO2.

3.3.

Adult Studies

The 28 studies included in total 604 participants (range 9 to 50). The overall mean ${\mathrm{ScvO}}_2$ was 77.4% (SD 10.5%), ${\mathrm{SjvO}}_2$ 68.1% (SD 15.3%), and ${\mathrm{refStO}}_2$ 66.9% (SD 11.5%). NIRS ${\mathrm{rStO}}_2$ and ${\mathrm{refStO}}_2$ had an overall MD of 0.2% and LOA of $-13.0\%$ to 13.4%. In comparison, the MD between the NIRS ${\mathrm{rStO}}_2$ and ${\mathrm{SjvO}}_2$ and ${\mathrm{ScvO}}_2$ was $-0.5\%$ with LOA $-20.7\%$ to 19.7% and $-8.8\%$ with LOA of $-24.1\%$ to 6.5%, respectively (Fig. 5). The regression coefficients of the regression lines between ${\mathrm{ScvO}}_2$ (Fig. 6), ${\mathrm{SjvO}}_2$ (Fig. 7), and $\mathrm{ref}\_{\mathrm{StO}}_2$ (Fig. 8) against ${\mathrm{rStO}}_{2(\mathrm{NIRS})}$ are shown in Table 2.

Fig. 5

Adult studies; MD and associated LOA between ${\mathrm{refStO}}_2$, ${\mathrm{SjvO}}_2$, ${\mathrm{ScvO}}_2$, and NIRS ${\mathrm{rStO}}_2$ within the different groups. Weighted MD and LOA are also shown.

Fig. 6

NIRS ${\mathrm{rStO}}_2$ versus central venous saturation ${\mathrm{ScvO}}_2$ in adults. The horizontal line represents the overall mean ${\mathrm{rStO}}_2$; the vertical line represents the mean ${\mathrm{ScvO}}_2$.

Fig. 7

NIRS ${\mathrm{rStO}}_2$ versus jugular venous saturation (${\mathrm{SjvO}}_2$) in adults. The horizontal line represents the overall mean ${\mathrm{rStO}}_2$; the vertical line represents the mean ${\mathrm{SjvO}}_2$.

Fig. 8

NIRS ${\mathrm{rStO}}_2$ versus reference saturation (${\mathrm{refStO}}_2$) in adults. The horizontal line represents the overall mean ${\mathrm{rStO}}_2$; the vertical line represents the mean ${\mathrm{refStO}}_2$.

Table 2

Adult studies, regression between ScvO2, SjvO2, refStO2, and rStO2.

All included studies are presented in subgroups in Table 3. The table shows the number of subjects in each study and, if available, the number of datapoints. Furthermore, MD, precision, and $A_{\mathrm{RMS}}$ are shown for each study. The consistency between original results and results obtained by Grabit was assessed by calculating MD of the differences between original MD’s and those obtained by Grabit. We found a MD of 0.09% (SD 0.34%).

Table 3

All included studies divided into subgroups.

4.1.

Principal Findings

We found that although the NIRS ${\mathrm{rStO}}_2$ most closely estimates a weighted average of arterial and venous blood (${\mathrm{refStO}}_2/\mathrm{ref}\_{\mathrm{rStO}}_2$) in both adults and children, the agreement is poor with LOA of more than $+/-13\%$ in both populations. Moreover, we found that irrespective of whether venous blood was drawn from the central circulation or from the jugular bulb, or the theoretical arterial component was accounted for, NIRS ${\mathrm{rStO}}_2$ was increasingly overestimating blood saturation in the low ranges, i.e., NIRS ${\mathrm{rStO}}_2$ has oxygen-level dependent bias. We found no clear systematic differences between the different instruments.

4.2.

Appraisal of Our Data Analysis Methods

We believe that this is the most exhaustive review of the literature on NIRS validation yet.

We utilized a MATLAB script “Grabit” enabling us to incorporate data from almost all published work. The use of Grabit, however, has limitations. The quality of scaling and calibration of the axes depend on how clearly the values of the axes in the original linear plot are specified, and extracting data from a scatter plot with many and clustered data points may introduce errors. Data points with the greatest distance from the linear regression line are typically the easiest to discriminate, whereas clustered data points may be overlooked. This may affect the Bland–Altman analysis. To assess this possible problem, we compared the original MD with the MD calculated with data retrieved with Grabit and found good consistency with an overall difference of 0.09% (SD 0.34%). Another problem when using data of this nature is that several data points may be from the same study participant. This is very rarely indicated in the plot and therefore not accounted for in the statistical analysis.

When repeated measurements are ignored, the estimated variance will be biased toward zero. We, therefore, chose to perform simple regressions ignoring the potential errors on the measurement of blood samples and to illustrate the regressions with conventional $X-Y$ plots, although overall, these weaknesses make it unreasonable to attempt any formal statistical comparison among instruments. The review has revealed that the statistical analyses in many studies are less than perfect. There is, in our opinion, not a single clearly superior statistical method, and the most important issue might be that the study design is appropriate, i.e., data that enable estimation of the precision of each method and any level-dependent bias. The Bland–Altman analysis is appealing due to its simplicity, but often differences in precision between methods and level-dependent bias are neglected.

4.3.

Validation with Blood Samples

We found only minor MDs between the NIRS ${\mathrm{rStO}}_2$ and the reference methods (Figs. 1 and 5), which was surprising as the reference that takes the arterial blood contribution into account must by nature be higher than the simple venous references.

Furthermore, it is obvious from the wide LOA and by looking at the scatterplots, that the precision of NIRS ${\mathrm{rStO}}_2$ in estimating venous blood saturation is poor. It is also clear that NIRS ${\mathrm{rStO}}_2$, irrespective of instrument, has an oxygenation level-dependent bias, i.e., the difference between ${\mathrm{rStO}}_2$ and venous saturation changes with de- and increasing average of saturation.

When the agreement between methods varies with the level of the measured, the difference between the methods should be regressed on the average of the two methods. LOA is then calculated from the residual SD from the regression.⁶⁵ Failing to do so will result in the LOA being too wide. Unfortunately, the appropriate statistical analysis has seldom been applied even in the studies with very obvious level-dependent bias.^{20,28,31,33,38,44,46,52,59}

The poor precision of the NIRS estimates is strikingly consistent throughout the studies. An explanation could be that NIRS is an estimate of the local tissue oxygenation, whereas the blood samples represent regional oxygenation. It has been suggested that jugular bulb oximetry is in fact insensitive to regional ischemia due to “flow-weighting”;⁶⁶ however, none of the included studies deal with procedures or patients that are in particular risk of local ischemia.

Another explanation could be that systematic differences between different regions of the brain would translate into variable agreement depending on a sensor position. However, there are no data suggesting that different regions of the brain are systematically oxygenated differently.^67,68 Sørensen et al.⁶⁹ examined the effect of skin oxygen saturation and found that whole body heating and administration of norepinephrine affected ${\mathrm{rStO}}_2$, whereas ${\mathrm{SjvO}}_2$ remained unchanged. This suggests that local fluctuations in skin saturation may contribute to variations in the ${\mathrm{SjO}}_2$ ${\mathrm{rStO}}_2$ agreement. In that regard, it is also likely that skin pigmentation influences the NIRS signal as has been well documented being the case with both pulse oximetry^70,71 and NIRS.⁸ Unfortunately, the included studies provide very little information on individual skin pigmentation.

Theoretically, it would thus be expected that the pediatric studies would reveal better accuracy and precision as the skin is thinner and the sensor-to-cortex distance is shorter. Theoretical modeling and clinical data show that the average depth of penetration of NIR light is directly proportional to the separation of the emitting and receiving optodes.⁷² However, the signal will decrease as the light travels longer in. The separation is the same on the pediatric/neonatal and adult sensors on INVOS and NIRO, whereas NONIN and FORE-SIGHT have shorter distance on the pediatric/neonatal sensors. In both the pediatric and the adult studies, the agreement between NIRS and refStO2 was best with MD 1.2% (LOA of $-12.5\%$ to 10.1%) in children and 0.2% (LOA $-13.0\%$ to 13.4%) in adults, but the results do not support the notion that NIRS is better suited for the pediatric population as the LOA are quite similar throughout (Figs. 1 and 5).

As pointed out by Bland and Altman,⁶⁵ lack of repeatability can interfere with the comparison of two methods. If one method has poor repeatability, the agreement between the two methods is bound to be poor. The repeatability of different NIRS monitors has been examined. Hyttel-Sorensen et al.¹⁰ found repeatability ranging from 2.0% to 5.4% (within subject SD) with various instruments, whereas in another study comparing INVOS 5100, NONIN-EQUANOX 8004CA, and FORESIGHT, $S_w$ was 2.9, 4.9, and 2.0, respectively.⁷ Hessel et al.⁷³ found a $S_w$ of 4.8 with INVOS 5100 and 2.8 with FORESIGHT. This level of repeatability may be the single most important factor in the poor ability of NIRS to estimate the blood sample saturations.

There are several drawbacks when using blood samples to validate NIRS. First, the reflected NIRS signal is influenced by both the venous, capillary, and arterial blood. It is uncertain how much each vascular bed contributes to the overall light attenuation, i.e., the volume ratio could be different between subjects and within subjects over time. Ito et al. showed that with changes in ${\mathrm{PaCO}}_2$ (partial pressure of carbon dioxide in arterial blood), the arterial blood volume followed changes in cerebral blood volume. The same was not observed in the case of the capillary or venous blood volume (29). It is notable that the commercial instruments assume different A:V (arteriovenous) ratios and likely are calibrated against different references. Second, the contribution of extra-cerebral tissue to both the NIRS signal and blood samples is likely not similar.²⁰ Third, it has been shown that the rate of withdrawal influences the jugular venous saturation, resulting in higher saturation with faster withdrawal.⁷⁴ Use of continuous saturation monitors might be a potential solution to decrease such error, but when looking at studies using in-line saturation monitors,^14,25–27 greater negative bias is found.

Fourth, it is not really possible to draw blood from the jugular bulb or superior vena cava in preterm infants. Our search identified two studies that used a neonatal sensor, but neither of the studies included neonates. This reflects a major challenge because the neonatal population is in many regards the most likely to benefit from cerebral NIRS monitoring as no other modality has been approved to provide continuous estimation of blood flow in neonates. Furthermore, the short distance from skin to cerebral cortex is ideal for NIRS measurements. It is thus not surprising that NIRS is increasingly used in the neonatal intensive care, but validation in this population must be done by other methods than what used in older patient groups.

4.4.

Future Near-Infrared Spectroscopy Validation

It is remarkable that despite this rather large dataset, we are unable to draw firm conclusions. A group under International Organization for Standardization is currently working on a guideline for NIRS validation. Looking at the heterogeneity of the study results presented here, it would be a step forward to have some common framework for future validation. From our review, it would seem appropriate to recommend a statistical analysis. It would make comparison of study results more straightforward. It could also guide the design of the studies. Many studies lack estimation of precision/repeatability and future studies should aim at deciphering the different levels of variation in the NIRS versus blood sample relationship.

A different approach is using a liquid blood lipid phantom.^75–77 This is inspired by the work of Suzuki et al.⁷⁸ that validated the NIRO system in a blood lipid phantom with oxygen consuming yeast to drive the oxygenation down. This model has the advantage that the oxygenation can be changed from 100% to close to zero. Moreover with a liquid phantom, it is possible to establish a reference saturation by either co-oximetry,⁷⁸ measurements of ${\mathrm{pO}}_2$ (partial pressure of oxygen), ${\mathrm{pCO}}_2$ (partial pressure of carbon dioxide), pH and calculate an estimate of saturation, or other optical methods.^75,77 It is possible to create different combinations of scattering and absorption properties by altering the content of lipid and hemoglobin. Lastly, a layered design can be implemented by having the sensors placed on solid windows with known optical properties. The method allows validation of the neonatal sensors, which cannot be tested by blood sampling, but many aspects still need to be resolved for it to be more broadly implemented. First of all, the optical properties must be defined and some common guideline for the construction of the phantom for each relevant population (neonatal, pediatric, and adult) should be made. It also needs more robust comparison with clinical NIRS testing before it can be an alternative to the traditional validation by blood sampling.