1.

Introduction

The idea behind the current proposal for a data fusion system derives from the evidence of some specific problems we encountered during our trials with radar and infrared search and track (IRST) for surveillance and tracking. We can summarize and represent what was highlighted during the tests in the following simple and concise way:

This paper describes a simple but effective method to remove the above limitations in the use of the two systems and to operate without the need of any kinematic ranging operation.

Currently, radar and IRST are operating together on various platforms and are considered complementary from an operational point of view. In general, radar systems are effective in long range detection, they are capable of providing precise target range and they are active systems; IRST systems are very precise in angular target tracking and they are completely passive systems.

The proposed data fusion approach intends to exploit the capabilities of the two sensors by mixing their specific characteristics, considering them as part of an integrated system.

2.

Definition of the Context for Data Fusion

We assume at beginning that both the sensors, radar and IRST, provide detection data relevant to the target at the same sampling time $T_s$ (synchronous systems). Later on we will remove this limitation to deal with systems with different sampling time also (asynchronous systems).

We will disregard all the issues related to the registration problem^4,5 of the two sensors assuming that they have a common axis system that, in general, is the one of the host platform. Figure 1 is a top level block diagram of the data fusion system.

Fig. 1

Top level block diagram of the system.

2.2.

Data Processing and Fusion System

This part is devoted to the tracks from noise or clutter. It is based on time-by-time correlation of the detection. Data processors (DAP) perform estimation and prediction of tracks on the basis of kinematics. The fusion system, placed at the end of the DAPs, mixes the tracks data. Here we assume the DAPs and the fusion system are located in the same functional unit even if it is not strictly necessary.

The data fusion system functional block will be expanded later on.

For our purpose we assume that, for each detection, the following data, in input to the radar data processing functional block, are available:

$$\text{range}r(k·T_s)$$

and

$$\text{bearing}\beta (k·T_s).$$

For the IRST, for each detection, the following data, in input to the IRST data processing functional block, are available: azimuth

$$\text{azimuth}\phi (k·T_S)$$

and

$$\text{elevation}\vartheta (k·T_S).$$

The angles $\beta$ and $\varphi$ derive from different sources, but they refer to the same angle although with different measurement noise.

For compactness, the sampling time $T_s$ is not indicated in the following formulas and whenever necessary we will omit the sampling counter $k$, too.

The geometry associated to the above parameters is schematically presented in Fig. 2.

Fig. 2

Geometry of the scene.

The noise associated with the input data is assumed white, Gaussian, and uncorrelated.

We convert the data of radar from polar to Cartesian coordinates to avoid the use of the extended Kalman filters,

$$\{\begin{array}{l}x(k)=r(k)·\cos [\beta (k)]\\ y(k)=r(k)·\mathrm{sen}[\beta (k)]\end{array}.$$

It is known that the above conversion produces correlation and bias in the noise of the converted coordinates, but that bias can be evaluated and therefore removed.⁶

Now we can proceed in defining the output elements, the tracks, of the radar data processor (by the usual symbols for the description of dynamic systems):

• • estimation of the present state: $[{\widehat{x}}_R(k|k),{\widehat{y}}_R(k|k)]$

• • estimation of the future state (prediction): $[{\widehat{x}}_R(k+1|k),{\widehat{y}}_R(k+1|k)]$

• • state errors estimated variances ${\sigma }_x^2$ and ${\sigma }_y^2$

• • estimated correlation between the state errors: ${\rho }_{xy}=E\{\mathrm{\Delta }{x}_R·\mathrm{\Delta }{y}_R\}$, where $\mathrm{\Delta }{x}_R={\widehat{x}}_R-x$ and $\mathrm{\Delta }{y}_R={\widehat{y}}_R-y$ are the estimation errors on ${\widehat{x}}_R$ and ${\widehat{y}}_R$, being $x$ and $y$ the true values of the slant coordinates. $E\{\}$ stands for the expected value.

The output elements, the tracks, of the IRST data processing function are the following:

• • estimation of the present state: $\lfloor \widehat{\phi }(k|k),\widehat{\vartheta }(k|k)\rfloor$

• • prediction of the future state: $\lfloor \widehat{\phi }(k+1|k),\widehat{\vartheta }(k+1|k)\rfloor$

• • estimated variances ${\sigma }_{\phi }^2$, ${\sigma }_{\vartheta }^2$.

From the estimates ${\widehat{x}}_R$ and ${\widehat{y}}_R$ we can get back the range and bearing filtered data:

Eq. (1)

$$\{\begin{array}{l}\widehat{r}=\sqrt{{\widehat{x}}_R^2+{\widehat{y}}_R^2}\\ \widehat{\beta }=\mathrm{arctg}\left(\frac{{\widehat{y}}_R}{{\widehat{x}}_R}\right)\end{array}.$$

Since the estimation errors on ${\widehat{x}}_R$ and ${\widehat{y}}_R$ are generally different at every sampling time and are correlated, too, the computation of the variance is not straightforward and some approximation must be taken to obtain (see Appendix A):

Eq. (2)

$$\{\begin{array}{l}{\sigma }_r^2=e^{-{\sigma }_{\beta }^2}[{\sigma }_x^2·(\cos h{\sigma }_{\beta }^2·{\cos }^2\widehat{\beta }+\sin h{\sigma }_{\beta }^2·{\sin }^2\widehat{\beta })\\ +{\sigma }_y^2·(\cos h{\sigma }_{\beta }^2·{\sin }^2\widehat{\beta }+\sin h{\sigma }_{\beta }^2·{\cos }^2\widehat{\beta })+2·{\rho }_{xy}·\sin \widehat{\beta }·\cos \widehat{\beta }]\\ {\sigma }_{\beta }^2=\frac{{\widehat{y}}_R^2·{\sigma }_x^2+{\widehat{x}}_R^2·{\sigma }_y^2-2·{\widehat{x}}_R·{\widehat{y}}_R·{\rho }_{xy}}{{\widehat{r}}^4}+2·\frac{{\sigma }_x^2·{\sigma }_y^2-{\rho }_{xy}^2}{{\widehat{r}}^4}\end{array}.$$

Now, to implement the fusion of the data coming from two different sensors, we have to perform an additional step thus obtaining a homogeneous set of measurements.

By using elevation $\vartheta$ from the IRST and the data from the radar, we can write the equations:

Eq. (3)

$$\{\begin{array}{l}{\zeta }_{x_R}={\widehat{x}}_R·\cos \widehat{\vartheta }=\widehat{r}·\cos \widehat{\beta }·\cos \widehat{\vartheta }\\ {\zeta }_{y_R}={\widehat{y}}_R·\cos \widehat{\vartheta }=\widehat{r}·\sin \widehat{\beta }·\cos \widehat{\vartheta }\\ {\zeta }_z=\widehat{r}·\mathrm{sen}\widehat{\vartheta }\end{array}.$$

While by using azimuth $\varphi$ elevation $\vartheta$ from the IRST and range $r$ from the radar we have:

Eq. (4)

$$\{\begin{array}{l}{\zeta }_{x_I}=\widehat{r}·\cos \widehat{\phi }·\cos \widehat{\vartheta }\\ {\zeta }_{y_I}=\widehat{r}·\sin \widehat{\phi }·\cos \widehat{\vartheta }\\ {\zeta }_z=\widehat{r}·\mathrm{sen}\widehat{\vartheta }\end{array}.$$

Now we have two homogeneous sets of data: the vectors ${\mathit{\zeta }}_R={[{\zeta }_{x_R}{\zeta }_{y_R}{\zeta }_z]}^T$ and ${\mathit{\zeta }}_I={[{\zeta }_{x_I}{\zeta }_{y_I}{\zeta }_z]}^T$, with $T$ as the symbol of transpose operation, where of course ${{\mathit{\zeta }}_Z}_R={\zeta }_{z_I}={\zeta }_z$. The expressions for the most important statistic figures are reported in Appendix B. In particular it is shown that the expected values of the measurement errors have a bias. When the bias is not negligible, a suitable mitigation process has to be considered.⁶

In a multiple target tracking scenario we have to associate the data from many possible targets. Simply the $r^{\prime }$ track coming from the radar and the $i^{\prime }$ track from the IRST are considered to be linked to the same object if, said $X_R$ and $X_I$ their respective states,

Eq. (5)

$$X_R(t)=X_I(t)\forall t$$

being $t$ the time. In our case, ${\mathit{\zeta }}_R=X_R$ and ${\mathit{\zeta }}_I=X_I$. But ${\mathit{\zeta }}_R$ and ${\mathit{\zeta }}_I$ are affected by estimation errors that rarely allow Eq. (5) to be verified. In order to accept the hypothesis that two tracks coming from different sensors concern the same object (track to track association process), we have, therefore, to test that the following differences

$$\{\begin{array}{l}\mathrm{\Delta }{x}_{\mathrm{RI}}={\zeta }_{x_R}-{\zeta }_{x_I}\\ \mathrm{\Delta }{y}_{\mathrm{RI}}={\zeta }_{y_R}-{\zeta }_{y_I}\end{array}$$

are inside a given bound for every $k$. So to properly associate ${\mathit{\zeta }}_R$ and ${\mathit{\zeta }}_I$ to the same target we proceed with the following criteria:

Condition 1:

For all tracks compute

Eq. (6)

$$|\widehat{\beta }-\widehat{\phi }|<\lambda ·({\sigma }_{\beta }+{\sigma }_{\phi }),$$

where $\lambda$ is a suitable constant. Equation (6) is justified by considering that the error of $\widehat{\beta }$ and $\widehat{\phi }$ are independent and Gaussian with 0 mean.

Condition 2:

For all tracks satisfying the previous criteria compute:

Eq. (7)

$$\{\begin{array}{l}{\zeta }_{x_R}(k)={\widehat{x}}_R(k|k)·\cos \widehat{\vartheta }(k|k)\\ {\zeta }_{y_R}(k)={\widehat{y}}_R(k|k)·\cos \widehat{\vartheta }(k|k)\\ {\zeta }_{x_I}(k)=\widehat{r}(k|k)·\cos \widehat{\vartheta }(k|k)·\cos \widehat{\phi }(k|k)\\ {\zeta }_{y_I}(k)=\widehat{r}(k|k)·\cos \widehat{\vartheta }(k|k)·\sin \widehat{\phi }(k|k)\\ {\zeta }_z(k)=\widehat{r}(k|k)·\sin \widehat{\vartheta }(k|k)\end{array}.$$

Then, under the Gaussian approximation on ${\zeta }_R$ and ${\zeta }_I$, verify that the following conditions are satisfied:

Eq. (8)

$$\frac{{[{\zeta }_{x_R}(k)-{\zeta }_{x_I}(k)]}^2}{{\sigma }_{x_{RI}}^2(k|k)}+\frac{{[{\zeta }_{y_R}(k)-{\zeta }_{y_I}(k)]}^2}{{\sigma }_{y_{RI}}^2(k|k)}-2·\frac{{\rho }_{\mathrm{\Delta }xy}}{{\sigma }_{x_{RI}}(k|k)·{\sigma }_{y_{RI}}(k|k)}·[{\zeta }_{x_R}(k)-{\zeta }_{x_I}(k)]·[{\zeta }_{y_R}(k)-{\zeta }_{y_I}(k)]<{\gamma }_0$$

being ${\sigma }_{x_{RI}}^2$, ${\sigma }_{y_{RI}}^2$ the variances and ${\rho }_{\mathrm{\Delta }xy}$ the correlation coefficient computed in Appendix C. With the approximation that ${\zeta }_R$ and ${\zeta }_I$ are Gaussian, ${\gamma }_0$ is a suitable threshold value evaluated by using techniques based on ${\chi }^2$ test.

Condition 3:

For all tracks satisfying the previous criteria compute:

Eq. (9)

$$\{\begin{array}{l}{\zeta }_{x_R}(k+1)={\widehat{x}}_R(k+1|k)·\cos \widehat{\vartheta }(k+1|k)\\ {\zeta }_{y_R}(k+1)={\widehat{y}}_R(k+1|k)·\cos \widehat{\vartheta }(k+1|k)\\ {\zeta }_{x_I}(k+1)=\widehat{r}(k+1|k)·\cos \widehat{\vartheta }(k+1|k)·\cos \widehat{\phi }(k+1|k)\\ {\zeta }_{y_I}(k+1)=\widehat{r}(k+1|k)·\cos \widehat{\vartheta }(k+1|k)·\sin \widehat{\phi }(k+1|k)\\ {\zeta }_z(k+1)=\widehat{r}(k+1|k)·\sin \widehat{\vartheta }(k+1|k)\end{array}.$$

Then verify that the following conditions are satisfied:

Eq. (10)

$$\frac{{[{\zeta }_{x_R}(k+1)-{\zeta }_{x_I}(k+1)]}^2}{{\sigma }_{x_{RI}}^2(k+1|k)}+\frac{{[{\zeta }_{y_R}(k+1)-{\zeta }_{y_I}(k+1)]}^2}{{\sigma }_{y_{RI}}^2(k+1|k)}-2·{\rho }_{\mathrm{\Delta }xy}(k+1|k)·\frac{[{\zeta }_{x_R}(k+1)-{\zeta }_{x_I}(k+1)]·[{\zeta }_{y_R}(k+1)-{\zeta }_{y_I}(k+1)]}{{\sigma }_{x_{RI}}(k+1|k)·{\sigma }_{y_{RI}}(k+1|k)}<{\gamma }_1,$$

where ${\sigma }_{x_{RI}}^2(k+1|k)$, ${\sigma }_{y_{RI}}^2(k+1|k)$, and ${\rho }_{\mathrm{\Delta }xy}(k+1|k)$ can be computed as in the previous step, but using the predicted data, and ${\gamma }_1$ is evaluated in the same way as ${\gamma }_0$.

We define “mixed detection” the system of Eq. (7) obtained by combining the data of two tracks which pass the above discrimination process. Equation (7) can be interpreted as a set of measures, whose measurement errors are ${\sigma }_r^2$, ${\sigma }_{\beta }^2$, ${\sigma }_{x_R}^2$, ${\sigma }_{y_R}^2$,${\sigma }_{x_I}^2$, ${\sigma }_{y_I}^2$, and ${\sigma }_z^2$ evaluated in the Appendices.

3.

Fusion and Tracking

${\zeta }_{x_R}(k)$ and ${\zeta }_{x_I}(k)$ both represent an estimation of the same quantity, as well as ${\zeta }_{y_R}(k)$ and ${\zeta }_{y_I}(k)$. Now, the classical way to proceed for the fusion is by the following equation:⁷

Let $\mathrm{\Delta }{\mathit{\zeta }}_{RI}$, $\mathrm{\Delta }{\mathit{\zeta }}_I$, and $\mathrm{\Delta }{\mathit{\zeta }}_R$ be the errors on ${\widehat{\mathit{\zeta }}}_{RI}(k),{\mathit{\zeta }}_I(k),{\mathit{\zeta }}_R(k)$, we have:

We consider $({\mathit{\zeta }}_R,{\mathit{\zeta }}_I)$ as two different detections which have been determined by the same sensor from the same source. Now we designate

In that case⁸

${\zeta }_{\alpha }(k)$ is the input to the processor devoted to the fused tracks and $R(k)$ is the covariance matrix of its error.

All the mixed detections meeting the criteria

Fig. 3

Data fusion system diagram.

4.

Asynchronous Systems

Now we remove the limitation that radar and IRST are synchronous. Our main objective is to align all data in time in order to implement the track-to-track association process conditions 1 to 3 described in Sec. 2.

Let us designate $T_R$ and $T_I$ the sampling periods of radar and IRST, respectively. We select for the following description the sampling period $T_R$ of radar as reference and as the time when all the operations of the fusion system start.

Let

Fig. 4

Timing diagram.

The output data from the IRST referred to the radar time $t_R=k·T_I+\mathrm{\Delta }t$ are given by

The Eq. (23) are necessary to verify conditions 1 and 2. Once obtained the predicted state ${\widehat{\xi }}_R(t_I+T_I)$ and the predicted state error covariance matrix $P_R(t_I+T_I)$ we are ready to run the condition 3.

An alternative method, called interpolation between samples, certainly faster, but mainly, which requires no knowledge of the state equations of both the tracking systems, is as follows: since it is always $\mathrm{\Delta }t\le T_I$, assuming a three-dimensional state, it is reasonable to suppose a constant acceleration of the IRST track within the time interval $\mathrm{\Delta }t$. Furthermore, in the absence of further information between two successive sampling times, we obtain the variances in an intermediate point as a linear interpolation between the variances of the estimated and predicted state errors. So we rewrite Eq. (23) to align the azimuth and elevation angles $\phi (t_I)$, $\vartheta (t_I)$ and the state errors variances ${\sigma }_I^2={[\begin{array}{cc}{\sigma }_{\phi }^2& {\sigma }_{\vartheta }^2\end{array}]}^T$ to the time $t_R$ as

For condition 3 we will proceed in a simpler way using the linear interpolation for both the positions ${\widehat{x}}_R$ and ${\widehat{y}}_R$ and the variance ${\sigma }_R^2={[{\sigma }_x^2{\sigma }_y^2]}^T$ as

5.

Simulation Results

We generated the target motions in order to stress the system and to check the robustness of algorithms also incurring in the risk to create pour realistic scenarios. Clutter is generated and uniformly distributed inside the observed volume. Both IRST and radar utilize the probabilistic data association (PDA) algorithms^9–12 for detection-gate association. Important is also the use of the JPDA algorithm in case of detection shared by more tracks. Moreover, in order to make more accurate the tracking and to reduce the false track probability, we assume the use of the interacting multiple model (IMM) algorithm.^9–12

The sampling time of the IRST is $T_I=157\mathrm{ms}$, while the radar one is $T_R=1\mathrm{s}$. To synchronize both data flows we used the interpolation between samples method. The radar measures range errors is 75 m rms and bearing error 0.5 deg rms in one case, 1.0 deg rms in the second. The IRST angular measurements error is 0.6 mrad rms.

Different motion models are used by IMM for the different process phases.

For the radar.

For the formation tracks the models are:

For the established tracks the motion models are:

For the IRST.

For the formation tracks we have:

For the established tracks, it is:

The data fusion system is characterized by the following features:

$K_0=3$, ${\gamma }_0=2.5$, ${\gamma }_1=3.75$, and a gate threshold $\gamma =20$.

Even for the fusion system we used the IMM algorithm characterized by these three motion models:

For the motion along the altitude direction we adopted only a linear motion model characterized by an acceleration process rms of $0.025\mathrm{m}/{\mathrm{s}}^2$ and time constant ${\tau }_z=60\mathrm{s}$.

The probability of correct association of the mixed detection with the predicted tracks, i.e., of correct fusion of the data from the two sensors, was practically 100% in all performed tests, with the exception of the case where the distance between two targets was close to the resolution of the two sensors and the relative velocity was close to zero. In that condition, the probability falls to 50%.

We consider as merit figures for the data fusion system:

We checked the above merit figures by using a target placed at the distance greater than 70 km, an altitude of 50 m, approaching the observer with a constant velocity of $250\mathrm{km}/\mathrm{h}$ along the $x$- and $y$-directions with an angle of 45 deg with respect to the observer.

Figure 5 shows the behavior of the estimation errors at the time $k$ based on 0.5 deg rms bearing error, while Fig. 6 on 1.0 deg rms bearing error.

Fig. 5

Estimation errors (m) (radar angular measurement error 0.5 deg): (a) $x$-axis (m); (b) $y$-axis (m); and (c) altitude (m). (—— fused track estimation error, – – – radar track estimation error).

Fig. 6

Estimation errors (m) (radar angular measurement error 1 deg): (a) $x$-axis (m); (b) $y$-axis (m); and (c) altitude (m). (—— fused track estimation error, – – – radar track estimation error).

It is possible to see that the error in bearing impacts the performance of the radar as a standalone system, but it is practically negligible when data fusion is implemented. The general benefit in accuracy is ever evident.

It is interesting to observe in Figs. 5(a), 5(b), 6(a), and 6(b), the difference in behavior of radar estimation error with respect to data fusion estimation error: due to the polar-to-Cartesian coordinate conversion in radar data, to an error on the $x$-axis corresponds an error on $y$-axis, proportional and of opposite sign. In the specific case of the figures where the trajectory of the target is 45-deg off-axis, $\mathrm{\Delta }x\approx -\mathrm{\Delta }y$ (dash lines in the figures). The data fusion system significantly mitigates that effect (solid lines).

It is interesting to observe the behavior of the fused tracks: when the coefficient $\alpha$ of the fusion equation, is around 0.5, i.e., when radar and IRST concur with equal weight to the fused detection, the equal-probability ellipse (gate) of the fused track results orthogonal to the observer-target direction (Fig. 7). The angular radar error causes an error proportional to $\widehat{r}·{\sigma }_{\beta }$, the cross-range error, orthogonal to the range itself.

Fig. 7

The gate when $\alpha$ is around 0.5.

When the coefficient $\alpha$ tends to 0 the IRST data become prevailing in ruling the tracking. All the uncertainty due to the radar bearing disappears and the cross-range error is determined mainly by the IRST error angle. The gate is mainly determined by the radar error of the range estimation. The gate collapses almost in a segment (Fig. 8).

Fig. 8

The gate collapses almost in a segment when is $\alpha$ is around 0.

To verify the bias in the estimation error along each coordinate axis we used the normalized mean estimation error (NMEE)¹³ based on $N$ Monte Carlo runs. Using a confidence region of 95%, we consider the error without bias if it results

Our NMEE test is constituted of $N=30$ Monte Carlo runs of length $K=180$ samples (3 min long), using the target of the previous example. Figure 9 shows the graphs of the normalized estimated mean error for 0.5 deg rms bearing error, the bounds of the confidence region, $[-0.359,0.359]$, and the sample mean in each direction. We can observe that the samples $\mu (k)$ are within the confidence interval, except for some isolated points.

Fig. 9

NMEE (radar angular measurement error 0.5 deg): (a) $x$-axis; (b) $y$-axis; and (c) altitude. (—— NMEE, – – – mean value, - · - · - · bounds).

Finally, Fig. 10 provides the graphs of the NMEE based on $N=60$ Monte Carlo runs, but with an angular radar measurement error of 1 deg. Also in this case we can observe that the NMEE is within the 95% bounds of the confidence interval, now $[-0.254,0.254]$, except for some isolated points, showing a still acceptable zero mean estimation error of the fusion system.

Fig. 10

NMEE (radar angular measurement error 1 deg): (a) $x$-axis; (b) $y$-axis; and (c) altitude. (—— NMEE, – – – mean value, - · - · - · bounds).

6.

Conclusions

The algorithm to fuse tracks, proposed in this paper, provides a general procedure to manage not homogeneous data as in the case of radar and IRST. The paper considers both synchronous and asynchronous sensors. For that, it introduced a specific synchronization technique of the data flows. The simulation results prove the validity of the method through the significant improvement in the accuracy of tracking and in the absence of bias in the state error of the tracks.

The extension of the algorithm to the case of three-dimensional radar is fairly straightforward: in that case, in addition to the bearing, the radar also provides measurements of elevation making the generation of the mixed detection more immediate. Indeed, the comparison between radar and IRST tracks is no longer extended to all objects that are located inside a given band of uncertainty around the bearing of the radar track, as suggested by Eq. (6), but only to a reduced set of targets identified also by their elevation. That involves a computing time smaller than required by the case treated up to now.

The method can be also applied to tracking in polar coordinates by using the enhanced Kalman filter. In that case, the evaluation of the range and bearing variances is no more necessary. The enhanced Kalman filter introduces, as cons, a greater computing effort and instability in the case of significant measurement error or process noise or rough initializations.

It is clear that sensors, electro-optical, radar, or others, continue to be the core of the process of sensor data fusion because they provide the measurement of the reality. But the sensor data fusion can effectively mitigate defects and improves overall performance also in terms of reliability and data integrity, when these factors are important for the application.¹⁴ That by means of the appropriate management of fertile data and redundant data provided by the sensors as described in the paper.

As far as the sensor data fusion methods and techniques, object of continuous improvement in formalization,^9,15,16 we think that the paper provides an effective theoretical contribution and an appropriate implementation solution.