2.1.

Lidar–Camera Sensor Fusion

Generally, lidar–camera sensor fusion has been used for object detection. Image processing using a camera distinguishes objects based on the color information of an image. Therefore, various color information is used for object detection, but the detection performance is obviously affected by temporal changes of light. Also, since image processing is performed on a pixel-by-pixel basis in every image, it requires complicated calculations. A typical example is a histogram of oriented gradient (HoG). HoG is a technique for calculating a gradient from a portion of an image. Based on the calculated gradient, the object is segmented with the results of the histogram.³ Lidar processing for object detection estimates the contour based on the object information, such as the width, length, height, and position. However, it is difficult for the lidar-based detection method to obtain features, including the color of an object. To compensate their shortcomings of individual sensors, lidar–camera sensor fusion is used. Through sensor fusion, the computational complexity of image processing is reduced and the detection performance is improved. Lidar–camera sensor fusion is usually composed of three steps: calibration, feature extraction, and classification.^3,13

The first step is a calibration process that matches the coordinates of lidar and camera. In a calibration step, the distance between an object and two sensors is measured, and the position difference is compensated. The second step is to extract object information using lidar. In case of 2-D lidar process, its width and thickness are obtained, whereas 3-D lidar process extracts its width, thickness, and height of an object. The final step is the object classification using camera. The image obtained from a camera is categorized on its position of an object basis. The classification process is performed by using machine learning algorithms, such as support vector machine and AdaBoost.^2,3 AdaBoost is one of adaptive boosting and accomplishes the final strong classifier by adding weighted results to multiple weak classifiers.³

Another technique to apply in lidar–camera sensor fusion is occlusion reasoning methods. In the case of partially occluded objects, it is difficult to extract its features of a target accurately. Typical occlusion reasoning methods are edge-contour based reasoning and frame comparison reasoning. Edge-contour based reasoning is a method to infer an object by means of its edge and contour of an occluded object.^8–10 Frame comparison reasoning uses a continuously input image dataset in accordance with time. This method determines the occlusion by analyzing the continuous image data. The occluded object is inferred by comparing the current and previous frames. This method is very effective in case a short-time instant occlusion happens. However, it is difficult to estimate a consecutive occlusion because it is limited by buffer size.^11,12,14 These fusion methods still have a reliability problem due to sensitivity to light change. In cases in which a sensor is highly sensitive to light change, an image of an object can be created even in low light conditions, but the image loses contrast and appears blurry because of increased noise. In a real driving environment, this change tends to cause sensors to miss obstacles. In addition, it is difficult for lidar–camera sensor fusion to simultaneously detect objects and estimate their speeds in a single frame. Therefore, to overcome this limitation, lidar–radar sensor fusion has been proposed to detect occluded objects and their movement.

2.2.

Lidar–Radar Sensor Fusion

Lidar–radar sensor fusion is robust to environmental change than camera because it uses the laser and radio frequency signal.^4,5,15–19 To our best of our knowledge, conventional lidar–radar fusion has been mainly focused on detecting moving vehicles, and researches on pedestrian detection have not been investigated yet before. In previous methods, a vehicle is estimated by obtaining its width, length, and shape from lidar and then acquiring its velocity from radar.⁴

A typical occlusion detection method using lidar–radar sensor fusion is modeling-based method.²⁰ The vehicle measured by lidar shows an L-shape based on the shape of the front and side. However, in the case of non-line of sight (LoS) vehicles that are partially obscured by other obstacles, the lidar measurement data do not form a perfect L-shape. This imperfect shape is compensated for L-shape through Ramer algorithm.^20,21 However, they have some limitations to detect a pedestrian since a human has an arbitrary shape.

In this paper, we propose a new method for a partially occluded pedestrian detection in lidar–radar sensor fusion. In our proposed method, we introduce new concepts of an occluded depth generation and occlusion RoI for occluded pedestrian detections. In addition, we will explain the procedure of our proposed detection scheme in the next section.

3.1.

Object Detection

The radar measures the range and velocity information of the object.¹⁵ The lidar measures the distance, angle, and height of an object. The measurement data are represented as many scattering points. Through the clustering process, the scattering points are combined into several groups that are strongly dependent on the number of objects. In general, there are many methods for clustering lidar scattering data, such as distance-based clustering, standard deviation clustering, and K-means. Distance-based clustering is a popular method for grouping into the same cluster if the distance between each scattered point is within a threshold value. The threshold is calculated by a vector norm operation in accordance with distance. Standard deviation clustering is computed using the standard deviation of the object points obtained from lidar and generates a cluster if the threshold is not exceeded. This method is suitable for clustering objects in similar locations but, in contrast to distance-based clustering, it is not suitable for separating adjacent objects of different types. K-means is a well-known clustering technique. However, the clustering performance is strongly dependent on the initial value. In this paper, we used a simple distance-based clustering method.

Figure 2 shows their human lidar features according to various human postures. As can be seen in Fig. 2, the lidar data have unique features forming a streamlined shape. In order to characterize human lidar features, the lidar data are approximately fitted to a quadratic and higher-order polynomial function. The human slope features, with quadratic polynomial curve fitting, are uniquely distinguished from other objects. In this experiment, we use a low-resolution 2-D lidar (RPLIDAR produced by Robopeak). One of human characteristics is clearly represented by a quadratic curve, even though the low-resolution lidar has fewer data points than the high-resolution lidar. The approximated quadratic curve based on the lidar data is the human fitting curve. We extend this technique to the high-resolution lidar to extract human slope features.

Fig. 2

The fitting curves versus lidar measurement data according to several human postures.

3.2.

Sensor Fusion

This subsection describes the various processes of RoI generation and the sensor fusion schemes. The radar RoI is determined by the range and azimuthal angle of the radar output. In addition, the radar can measure a walking human even in a partially occluded environment. During walking, the human Doppler distribution has unique repetitive Doppler and micro-Doppler patterns because one leg is fixed as the other takes a step forward.²² Due to this radar Doppler pattern, a human is distinguishable from other obstacles. This human radar Doppler pattern is called a human radar feature. The lidar RoI is set based on the measured width, length, height, and slope of the fitting curve. Referring to Ref. 1, the thresholds for human width and length are set to 1.2 m, and the thresholds of height are from 0.8 to 2 m. In addition, the slope threshold of the human fitting curve applied in this paper is set to 10. The slope of the fitting curve is the maximum slope, as shown in Fig. 2. The width, length, height, and slope to generate the RoI are defined as the lidar human feature.

Generally, because they detect only LoS objects, detection algorithms based on lidar have difficulty in detecting whether or not objects are partially occluded by obstacles. To confirm the possibility of existing objects being hidden behind other obstacles, we define the new concept of occluded depth. The occluded depth is the area behind the object measured in the lidar; it is not measured in the lidar. However, an object may exist within this region. The occluded depth is set to the area filled from the outermost point of the object measured by lidar to the maximum detection distance of the lidar.

Figure 3 shows the creation of radar RoI and lidar RoI, how to make a connection between occluded depth and lidar RoI, how to generate occlusion RoI using occlusion depth and radar RoI, and how to create fusion RoI. We assume that the relative position difference between radar and lidar is compensated for after calibration processing. Figure 3(a) shows the creation of radar RoI. As shown in Fig. 3(a), radar RoI is expressed as an area, detected by radar, surrounding a target of interest. In other words, radar RoI is the rectangular area $M$ surrounding four edge points $m_1=(r_1\sin \frac{{\theta }^r}{2},r_1\cos \frac{{\theta }^r}{2})$, $m_2=(-r_1\sin \frac{{\theta }^r}{2},r_1\cos \frac{{\theta }^r}{2})$, $m_3=(r_1\sin \frac{{\theta }^r}{2},r_2\cos \frac{{\theta }^r}{2})$, and $m_4=(-r_1\sin \frac{{\theta }^r}{2},r_2\cos \frac{{\theta }^r}{2})$, where ${\theta }^r$ is the radar angular resolution and $r_1=d_r-\frac{\mathrm{\Delta }\mathrm{r}}{2}$ and $r_2=d_r+\frac{\mathrm{\Delta }\mathrm{r}}{2}$. And $d_r$ is the distance between the radar and the target of interest and $\mathrm{\Delta }r$ is the radar range resolution, equivalent to $c/2B$, in which $\mathrm{c}$ is the velocity of light and $B$ is the bandwidth. Radar RoI is strongly dependent on the radar range resolution and the angular resolution. Figure 3(b) shows the generation of lidar RoI. Like radar RoI, the lidar RoI is expressed as the rectangular area $G$ including four edge points surrounding targets of interest, that is, $g_1=(l_1\sin \frac{{\theta }^l}{2}+\frac{\mathrm{\Delta }\mathrm{w}}{2},l_1\cos \frac{{\theta }^l}{2})$, $g_2=(-l_1\sin \frac{{\theta }^l}{2}-\frac{\mathrm{\Delta }\mathrm{w}}{2},l_1\cos \frac{{\theta }^l}{2})$, $g_3=(l_2\sin \frac{{\theta }^l}{2}+\frac{\mathrm{\Delta }\mathrm{w}}{2},l_2\cos \frac{{\theta }^l}{2})$, and $g_4=(-l_2\sin \frac{{\theta }^l}{2}-\frac{\mathrm{\Delta }\mathrm{w}}{2},l_2\cos \frac{{\theta }^l}{2})$, where ${\theta }^l$ is the lidar measured horizontal angle of the target of interest, $d_l$ is the distance between the lidar and the surface of the target, and $l_1=d_l-\frac{\mathrm{\Delta }\mathrm{l}}{2}$ and $l_2=d_l+\frac{\mathrm{\Delta }\mathrm{l}}{2}.$ $\mathrm{\Delta }\mathrm{w}$ and $\mathrm{\Delta }l$ are predetermined width and length boundaries, respectively.¹ Figure 3(c) shows how to generate the occluded depth from lidar measurement. As shown in Fig. 3(c), occluded depth is created by the rectangular area P consisting of four edge points $p_1=(d_l\sin \frac{{\theta }^l}{2},d_l\cos \frac{{\theta }^l}{2})$, $p_2=(-d_l\sin \frac{{\theta }^l}{2},d_l\cos \frac{{\theta }^l}{2})$, $p_3=(d_{\max }\sin \frac{{\theta }^l}{2},d_{\max }\cos \frac{{\theta }^l}{2})$ and $p_4=(-d_{\max }\sin \frac{{\theta }^l}{2},d_{\max }\cos \frac{{\theta }^l}{2})$, where $d_{\max }$ is the lidar maximum detection distance. Occluded depth is generated from the objects detected in lidar. Figure 3(d) shows how to create occlusion RoI from occluded depth and radar RoI. We can identify the possibility of the existence of any occluded objects by utilizing occluded depth and radar RoI because radar measures any objects within the occluded depth; lidar, however, cannot measure such objects. Thus, occlusion RoI is a new region generated by overlapping the occluded depth and radar RoI, which process is necessary to find any occluded objects. As shown in Fig. 3(d), the occlusion RoI is created by the overlapping region $O$ consisting of four edge points $o_1=(r_3\sin \frac{{\theta }^l}{2},r_3\cos \frac{{\theta }^l}{2})$, $o_2=(-r_3\sin \frac{{\theta }^l}{2},r_3\cos \frac{{\theta }^l}{2})$, $o_3=(r_4\sin \frac{{\theta }^l}{2},r_4\cos \frac{{\theta }^l}{2})$ and $o_4=(-r_4\sin \frac{{\theta }^l}{2},r_4\cos \frac{{\theta }^l}{2})$, where ${\theta }^r>{\theta }^l$ indicates that the radar angular resolution ${\theta }^r$ is greater than the lidar measurement horizontal angle ${\theta }^l$. Note that the equations for the four edge points of occlusion RoI are the same as those for the four edge points of radar RoI, except for ${\theta }^l$ is used instead of ${\theta }^r$. Figure 3(e) shows the creation of fusion RoI, an overlapped region produced by combining radar RoI and lidar RoI. As shown in Fig. 3(e), the fusion RoI is generated by the overlapping region surrounding the four edge points $F_1$ to $F_4$. In other words, the fusion RoI $F$ can be simply given by $\mathrm{M}\cap \mathrm{G}$, where M and L are the radar and lidar RoIs, respectively.

Fig. 3

Procedure of RoI generation: (a) radar RoI, (b) lidar RoI, (c) occluded depth, (d) occlusion RoI, and (e) fusion RoI.

3.3.

Pedestrian Detection

An occluded object detection is a processing step to identify any object located in the occluded depth. The occluded objects are detected by utilizing the radar measurement results such as the measured range, velocity, and Doppler pattern. In this paper, we identify any hidden object as a pedestrian within the occlusion RoI by using this radar human feature. In the pedestrian detection step, in case of an occluded person, the radar human feature is only used for pedestrian detection. However, in case that an object is outside from an occlusion RoI, a person is detected by using the radar human feature and the lidar human feature that are labeled by width, length, height, slope of fitting curve, and Doppler.

Figure 4 is a step-by-step procedure of lidar and radar sensor fusion for pedestrian detection based on the occluded depth. Figure 4(a) is a snapshot picture showing a situation in which a pedestrian is obscured by another pedestrian. Figure 4(b) is the lidar measurement data. Figures 4(c) and 4(d) are the radar RoI and lidar RoI, respectively. The lidar measures only the front one of two pedestrians. Figures 4(e) and 4(f) show the occluded depth, and the fusion and occlusion RoIs, respectively. As mentioned above, the fusion RoI is generated by superimposing lidar RoI and radar RoI. And, the occlusion RoI is built by overlapping the occluded depth and radar RoI. These fusion RoI and occlusion RoI are specified zones for occluded and not occluded object detection, respectively. Figure 4(g) is the final pedestrian detection result showing occluded pedestrian represented by occlusion RoI as well as a LoS pedestrian in fusion RoI. The occluded pedestrian is detected from the radar Doppler pattern within the occlusion RoI. And also, the pedestrian detection is done by using the radar human feature and the lidar human feature within the fusion RoI.

Fig. 4

Procedure of lidar and radar sensor fusion scheme based on occluded depth: (a) a snapshot picture of an occlusion situation, (b) the lidar measurement data, (c) the radar RoI, (d) the lidar RoI, (e) the generated occluded depth, (f) the fusion RoI and occlusion RoI, and (g) the pedestrian detection results.

4.1.

Experimental Setup

We used a Velodyne VLP-16 lidar and a 24-GHz frequency modulated continuous wave (FMCW) radar^23,24 in the experiment. Table 1 shows their critical specifications. The radar was attached to its bumper position of a vehicle and the lidar was installed on about 2 m height from the ground. They have similar configurations of the autonomous vehicle. Experiments were performed at varying obstacle distances by about 6, 8, and 10 m. Multiple target experiments were made under the environment that two moving people get close and separate each other. The developed logging board and software²⁵ were used to collect the radar measurement data.

Table 1

The specifications of the lidar and radar used in the experiments.

The radar used a fast-ramp based FMCW modulation.¹⁵ The signal reflected from an object was sampled by means of an analog digital converter. The sampled signal per ramp was transformed by the range-frequency spectrum through a range-FFT. Generally, the signal reflected from a pedestrian can be masked off due to a strong surrounding clutter because a pedestrian has a relatively low reflectance. Thus, a moving target indication removed clutter components and extracted only a moving object. And then, a range-Doppler map was built from a Doppler-FFT processing. The object was finally extracted by finding peaks with an adaptive threshold based on cell averaging constant false-alarm rate in a 2-D range-Doppler map. The obtained distance and velocity are the object information in the radar RoI. The Doppler pattern of a pedestrian was obtained from a Doppler-FFT processing.

4.2.

Experimental Results

Many experiments are made in various indoor and outdoor situations. In the experiments, targets mean both ordinary and partial occluded pedestrians. Figure 5 summarizes the indoor experimental scenarios and some typical examples of final detection results. In the indoor scenario (1), the pedestrian moves without any occlusion. In scenario (2), pedestrians are partially hidden by obstacles. In scenario (3), pedestrians are hidden temporarily and they move out any LoS space. In the experimental results, lidar RoI is represented by $L\_\mathrm{RoI}$, radar RoI by $R\_\mathrm{RoI}$, occluded depth by $Occ\_D$, fusion RoI by $F\_\mathrm{RoI}$, and occlusion RoI by $O\_\mathrm{RoI}$. Also, the symbol $\#P$ means that a person has been detected.

Fig. 5

Indoor experimental environment.

Figure 6 shows an example of pedestrian detection results in several indoor open environments. In an open environment, as shown in Fig. 6, a pedestrian was clearly measured in both the lidar and the radar, simultaneously. Figure 6(a) is the experimental real picture. Figure 6(b) shows the lidar measurement data. The horizontal boundary in the radar RoI was determined by the azimuthal coverage of radar. The vertical boundary is set to any predetermined threshold that identifies a pedestrian in Ref. 1. In Figure 6(c), the radar RoI and the lidar raw data cluster are represented by the dash-dot rectangle and black points, respectively. Since there is only one moving object in the radar RoI, the object in this cluster was moving. The lidar RoI was indicated by the dot rectangle, which made by clustering the measured data with the appropriate human width, length, height, and slope thresholding. To obtain its precise position of an object, the fusion RoI was extracted by overlapping the radar RoI and lidar RoI each other, as shown in Fig. 6(e). The fusion RoI represents line rectangle, as shown in Fig. 6(f). The occluded depth was assigned as an area filled between two lines in Fig. 6(g). Figure 6(f) shows the final detection results in the situation without any barrier. To confirm a pedestrian within the fusion RoI, the radar measured Doppler pattern was used. Figures 6(i) and 6(j) show the radar measurement range and Doppler spectra in frequency domain, respectively. Figure 6(j) is the velocity distribution of a pedestrian measured from radar. The measured instantaneous velocity has a broad distribution about $5\mathrm{km}/\mathrm{h}$ due to a pedestrian movement. Figures 6(k) and 6(l) show the measurement range and Doppler variations along a time of the radar output for a moving pedestrian, respectively. A pedestrian moved back and forth repeatedly in front of the sensors. As depicted in Fig. 6(l), the radar Doppler distribution is caused by the leg and arm sway during a human movement. Generally, a moving pedestrian has repetitive Doppler and several micro-Doppler patterns when the pedestrian moves one leg fixes and the other swings.

Fig. 6

The experimental results in an indoor scenario: (a) experimental environment, (b) the lidar measurement data, (c) the radar RoI, (d) the lidar RoI, (e) the RoI overlay for extraction of the fusion RoI, (f) the Fusion RoI, (g) the occluded depth, (h) the final pedestrian detection result, (i) the radar range spectrum, (j) the radar velocity spectrum, (k) the radar range variations of a pedestrian with time, and (l) the radar Doppler variations of a pedestrian with time.

Figure 7 summarizes the outdoor experimental situations and some typical examples of final detection results. In the experiments, single-target and multitarget were taken into considerations. Also, various occluded scenarios were included considering momentary and partial occlusion in both temporal and spatial events. The outdoor scenarios (1), (2), and (3) have the same configuration as the indoor scenarios. The outdoor scenarios (4) and (5) were considered for multiple targets and partial occlusion situations.

Fig. 7

Outdoor experiment situations and some final detection results.

Figure 8 shows the pedestrian detection result in an outdoor scenario (2) with the front occlusion. Figure 8(a) is a snapshot picture showing an occluded pedestrian situation in the outdoors. Here, a pedestrian is partially occluded.

Fig. 8

The experimental detection results for the front occlusion scenario: (a) a snapshot picture showing a partially occluded pedestrian in an outdoor experimental environment, (b) the LIDAR measurement data, (c) the occluded depth and the RADAR RoI, (d) the partially occluded pedestrian detection result, (e) the measured RADAR range spectrum, (f) the measured RADAR Doppler spectrum, (g) the RADAR range variations of a pedestrian with time, and (h) the RADAR Doppler variations of a pedestrian with time.

Figure 8(b) is the measured lidar data. In the measured lidar data, the pedestrian is not measured because it is occluded by the front obstacle. Figure 8(c) shows the radar RoI and the occluded depth. The occluded pedestrian can be detected by overlapping the radar RoI and occluded depth. Figure 8(d) is the occluded pedestrian detection result using the radar Doppler distribution. Figure 8(e) is the radar measured range spectrum corresponding to the obstacles, trees, and a pedestrian. Figure 8(f) is the radar measured Doppler spectrum. A broad peak is observed at a velocity of about $5\mathrm{km}/\mathrm{hr}$ and this peak corresponds to an occluded pedestrian. Figures 8(g) and 8(h) are the radar range and Doppler variations along time in a condition that a pedestrian is moving with a partial occlusion, respectively.

Figure 9 shows one of detection results for an occluded pedestrian in many outdoor experiments. Figure 9(a) is a real experimental picture. A pedestrian is partially occluded by the front barrier. Figure 9(b) is the lidar measurement data. In spite that a pedestrian exists behind obstacles, the pedestrian is not detected. Figure 9(c) is the occluded depth and the radar RoI. In the proposed algorithm, the occluded depth and the radar RoI overlay each other, and the occlusion RoI is generated by superimposing the radar RoI and the occluded depth. Figure 9(d) shows the detection result of an occluded pedestrian by using occlusion RoI. Figures 9(e) and 9(f) are the radar measured range and Doppler spectra, respectively. Two peaks corresponding to obstacles and a pedestrian are clearly separated in the range spectrum and a broaden peak in the velocity of about $5\mathrm{km}/\mathrm{h}$ is observed in the Doppler spectrum. Figure 9(g) is another picture of the same experiment. Figure 9(h) is the lidar measurement data. A pedestrian is detected close to the adjacent obstacle. Figure 9(i) is the estimated radar and lidar RoIs. Figure 9(j) is the fusion RoI and the final pedestrian detection results. A pedestrian and its neighboring obstacles are separated by using the radar velocity information within the fusion RoI. Figures 9(k) and 9(l) show the measured range and Doppler spectrum of the radar output for the objects, respectively. A pedestrian and its neighboring obstacles in the range spectrum are merged because of very close distance each other. However, a pedestrian is clearly distinguishable from stationary obstacles in the Doppler spectrum. Figures 9(m) and 9(n) show the range and Doppler measurement along a time of radar output for the pedestrian movement in this experiment, respectively.

Fig. 9

The experimental detection results for a single occluded pedestrian: (a and g) a snapshot pictures showing an outdoor experimental environment, (b and h) the lidar measurement raw data, (c) The occluded depth and the radar RoI, (d) the occlusion RoI and occluded pedestrian detection result, (e) the radar measured range spectrum for the pedestrian detection, (f) the radar measured Doppler spectrum for the pedestrian detection, (i) the predicted fusion RoI, (j) the pedestrian detection result within the fusion RoI, (k) the radar measured Doppler spectrum, (l) the radar measured Doppler spectrum, (m) the radar range variations of a pedestrian with time, and (n) the radar Doppler variations of a pedestrian with time.

To verify the detection performance in various complex outdoor environments, we also performed multiple target experiments. Figure 10 shows the multitarget experimental results considering that an occlusion partially occurs. The experiment was carried out in real pavements. Figures 10(a) and 10(g) show actual experimental pictures about situations where two pedestrians are moving. Figure 10(b) shows the lidar measurement raw data. In this situation, an occlusion did not occur, and both pedestrians and the surrounding objects were measured in the raw data. In Fig. 10(c), the $R\_\mathrm{RoI}$ stands for radar RoI. Because the two pedestrians moved very closely, they were located inside one unified radar RoI marked by a dash-dot square. The lidar RoI is shown in two dot boxes. Figure 10(d) shows the fusion RoI generated from the respective detection result of each pedestrian. Figure 10(e) shows the range profile of the radar output for two pedestrians. There are many other clutters besides two pedestrians.

Fig. 10

The experimental detection results in real complex environments: (a and h) a snapshot picture, (b and i) the lidar measurement raw data, (c) the predicted fusion RoI, (d) the fusion RoIs and pedestrian detection results, (e) the measured radar range profile, (f) the measured radar Doppler profile for pedestrian #1, (g) the measured radar Doppler profile for pedestrian #2, (j) the occluded depth and the radar RoI, (k) the partially occluded pedestrian detection result, (l) the measured radar range profile, (m) the measured radar Doppler profile, (n) the radar range variations of two pedestrians with time, and (o) the radar Doppler variations of two pedestrians with time.

Figures 10(f) and 10(g) show the Doppler profiles of radar outputs for pedestrian #1 and pedestrian #2, respectively. According to these radar Doppler profiles, pedestrians are distinguished from clutters. Figure 10(h) is a snapshot picture that one person is partially obscured by the other person in the same scenario. Figure 10(i) shows the lidar measured data. A pedestrian is clearly represented by many scattering points while a partially occluded pedestrian has sparse points. Figure 10(j) shows the radar RoI and the occluded depth. The radar RoI and occluded depth are marked in the dash-dot squares and the in gray line, respectively. In addition, the lidar RoI and the occlusion RoI are described by the dot box and the blue line box, respectively. Figure 10(k) is the final pedestrian detection result. Here, the radar Doppler human pattern and human fitting curve were utilized to identify a pedestrian. Figure 10(l) shows the radar range profiles for two pedestrians in close proximity. As shown in Fig. 10(l), in case that two pedestrians are very close, they cannot be separated by only using radar range profiles. However, our proposed algorithm distinguishes two pedestrians with the Doppler difference in the radar Doppler profile. Figures 10(n) and 10(o) show radar range and Doppler variations along a time for two pedestrians, respectively.

According to various indoor and outdoor experiments, we proved that our proposed scheme is very effective to detect a partially occluded pedestrian. Table 2 summarizes the pedestrian detection rates analyzed from various experiments. In the previous studies, sensor fusion methods based on lidar and radar have not yet been investigated for detecting occluded pedestrians. In this paper, in order to verify the effectiveness of the proposed sensor fusion method at detecting partially occluded pedestrians, we have demonstrated detection improvement using the cases with and without the proposed method. Without our proposed method, the detection rates in indoor and outdoor are about 45% and 53%, respectively. This is caused by no detection of occluded pedestrians. However, with our proposed method, the detection rate of about 89% or more is obtained in both indoor and outdoor because occluded pedestrians are detected by using an occluded depth and the radar human feature. According to the experimental results, our proposed sensor fusion scheme makes much more improvement in detecting a partially occluded pedestrian in temporal and spatial events. Even though in the case of real road environments surrounded by various obstacles such as trees and light poles and so on, our proposed method has much higher detection rate of about 89%.

Table 2

Detection results of indoor and outdoor scenarios.