2.1

Effective area identification

According to the affine transformation model and certain sampling range and sampling interval of tilt coefficient and longitude angle, the original image is perspective transformed to obtain the full perspective space, i.e., the affine transformation space, as shown in Figure 1a, which shows part of the image after perspective transformation. To avoid the computation of the detector in the invalid region, the four boundary corner points of the original image are tracked to construct the effective region mapping of the affine space, as shown in Fig. 2.1b.

Figure 1

(a) affine space construction (b) Effective area identification extraction and aggregation

2.2

Feature point detection

This paper uses a modified FAST detector similar to the ORB algorithm for feature point detection in the feature point detection stage. FAST corner point is defined as: if any point to be detected is smaller or larger than the gray value of a certain number of sampled points in the neighborhood, it will be identified as a candidate corner point.

The feature points obtained by the FAST operator do not have scale information and orientation information, and the use scenario is limited. For scale invariance, this paper builds Gaussian pyramids by borrowing LOG detectors, DOG detectors and other multi-scale detectors. In order to achieve rotational invariance, the FAST corner points must be made directional, and the intensity center of the sampled region can be used as the principal direction of the feature points. Therefore, the detector in this paper obtains the feature point information with scale σ, direction θ, and fractional response R in the effective region of the affine transformation space.

2.3

Feature Point Description

(1)

ORB algorithm descriptor

The ORB algorithm compares the feature points with the gray scale information of the sampled points in the surrounding neighborhood during feature description to obtain a set of binary strings as feature point descriptors.

Where p is the current feature point, I(x), I(y) are the gray scale values corresponding to any sampled point pair x, y, respectively. For any feature point, N pairs of sampled points are obtained in its neighborhood, and an N-bit binary descriptor can be obtained in accordance with equation (1), as shown in equation (2), and N is 256 in the ORB algorithm.

(2)

Comparison of feature points and sample points

ORB descriptors use statistical learning to obtain sampled point pairs in order to improve the distinguishability of descriptors and reduce the correlation between descriptors. In order to improve the distinguishability and robustness of the descriptors, this paper adds the gray scale comparison information between the feature points and the sampled points based on the original descriptors.

Where denotes the gray scale comparison information between feature point p and sample point x. Then, using equation (3) to compare whether the gray scale comparison results between feature point p and sample point pair (x, y) are the same, if they are, the descriptor corresponds to position 1, otherwise it is set to 0. Then the binary descriptor part containing the gray scale comparison information between feature point and sample point can be expressed as (5)

(3)

Descriptors for estimating gradient information

Due to the effect of viewpoint difference and illumination change, the gray scale value of the same local region in the image to be matched will change to a certain extent. In order to further reduce the sensitivity of the descriptor to illumination and viewpoint change, this paper introduces the gradient information into the binary descriptor.

In this paper, the gradient of the sampling point to each direction is first calculated, then the gradient difference in the corresponding direction between x, y at any sample point is compared to obtain the binary number as part of the descriptor.

The above feature information is integrated to obtain the final binary descriptor as shown in equation (7) :

where, denotes the original ORB descriptor information, and represent the gray scale contrast information between feature points and sampled points and the gradient contrast information between sampled point pairs, respectively.

The detection and description of the above feature points are obtained in the affine transformation space constructed from the original image, and then all feature points need to be inverse-view transformed and remapped into the original single-view image, and finally the feature points and their corresponding binary descriptors based on the affine transformation space proposed in this paper can be obtained. The brief flow is shown in the figure 2.

Figure 2

Brief flow of the algorithm in this paper

3.1

Experimental data set and environment

The image datasets Adam group, magazine group and cup group have large perspective variations. Figure 3 shows some of the images in the dataset, where the Adam group characterizes Absolute Tilt, and nine images are selected, which are positive angle of view images, left and right tilt 45°, 65°, 75°, 80° images. Figure 4 shows some of the images in the dataset, magazine group characterizes Transition Tilt, and 10 images are selected, which are 0° tilt and 10°, 20°, 30°, 40°, 50°, 60°, 70°, 80°, and 90° tilt angles. The data were obtained using the Wall group of images characterizing the viewpoint change in the Oxford set, as shown in Fig. 5, with the images from a to f plotted in order of viewpoint change enhancement.

Figure 3

the image datasets Adam group (Absolute Tilt)

Figure 4

the image datasets magazine group (Transition Tilt)

Figure 5

the image datasets Wall group

The experimental environment is VS2015 and OPENCV3.0, and the algorithm is implemented in C++. Since the feature matching algorithm proposed in this paper is based on Affine transform space, Gradient contrast information and ORB descriptors, it is uniformly identified as AGORB (Affine-Gradient-ORB) in the experiments.

3.2

Image feature extraction experiments with different viewpoint variations

Experiment 1 counts the number of valid matching point pairs between pairs of images with different viewpoint variations. As shown in Figure 6, the Adam group counted the number of valid matching pairs between the left (right) 5° view and the positive view, left (right) 65° view, left (right) 75° view, and left (right) 80° view images, respectively; the magazine group counted the number of valid matching pairs between the 10° to 90° view pairs, respectively, based on the 0° tilt view. The cup group counted, separately, the number of valid matching point pairs for each algorithm in image pairs 1-2, 1-3 … 2-6.

Figure 6

Effective matching point results for Adam, magazine and cup groups

As can be seen from the figure, the traditional SIFT and ORB descriptors are difficult to find a certain amount of effective matching point pairs in the image pairs with large changes in view angle. For example, in the Adam group, when the view angle of another image to be matched changes more than 75° (i.e., image pairs 45&75), SIFT and ORB have basically failed and it is difficult to find effective matching point pairs.

The algorithms of ASIFT and this paper, which introduce the affine transformation space, solve the problem of image matching under large viewing angle by simulating the change of viewing angle, and can still find a certain amount of effective matching point pairs even between pairs of images with large viewing angle, which is much better than the traditional SIFT and ORB algorithms.

Also from the comparison, it can be seen that compared with the ASIFT algorithm, the algorithm proposed in this paper incorporates the ORB algorithm into the affine transformation space, and in order to improve the description performance of the binary descriptors, the algorithm adds a certain amount of gradient contrast information to the original gray contrast information, which effectively improves the description ability of the feature descriptors and thus obtains more effective matching point pairs.

Tables 1 and 2 statistically show the matching time between ASIFT and this paper’s algorithm in the experiments. The comparison experiments on multiple image pairs show that the time used by the algorithm in this paper is 39.68%-44.65% less than the traditional ASIFT in the Adam group and 40.80%-59.62% less in the magazine group.

Table 1

Feature point detection time comparison (Adam)

Table 2

Feature point detection time comparison (Magazine)

This is due to the fact that although the algorithm in this paper also obtains the affine invariance by constructing the affine transformation space, it improves the execution efficiency of the algorithm by dividing the effective region when constructing the affine change space, which avoids the detection of feature points in the invalid region, and at the same time, this paper introduces the binary descriptor into the affine space, which greatly improves the speed of feature point description and matching.

3.3

Multi-gradient reference experiments

Experiment 2 was conducted to demonstrate the effect of the gradient contrast information introduced in the algorithm of this paper on the algorithm. In the experiments, AORB denotes the binary descriptor with no added gradient contrast information; AGORB1 denotes the binary descriptor with added gradient contrast information in two directions only; and AGORB denotes the binary descriptor with added gradient contrast information in four directions as proposed in this chapter.

As shown in Figure 7, the experiments measure the descriptor performance by counting the number of correctly matched point pairs in the same set of feature points. As can be seen from the figure, the resistance of the binary descriptor to viewpoint changes is significantly enhanced after the introduction of the affine transformation space. From the comparison of AORB, AGORB1 and AGORB, it can be seen that the number of correctly matched point pairs rises significantly after the introduction of the gradient contrast information, and the richer the gradient contrast information, the stronger the performance of the descriptor, and thus the greater the number of correctly matched point pairs. This experiment proves that the gradient contrast information can effectively reflect the local information of feature points, and the gradient features in multiple directions can effectively improve the matching performance and robustness of descriptors, and also proves the effectiveness of this algorithm. However, richer gradient comparison information also means a higher time complexity, which needs to be traded off in use on a case-by-case basis.

Figure 7

Number of correctly matched point pairs for ORB, AORB, AGORB1, and AGORB