2.1.

Superpixel Segmentation

The superpixel is the basic component of many land cover classification tasks, and it is generally used for land cover segmentation before classification. The concept of superpixels was first proposed by Ren and Malik¹⁴ in 2003. Superpixels are formed through object-based methods for homogeneous pixel merging. Pixel blocks with regular shapes are generated, and good boundary attachment characteristics are obtained. Many superpixel-based segmentation methods have been proposed, including the graph-based method proposed by Felzenszwalb and Huttenlocher,¹⁵ the Ncut (normalized cuts) method proposed by Shi and Malik,^16,17 the superpixel lattice method proposed by Moore et al.¹⁸ and the turbopixel method proposed by Levinshtein et al.¹⁹ The simple linear iterative clustering (SLIC) method proposed by Achanta et al.^20–22 in 2010 displayed good segmentation performance in images taken by conventional cameras and has been a main superpixel segmentation method.

However, remote sensing images are very large, and the density distributions of different land covers can vary greatly in the same image. Conventional segmentation methods, such as SLIC, need to set stable segmentation thresholds, such as the maximum number of seeds; consequently, these methods cannot obtain a variety of segmentation scales according to the density distribution of each land cover. Therefore, these methods cannot be effectively applied for segmentation tasks involving remote sensing images.

2.2.

Land Cover Classification

In land cover classification, as a first step, scholars usually use superpixels to segment images into objects. Therefore, the unit to be processed is converted from pixels to superpixels. The segmentation results are then sent to various machine learning classification models to obtain classification results. For example, Liu et al.²³ used a multiscale superpixel-guided filter approach and classified land cover using high-resolution remote sensing images and a support vector machine (SVM). Subsequent studies improved this method in two aspects: the superpixel segmentation method and the classification model. Zhang et al.²⁴ optimized the superpixel segmentation method and designed an improved SLIC method to improve segmentation accuracy. Gu et al.²⁵ and Martins et al.²⁶ replaced the SVM with a deep convolutional neural network to model the superpixel content, thereby effectively improving classification ability.

Since the FCN, which frees the classification task from the presegmentation step, was proposed by Long et al.⁵ in 2015, an increasing number of scholars have applied FCNs in various fields. Kumar et al.²⁷ used an FCN to achieve the end-to-end segmentation of medical images. Mountelos et al.²⁸ used FCNs to perform wiper segmentation for unmanned vehicle images. Maggiori et al.²⁹ first applied FCNs to land cover classification in 2016 and obtained a significant improvement in accuracy compared to the accuracy of traditional methods. Subsequent scholars have improved FCNs in various ways. McGlinchy et al.³⁰ used Unet to replace FCNs for land cover classification, and Niu et al.³¹ used a deep network to classify hyperspectral images. Many scholars have focused on improving the structure of neural networks, such as SegNet³² and RefineNet,³³ to improve classification accuracy. However, due to the boundary blur and hole problems caused by the convolution step, it is generally difficult to obtain a breakthrough by continuously modifying the structure of the network.

2.3.

Postprocessing Methods for Image Classification

Some scholars have also performed relevant research work on the postprocessing of image classification results and proposed some relevant postprocessing methods. In conventional approaches, simple morphological methods, such as expansion and decomposition, are generally used for optimization, and Wang and Zhang³⁴ used this approach to perform hole filling in airport segmentation tasks. However, morphological methods require the operator to understand the overall image distribution and accurately set the morphological parameters. Due to the large differences between the parameters of different images or even different parts of one image, it is difficult to achieve an ideal optimization result. In 2016, Zhou et al.³⁵ proposed a method for optimizing FCN classification results using a CRF, and the classification results for the PASCAL VOC2012 dataset were obviously improved compared to those based on other methods. Subsequently, Shen and Zhang³⁶ and Biao et al.³⁷ applied this method to various applications, such as brain segmentation in medicine and image segmentation for unmanned vehicles, and achieved certain improvements. In the past 2 years, Chu et al.³⁸ and Du and Du³⁹ applied CRF methods to optimize the results of land cover classification and achieved good optimization results in water classification and building classification, respectively. However, CRFs also need preset parameters, and the whole image is optimized by the same set of parameters, thus resulting in the fixed optimization strength; in addition, CRFs are unable to fit the remote sensing images with different land cover distributions.

3.1.

Iterative Self-Adaptive Superpixel Segmentation Algorithm

Due to the differences in the density and complexity of land covers in different remote sensing images, current methods may lead to insufficient or excessive segmentation. For example, the land cover distribution of villages and towns is often relatively simple and should not be densely segmented, although cities that have complex land cover distributions should be segmented with high densities to obtain better segmentation results.

To solve the problems above, we propose ISSS, which can be used to adaptively select the number of iterations according to the density distribution of various land covers in remote sensing images, and superpixels of different sizes and different intensities can be segmented according to the complexity of the land cover. Figure 2 shows the workflow details of the ISSS algorithm.

Fig. 2

Workflow of ISSS.

In Fig. 2, the ISSS algorithm uses SLIC with a low seed number to segment the image in the first step, and the rough superpixels are then obtained. The segmentation result of rough superpixels is shown in Fig. 3. The main reason for setting a small number of seeds is to avoid oversegmentation at the beginning. Then, a parameter $M$ is defined to calculate the image complexity of pixel values and the spatial distribution inside the superpixels. The larger the value of $M$, the more complex the land cover classes inside the superpixel are. In addition, a fixed maximum superpixel complexity threshold $U$ is preset by experience to determine whether the segmentation of superpixels is sufficient or not by comparing $U$ with the $M$ of the current superpixel. If the $M$ of the current superpixel is less than $U$, then the current superpixel has been fully segmented. Otherwise, it is necessary to use SLIC to segment the superpixel two or more times again until the $M$ is less than the $U$. We define a superpixel whose $M$ is less than $U$ as a leaf-superpixel; other superpixels are branch-superpixels. The ISSS is completed when all the superpixels are segmented into leaf-superpixels. $M$ can be expressed by the following equations:

Fig. 3

Results after the first SLIC superpixel segmentation.

In Fig. 4, to display the difference between leaf-superpixels and branch-superpixels and the effect of ISSS more intuitively, we show the segmenting of superpixels from branch-superpixels into leaf-superpixels using ISSS. The parts in orange are branch-superpixels, and the others are leaf-superpixels. The land cover classes in the leaf-superpixels are very simple. In contrast, the land cover classes in branch-superpixels are complicated due to insufficient segmentation. In addition, the number of branch-superpixels decreases as the number of iterations increases. Finally, the non-orange superpixels in Fig. 4(b) were not segmented in Fig. 4(c), thus indicating that ISSS does not produce excessive segmentation.

Fig. 4

Sample of iterative segmentation. (a) Sample superpixel after the first segmentation step. (b) Second segmentation result. (c) All leaf-superpixels after multiple iterations, which can be used as the final output of ISSS.

3.2.

LCOM-PC: Land Cover Classification Optimization Method Based on Patch Complexity

The distribution of land cover inside the leaf-superpixels after running ISSS should be relatively simple. However, due to the fuzzy boundary issue and other problems, the classification results of Unet in the leaf-superpixels’ region are not simple. In this paper, LCOM-PC is proposed to use the boundary information of leaf-superpixels to optimize the classification result of Unet by adjusting the land cover classes in the leaf-superpixel region.

We defined the classification results of Unet in the leaf-superpixel region as a leaf-patch. A parameter $S$ was defined to represent the distribution complexity of land cover in each leaf-patch. The formula of $S$ will be derived later. As shown in Fig. 5, the $S$ of each leaf-patch is calculated in the first step of LCOM-PC. Then, a fixed maximum leaf-patch complexity threshold $L$ is preset. LCOM-PC can determine whether a leaf-patch is appropriate by comparing $S$ with $L$. If $S$ is less than $L$, then the classification results inside the leaf-patch are simple enough; therefore, Unet has performed well in the leaf-superpixel region, and the leaf-patch remains unchanged. If $S$ is larger than $L$, then Unet has performed poorly in the leaf-superpixel region because the classification results inside the leaf-patch are too complicated. In this case, all the land cover types in a leaf-patch are optimized to the class that occupies the largest proportion of the leaf-patch. After all the leaf-patches are optimized, the final output of LCPP-ISSS can be obtained by merging them. To display the effect of LCOM-PC more intuitively, we show the optimizing of the leaf-patch in Fig. 6.

Fig. 5

Workflow of LCOM-PC.

Fig. 6

Process demonstration through example. (b) ISSS result of area in (a), which is in a red rectangle; (c) code of each leaf-superpixel in (b), (d) the classification result of Unet, and (e) optimized result of LCOM-PC. $S$ of {9,15,16}, which is larger than $L$; therefore, the classification result is adjusted to the main class. $S$ of other regions is less than $L$. Classification results remain unchanged.

The parameter $S$ is calculated by the following equations:

4.1.

Data and Study Areas

The images used in this experiment come from the Gaofen-2 satellite,⁴¹ which is equipped with two panchromatic and multispectral charge-coupled device camera sensors with respective resolutions of 1 and 4 m. This setup can provide a 45-km combined mapping band, which yields multispectral images of $6908\times 7300\text{pixels}$. Considering the high resolution, high quality, and abundant details of Gaofen-2 satellite images, such images provide an ideal data source for land cover classification and are highly suitable for verifying the effectiveness of postprocessing methods.

To verify that the proposed method has good robustness in different regions, we used 50 Gaofen-2 satellite images as the verification dataset, which contains images of most of the main cities in China, such as Wuhan, Beijing, and Shanghai. A sample image is shown in Fig. 7. The area in Fig. 7(a) is situated in Wuhan, Hubei Province, China, with central geographical coordinates of 30°6′ N and 114°2′ E; the image data were taken on February 12, 2015. We cut each image into several patches in $720\times 680$ with 50-pixel overlapping as input data to reduce computing memory usage.

Fig. 7

GaoFen-2 image and example data. (a) GaoFen-2 multispectral imagery, (b) example of a sampled image patch, and (c) ground truth of the image patch.

4.2.

Experimental Results and Performance Analysis

All programs were executed in TensorFlow and Python 3.5. Four different land cover classes (including water, buildings, vegetation, and others) were selected for the experiment.

LCPP-ISSS includes mainly two parts: segmentation and optimization, which are completed by ISSS and LCOM-PC, respectively. Therefore, we conducted experiments separately on the segmentation and optimization results. First, we verified the effectiveness and superior performance of LCPP-ISSS in segmentation by comparing ISSS with SLIC. Then, we compared LCPP-ISSS with the current mainstream method, DenseCRF, to verify the superiority of the proposed method in optimization.

4.2.1.

Segmentation performance comparison between ISSS and SLIC

We experimentally compared the ISSS and SLIC methods and the segmentation result of two typical images with a size of $720\times 680$, as shown in Fig. 8. As is shown in Fig. 8(a), when the seed number of SLIC is small, the areas of the white factory building and small lake (which are marked by a red circle) are insufficiently segmented and these areas are mixed in the same superpixel with other land covers. In Fig. 8(b), when we increase the seed number of SLIC, the white factory building and small lake are divided into multiple parts; i.e., there is oversegmentation. In Fig. 8(c), in ISSS, the factory building and the small lake are independently divided into a single superpixel. By comparison, ISSS has a better segmentation effect.

Fig. 8

Comparison of ISSS and SLIC segmentation performance. (a) SLIC with a small number of seeds, (b) SLIC with a large number of seeds, and (c) segmentation result of ISSS.

Figure 8 shows that the result of traditional segmentation methods (such as SLIC) is severely limited by some parameters (such as seed number), and it is difficult to find a stable seed to apply to all situations very well. On the contrary, ISSS is an adaptive iterative segmentation method and is unaffected by the parameters, so ISSS can achieve better segmentation results in most cases. To further compare the segmentation performance of SLIC and ISSS, we calculated the relationship between the number of superpixels and the average complexity of superpixels for the two methods, and this comparison is shown in Fig. 9.

Fig. 9

Comparison of the number of superpixels and the average complexity of superpixels based on ISSS and SLIC.

With the increasing number of superpixels, ISSS can obtain a lower average complexity with a smaller number of superpixels faster than SLIC. Although SLIC can eventually achieve a similar average complexity, SLIC needs to segment the image into more superpixels, and the convergence is relatively slow. As we mentioned above, LCOM-PC optimized classification by adjusting the land cover classes using superpixels as units. Therefore, the segmentation method that can achieve the same segmentation effect with fewer superpixels is better. Obviously, ISSS is more suitable than SLIC for postprocessing tasks.

4.2.2.

Optimization performance comparison between LCPP-ISSS and DenseCRF

DenseCRF is currently widely used in neural network classification postprocessing. Therefore, we compared the proposed method with DenseCRF to validate the performance of the proposed method in terms of optimization. We performed several comparative experiments, and the results are shown in Fig. 10. To better reflect the difference in the classification results of the different methods, we draw the circles around areas where the methods obtained significantly different classification results. The region where the results produced by LCPP-ISSS are closer to the ground truth than those produced by DenseCRF is marked with a white circle. The region where the results produced by DenseCRF are closer to the ground truth than those produced by LCPP-ISSS is marked with a black circle.

Fig. 10

Comparison of LCPP-ISSS and DenseCRF. (a) Original classification result, (b) classification result of DenseCRF, (c) classification result of LCPP-ISSS, (d) ground truth, and (e) input image.

As shown in Fig. 10, the number of white circles is much larger than the number of black circles. Therefore, compared with DenseCRF, LCPP-ISSS displays outstanding performance. In addition, the overall results show that the optimization result of LCPP-ISSS is closer to the ground truth than the optimization result of a simple Unet and DenseCRF. To comprehensively compare LCPP-ISSS with other methods and quantitatively assess the performance of these methods, the overall accuracy (OA), intersection over union (IOU), and mean intersection over union (mean-IOU) were employed. OA is the percentage of correctly predicted pixels among all pixels. The IOU is a statistical value that reflects the consistency between predicted labels and ground truth. Let $Q_{i,j}$ denote the number of pixels of class $i$ predicted as class $j$ and $N$ represent the number of all pixels. The values of OA, IOU, and mean-IOU range between 0 and 1, and a higher value indicates higher accuracy and better performance. OA, IOU, and mean-IOU can be expressed as

Eq. (7)

$$\mathrm{OA}=\frac{\sum _i{Q}_{ii}}{\sum _i\sum _j{Q}_{i,j}},$$

Eq. (8)

$$\mathrm{IOU}=\frac{\sum _i{Q}_{ii}}{\sum _j{Q}_{ji}},$$

Eq. (9)

$$\text{Mean}\text{-}\mathrm{IOU}=\frac{1}{N}\sum _i\frac{\sum _i{Q}_{ii}}{\sum _j{Q}_{ji}},$$

Eq. (10)

$$N=\sum _i\sum _j{Q}_{i,j}.$$

Table 1 shows that compared with DenseCRF, the proposed method provides a 4% increase in mean-IOU and a 10% increase in accuracy. Table 2 shows that the proposed method outperforms DenseCRF in terms of the mean-IOU of the different land cover classes, especially in terms of the mean-IOU of vegetation, which improves from 92.313% to 96.36% under the proposed method. In addition, LCPP-ISSS is also superior to DenseCRF in operability. DenseCRF requires many parameters to be set manually, such as kernel functions, which differ greatly for different kinds of images, and adjusting parameters is cumbersome. In comparison, LCOM-PC requires fewer parameters to be set in advance; consequently, this method can avoid many limitations related to manual intervention.

Table 1

Comparison of different indexes.

Method	Mean-IOU (%)	OA (%)
Unet	77.41	82.56
Unet-DenseCRF	86.53	85.25
Unet-LCPP-ISSS (proposed method)	90.94	95.52

Table 2

IOU of different land cover classes.

Classes	Unet	Unet-DenseCRF	Unet-LCPP-ISSS (proposed method)
Water	82.5	85.42	91.81
Buildings	68.36	75.33	80.85
Vegetation	86.06	92.313	96.36
Other	82.73	83.069	94.74