3.1

CNN architecture

In this paper, we use a CNN model consisting of a 2D convolution of 3, with each convolutional kernel of size 3. The initial convolutional layer is configured with an input channel of 2 and produces an output channel of 64; The second convolutional layer utilizes 64 input channels and produces 64 output channels, followed by a layer of nonlinear activation ReLU, and the third layer is computed using a convolution kernel size 3 in steps of 2, with 32 channels of output, followed immediately by a layer of ReLU nonlinear activation function. The convolutional layer’s output is ultimately transformed by a fully connected layer. This paper provides a detailed description of the model (see Fig. 1).

Fig. 1.

CNN model.

3.2

Multi-heads attention architecture

We present a novel neural network structure that combines attention mechanisms and convolutional layers and connects and unifies them by using a convolution with a kernel size of 1. We start our description with the convolutional part.

Convolutional layer: The model presented in this paper incorporates a two-part structure within its convolutional layer. The initial part is the standard convolution, which will convolve the input, using a kernel size of 3. The next step involves performing a convolution operation, utilizing an expansion factor of 2 to effectively enhance the perceptual field of the convolution layer, again using a kernel size of 3. The outputs of the two parts are cascaded together to form a single output. This cascaded convolutional layer structure helps the network to capture features at different scales, thus improving the performance of the network.

Attention mechanism: In the work in this paper, our attention mechanism differs slightly from the usual attention mechanism in that we first treat the input channels as heads and establish a mapping to a different set of attention heads. In this case, when provided with input of a specific shape (C, L, D), to start, we initially interchange the positions of the first and second dimensions and spread the matrix of shape X ∈ R^L×CD so that W_q, W_k, W_v ∈ R^L×CD becomes a linear transformation that can be learned (where C denotes the quantity of incoming channels or heads, the sequence’s size is denoted by L, and the dimension of each individual element in the sequence is denoted as D) applied to the matrix X ∈ R^L×CD. Next, we compute the attention score, which is the dot product of Q and K divided by a root under D. Q, K, V are the query, key and value, respectively, of the attention mechanism. These attention scores are then applied to V to obtain the outcome obtained from the attention layer O. Finally, we map O back to the 3D shape by permutation such that O ∈ R^L×CD. The output of our attention layer formulation is as follows:

In summary, given the input X, We execute the subsequent mapping:

Unifying the combination: The above work we have processed the input using the convolution and attention layers separately, next we need to join their results together for the next step. We used a separate matrix to store these results and used a convolution with a kernel size of 1 to unify them, and added layer parametrization and ReLU activation functions later on to further improve the performance of the model. This process resulted in what we call a hybrid multi-headed attention/extended convolution layer.

Classifier: In the final work, we spread the connection results from the previous work and send them to a fully connected linear feedforward neural network with a final layer using a softmax activation function to map the output to a category probability distribution for classification prediction. During training, Cross entropy serves as the employed loss function to minimise the gap between the predicted values and the true labels.

In the end, our model comprises two primary hybrid layer components in its architecture. The first part has 2 heads () (C = 2) and outputs 16 heads . The convolutional layer receives 2 channels of 2D input and outputs 32 channels of the same size, the inputs are subsequently evenly distributed to a second mixed layer component of identical dimensions. Finally, a single-layer neural network is used to classify the input, which contains a hidden layer of 1024 neurons, using Re LU as the activation function. The model is visualised (see Fig. 2).

Fig. 2.

Hybrid multi-headed attention/CNN convolutional model.

3.3

Metrics

In our study, we used AUC, a measure of data separability, and ROC curves, probability curves are generated by graphing the rate of correctly identified positives against the rate of incorrectly identified negatives, as measures for evaluating the model’s performance. We also measure the classification accuracy of the model using the F1 score, which is the summed average of accuracy and recall and is a measure of classifier accuracy. In addition we also use mean accuracy (MAP) ¹⁰ to assess the efficiency of information retrieval. MAP is evaluated by ranking the true labels by predicting the probability, A portion of the highest K probabilities, as determined by the subsequent equation:

where r_i is the real label of the r_i th electricity user, r_i=1 if it is a thief and 0 otherwise.

4.1

Data methodology

Zheng et al ⁷ proposed converting input 1D data into 2D data, where the data is organised into a grid-like structure in the spatial dimension, thus allowing spatially based models to better explore the structure and features of the data. Our work allows one-dimensional data, which is only temporally informative, to be extended to two-dimensional data with spatial information, allowing the use of computer vision models (e.g. 2D convolutional neural networks) to explore the periodicity and neighbourhood features in the data. This approach has been used in areas such as time series prediction and anomaly detection.

4.2

Missing data process

Missing data is a widespread problem and there are two common approaches to dealing with this situation in common literature research work. The first approach is to remove incomplete parts of the data extracted from the dataset, However, this method has the potential to overlook significant data. The second is to use interpolation or median values of data features to estimate the missing values ⁸. While these techniques have been shown to be effective, they make assumptions about the missing data and may therefore have a negative impact on the predictive model. In particular, if there is a systematic bias between the way the data are missing and the estimation method, then this bias may be amplified in the model, leading to inaccurate prediction results. Therefore, there is a need for some more effective methods to maintain the accuracy and robustness of the forecasting model with respect to missing data.

To handle missing values, the method creates a binary mask to indicate the location of the missing data. First, we need to determine the index of all generate a binary mask to identify and address the absent data. In this binary mask, the missing data positions are assigned a value of 1, While assigning a value of 0 to all other positions. This binary mask can be added as an additional channel to the model’s input to help the model better handle missing values. By using the binary mask, the model can clearly understand where the missing data is located and take appropriate action when dealing with the missing data, such as interpolating or otherwise processing the missing data. More information about this method can be found in the following chart (see Fig. 3).

Fig. 3.

Top left: original data, top right: missing data padded with zeros; Bottom left: binary mask, bottom right: final data.

4.3

Data preprocessing

The real SGCC data exhibits a strong long-tailed distribution of skewness and kurtosis. These features may have an Effect on the model’s performance and therefore the data needs to be processed appropriately to make the most of them. In Section 3.2, We explore strategies for handling incomplete data or anomalous values within the dataset. The analysis of the SGCC dataset concluded that most of the outlier cases were found in conventional electricity customers, We refrained from removing this data in order to retain valuable information. Before normalising the data, we treated the dataset as a time series and investigated it with a single factor evaluated at consistent time intervals. To assess possible correlations and periodicity, we collected data on electricity usage throughout the week, starting from Monday till Sunday and constructed correlation matrices between electricity usage and date for electricity theft customers and regular electricity customers. The above is shown in the chart below (see Fig. 4 and Fig. 5).

Fig. 4.

Correlation Matrix: Normal Electricity Customers.

Fig. 5.

Correlation matrix: Thieves.

The analysis in Figures 4 and 5 shows that the thieves’ electricity consumption patterns show a clear cyclicality and correlation, suggesting that the thieves may have some regularity in stealing electricity. In contrast, the electricity consumption patterns of the average electricity user are relatively smoother. We leverage these variances to enhance the model’s performance to better detect and predict the electricity usage behaviour of thieves.

Analysis of SGCC through the data has the phenomenon of heteroskedasticity, i.e. the trend of variation in the data is not constant, which results in the data distribution being asymmetrically positive or Leptokurtic, i.e. the right-hand side of the distribution exhibits significant variability, leading to an elongated tail, as shown (see Fig. 6). The fluctuating variability, which is not constant, can result in artificial interactions within the deep learning model. To address this issue, we used quantile uniform normalisation. The quantile uniform transformation can be applied to each feature data independently, distributing the most frequent values between (0,1). Specifically, the raw data are arranged in order of size and mapped onto a cumulative distribution function (CDF), and these values are then dispersed into a number of quartiles. In our work 10 quartiles were used. However, The Quantile transform poses a challenge in terms of the volume of data needed to execute the transformation. Lessons learned, at least 10×m samples are needed to create m quartiles. In practice, therefore, we need to take care to ensure that the data samples are sufficient to avoid errors due to insufficient data. The data distribution after processing is shown (see Fig. 7). It can be seen that the long tail phenomenon of the data distribution has been somewhat alleviated by uniform normalisation of the quartiles, thus providing a more accurate data base for subsequent deep learning modelling.

Fig. 6.

Power consumption data for 100 samples: raw data.

Fig. 7.

Power consumption data for 100 samples: data processed by Quantile transformation.

5.1

Attention augmented convolution network

We employed a convolutional network with attention augmentation ¹¹ as a comparison algorithm in our experiments. The algorithm is a self-attentive mechanism that is suitable for two-dimensional tasks and can be used as an alternative to convolutional neural networks. The process involves the integration of characteristics obtained from the convolutional layer by means of cascading with self-attention, thus improving the performance pertaining to the model. The findings are presented in Table 2.

Table 2.

Key results.

5.2

Baselines

Studies using innovative strategies in deep learning to detect electricity usage anomalies in big data of electricity are rare in the literature and the dataset used in this study is also rare. To compare with other methods, this study uses Wide and Deep techniques to model SGCC for electricity theft detection ⁷. The objective of the Wide component in this study is to acquire comprehensive knowledge, while the CNN layer focuses on extracting the characteristics of the power consumption data. The combination of these two related components produced excellent AUC performance index of up to 0.79% and a value of MAP@100 above 0.96.

5.3

Results and discussion

Key findings derived from our model are presented in Table 2. Layered k-fold training was performed on 50%, 75% and 80% of the data. The hybrid multi-headed attention/extended convolutional layer significantly outperforms the baseline. Furthermore, the results achieved by our two models, along with the attention-enhanced convolutional network, clearly demonstrate the substantial enhancement in data preprocessing brought about by the implementation of quantile transformation. The attention mechanism led to a significant improvement in F1 scores. Another remarkable behavior is that the attention model converges much faster compared to convolutional neural network model. In our examinations, the convolutional neural network required about 100 epochs to reach convergence, while the hybrid attention required about 20 epochs. The following chart shows the variation of scores over time for the hybrid multi-headed attention/extended convolutional layer (see Fig. 8).

Fig. 8.

Metrics by Epochs: 80% of the training set.