1.

INTRODUCTION

Currently, ageing population is becoming a severe issue for many countries in the world, brining great pressure to the medical resources¹. Smart Home (SH) technology is a desirable approach to extend the role of healthcare management from traditional medical agencies to private homes². A smart home normally has systems with network and sensors, usually using Internet of Things (IoT) technologies³. The SH system often includes a diversity of nonobtrusive sensors, such as wearable sensors, video cameras or environmental sensors. It is therefore necessary to obtain useful health information from the sensor data collected. Under this circumstance, Human Activity Recognition (HAR) becomes a very important step to infer the behavior of home residents, providing meaningful information to the medical workers and careers^4,5.

HAR has attracted widespread research interests recently, and a great number of approaches have been presented to solve this issue^6-10. However, the majority of these methods are proposed to handle data collected with single modality sensors. In a real smart home setting, fused sensor-based activity recognition is confronted with several challenges. In this paper, we make use of the SPHERE smart home platform⁵ as an exemplar. 1) There is high heterogeneity and uncertainty for sensors with different modalities. Almost all sensors, including wearable, environmental and video sensors, are reliant on network availability for data collection. 2) From the data fusion point of view, different sensors or features have different impacts on the activity recognition performance. Even inside the temporal window of one feature, different positions of the data sequence may have diverse influences on the outcomes of activity recognition as well. However, existing methods usually treat the features or temporal positions equally without considering the diversity. This inspires us to investigate the importance of the attention feature maps from the channel and temporal level.

Motivated by the above analysis, and inspired by Reference¹¹, we propose an attention network-based human activity recognition approach in this work. An end-to-end Convolutional Neural Network (CNN)-based training framework is developed to extract representative features for activities with fused sensor data. Specifically, an attention mechanism is introduced to put more emphasis on informative features in channel and temporal dimensions. In order to attain this goal, we apply a 1-dimensional Convolutional Block Attention Module (CBAM) with two sub-attention modules, i.e., channel-wise and temporal-wise attention modules, to capture the interdependencies between channel and temporal dimensions of its convolutional features. We summarize the main contributions of this paper as below:

The paper is organized as follows. We introduce the proposed method in Section 2. Section 3 reports the comparative study. Section 4 summarizes the paper, and discusses the future work.

2.

METHODOLOGY

2.1

Overview

We formulate the sensor-based human activity recognition as a time-series sequence classification problem in this work, where the samples are temporal sequences and the classes correspond to activities. Figure 1 illustrates the pipeline of our method. Given the multi-sensor HAR dataset, we first perform pre-processing on the original data to get a series of segments which are suitable for the deep network training. The output of the pre-processing acts as input for the consequent network, which is denoted as , where N indicates the number of data segments, T is the time step length and D denotes the feature dimension. The full activity label set is given as y_i ∈ {1,2,⋯, K}, where K is the total number of activity classes. The proposed method is designed to assign activity class label for each sample i ∈ {1,2,⋯, N}.

Figure 1.

Method illustration.

2.3

Attention-based network: CNN+CBAM

Given the extracted feature map as input, we next introduce an attention-based network, which we use CNN+CBAM to denote in this work. We illustrate the architecture in Figure 1. For the first convolutional layer, the feature maps are taken as the extracted raw features, and for the second layer, they are taken as the output of the preceding CBAM layer. The first convolutional layer contains 256 kernels with size 3, and the second convolutional layer contains 128 kernels with size 3. We set the stride of both convolution layers as 2 to downsample the feature maps. We use rectified linear units (ReLUs) as nonlinear activation functions. The first fully connected layer contains 640 units, while the second fully connected layer consists of 200 units. The Softmax layer outputs the class probabilities on the 20 activity classes. We employ cross entropy as the loss function, and ADAM optimizer¹⁷ is utilized as optimization method. We set the training batch size as 64 in the experiments, and empirically set the training epoch as 30.

We next introduce the design of the CBAM layer. CBAM was proposed in Reference¹¹ for image classification task. In this work, we revise the original CBAM layer to suit for time-series data classification. Let F denotes the input feature, CBAM block learns a 1-dimensional channel-wise attention map M_c ∈ ℝ^D×1 and a 1-dimensional temporal-wise attention map M_t ∈ ℝ^1×t, as displayed in Figure 2. The processing of CBAM can be described as:

Figure 2.

Diagram of CBAM and the attention sub-modules.

where ⊗ indicates element-wise multiplication. The input feature F sequentially multiply with attention maps Mc and Mt, finally obtaining the output F”.

• (1) Channel-wise attention module. We generate a channel-wise attention map through employing the relationship between features in different channels. The information in temporal dimension is firstly aggregated via utilizing an average pooling operation and a max-pooling operation. Afterwards, both descriptors are fed into a common network to generate the channel-wise attention map M_c. The shared network consists of an MLP (multi-layer perceptron) and a hidden layer.

• (2) Temporal-wise attention module. The temporal-wise attention map is produced by using the relationship of features in temporal dimension. The channel information of a feature map is aggregated by means of a max-pooling operation and an average pooling operation as well, producing two 1-dimensional maps. We then concatenate the two maps and convolve them with a 1-dimensional convolution layer, which generates the 1-dimensional temporal-wise attention map finally.

3.

EXPERIMENTS

In this section, we validate the effectiveness of the CNN+CBAM method by means of two groups of experiments. The first group of experiments (Experiment 1) shows the improvement on the overall classification accuracy (abbreviated as OCA) brought by attention modules, individually and combined.

The second group (Experiment 2) compares the OCA values with respect to the number and position of different CBAM modules.

The dataset is split into two subsets, a training subset with 80% of the samples and a validation subset with the remaining 20% of the samples in a stratified manner. We run 10 independent repetitions on all experiments obtaining new training and validation partitions, and then average them as the final results. All the experiments are implemented in Pytorch framework on a PC which is equipped with GeForce RTX 3090 GPU and 16-GB of RAM.

The baseline CNN architecture was chosen based on the validation performance of a group of architectures with different numbers of feature maps, and we selected the one with the best validation performance. The other compared approaches are with the same baseline architecture without further hyper-parameter tuning. The attention modules use the default parameter setting as specified in Reference¹⁰. It should be noted that the CNN+CBAM framework could be applied on different architectures of CNN. However, we only present the results on the CNN architecture as described in Section 2.3.

Figure 3 illustrates the OCA curves with respect to the epoch number, where Figure 3a shows the improvement brought by channel-wise and temporal-wise attention modules, while Figure 3b displays the results using different numbers and positions of CBAM modules. Tables 1 and 2 show the numerical results of the compared methods respectively.

Figure 3.

(a): Validation performance comparison between CNN, CNN+channel, CNN+temporal and CNN+2CBAMs (Experiment 1); (b) Validation performance comparison between different number of CBAM modules (Experiment 2).

Table 1.

OCA values (%) for Experiment 1.

Table 2.

OCA values (%) for Experiment 2.

From Figure 3a, we see that both the channel-wise attention module and temporal-wise attention module contribute to the improvement of the classification accuracy. The channel-wise attention module brings more improvement of OCA value than the temporal-wise attention module. This may be because the data instances we use are segmented by time sequence. However, the starting or ending of actual activities are not necessarily related to the time sequence, which means there is no clear relationship pattern between the time position and activities we study. The channel in our experiments correspond to sensor modalities. The accelerometer and RSSI signals are collected via the wristband wearable device, which are tightly related to the activities of participants. However, the PIR sensors and video sensors only have values when human are present before these sensors. In addition, the data collected by PIR sensors are noisy, because the PIR sensor sometimes reports false positive values in some cases. The channel-wise attention module learns the importance of different channels, and assigns higher weight on the important feature maps, and depress the unimportant ones, which explains the larger improvement brought by channel-wise attention module. From Table 1, we can find the channel-wise attention module enhances the OCA value by 0.90%, while the temporal-wise attention module improves the OCA value by 0.11%. The CNN+CBAM which contains both attention sub-modules outperforms the baseline method by 0.93%.

From Figure 3b and Table 2, we can find that the position and number of CBAM modules influence the OCA value. Incorporating two CBAM modules achieves better classification accuracy compared with one CBAM module. When only one CBAM module is integrated, the CBAM module located in the higher layer contributes more to the classification performance. This is because the higher level features are more complex and abstract than the raw features. Therefore, they are more informative for the network to decide which channel or temporal position to choose with respect to attention. From Table 2, we can observe that the first CBAM module improves the OCA value by 0.27%, while combining the second CBAM module promotes the OCA value by 0.49%.

4.

CONCLUSION

This paper presented a CNN pipeline with attention mechanism for recognizing human activity recognition in sensor fusion scenario. The pre-processing step of the pipeline involves missing value imputation, feature-wise normalization and windowing, which generates usable data segments for the network training. We integrated multi-level CBAM modules with a CNN to extract discriminative features for activities. Each CBAM module contains a channel-wise attention module and a temporal-size module, which are devised to capture the interdependencies between channel and temporal dimensions of their convolutional features. We performed a series of experiments, whose results on the real SPHERE dataset justify that the proposed method could effectively recognize human activities based on fused sensor data, and the attention modules further enhance the performance of deep neural network. In the future, we will develop joint activity and localization recognition using attention-enhanced network based on fused sensor data.