Multimodal fusion contrastive learning framework based on insole and wristband

Yuqi Zhu; Bin Luo; Qi Qiu; Tao Zhu

doi:10.1117/12.3034835

7 August 2024 Multimodal fusion contrastive learning framework based on insole and wristband

Yuqi Zhu, Bin Luo, Qi Qiu, Tao Zhu

Proceedings Volume 13224, 4th International Conference on Internet of Things and Smart City (IoTSC 2024); 132240H (2024) https://doi.org/10.1117/12.3034835
Event: 4th International Conference on Internet of Things and Smart City, 2024, Hangzhou, China

Abstract

Significant progress has been made in the field of Human Activity Recognition (HAR) through the application of deep learning. However, most existing studies employ supervised learning methods which require expensive and time-consuming labeled data acquisition. Therefore, contrastive learning has been applied to HAR, where data augmentation methods can effectively alleviate the problem of scarce labeled data. Furthermore, existing indoor activity recognition research based on sensors often uses multiple Inertial Measurement Unit (IMU) sensors, resulting in an obtrusive, uncomfortable, and unimodal sensor modality. Shoes, as a fundamental aspect of modern human life, offers advantages such as portability, concealment, and comfort compared to IMU sensors. Utilizing insoles as sensors for activity recognition data collection can effectively address the mentioned issues in practical applications. The fusion of insole and single IMU sensor data not only enhances robust predictions but also provides a more comfortable solution. In summary, we propose a contrastive learning framework based on the multimodal fusion of shoe insoles and single wristbands. This involves separately applying contrastive learning to the data from both sensors. Then, it computes cross-modal contrastive loss for distinct feature vectors to improve model performance. Results show that the multimodal fusion of shoe insoles and wristbands can yield superior outcomes when employing appropriate methods compared to single modality. Multiple cross-modal contrastive loss computations facilitates a more comprehensive understanding of the similarities and differences between feature vectors. Furthermore, even with scarce labeled data, contrastive learning excels.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Yuqi Zhu, Bin Luo, Qi Qiu, and Tao Zhu "Multimodal fusion contrastive learning framework based on insole and wristband", Proc. SPIE 13224, 4th International Conference on Internet of Things and Smart City (IoTSC 2024), 132240H (7 August 2024); https://doi.org/10.1117/12.3034835

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
6 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Sensors

Education and training

Data fusion

Data modeling

Machine learning

Sensor fusion

Performance modeling

Show All Keywords

Keywords/Phrases

Search In:

Publication Years