Cross-modal representation learning based on contrast learning

WeiYang Mao; Jshardrom Xia

doi:10.1117/12.2640128

1 August 2022 Cross-modal representation learning based on contrast learning

WeiYang Mao, Jshardrom Xia

Proceedings Volume 12257, 4th International Conference on Information Science, Electrical, and Automation Engineering (ISEAE 2022); 1225710 (2022) https://doi.org/10.1117/12.2640128
Event: 4th International Conference on Information Science, Electrical, and Automation Engineering (ISEAE 2022), 2022, Guangzhou, China

Abstract

Cross-modal retrieval refers to accurate retrieval between data from different modes. Its core task is how to measure the similarity between different modal data across semantic divides. In this paper, a novel cross-modal retrieval model CLCMR is proposed, which makes the identification of similar samples of different images more inclusive and the identification of different samples more discriminative. Specifically, the comparative learning method is introduced into the model, the pretraining dataset is processed by the data enhancement method, and the image Encode is pre-trained to make the image features extracted by the same sample closer The difference in image characteristics of similar samples improves their recognition accuracy and accuracy. In the downstream task, in order to ensure the effect of Encode, three loss functions are selected to ensure the nativeity of the data mapping of each mode into the common space, that is, to reduce the modal loss caused by the transformation space. Finally, in order to semantically align images and text, the content of Fast-RCNN was added to the CLCMR network framework for cross-modal retrieval. In this paper, several experiments are conducted on the Pascal Sentence dataset and the XmediaNet dataset, and the results show that the CLCMR cross-modal retrieval framework proposed in this paper is used The mAP value is better than some of the methods commonly used at present, which verifies the feasibility and perfection of the method.

Citation Download Citation

WeiYang Mao and Jshardrom Xia "Cross-modal representation learning based on contrast learning", Proc. SPIE 12257, 4th International Conference on Information Science, Electrical, and Automation Engineering (ISEAE 2022), 1225710 (1 August 2022); https://doi.org/10.1117/12.2640128

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
11 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Data modeling

Feature extraction

Statistical modeling

Performance modeling

Image enhancement

Image processing

Visualization

Show All Keywords

Keywords/Phrases

Search In:

Publication Years