Research and implementation of efficient retrieval algorithm in big data environment

Pan Gao; Shuhua Shao

doi:10.1117/12.3014436

9 January 2024 Research and implementation of efficient retrieval algorithm in big data environment

Pan Gao, Shuhua Shao

Proceedings Volume 12969, International Conference on Algorithm, Imaging Processing, and Machine Vision (AIPMV 2023); 129690H (2024) https://doi.org/10.1117/12.3014436
Event: International Conference on Algorithm, Imaging Processing and Machine Vision (AIPMV 2023), 2023, Qingdao, China

Abstract

Under the background of digital information age, faced with the increasing data scale and complexity, the application limitations of traditional centralized retrieval services are becoming more and more obvious, and it is urgent to improve the data structure expansion, incremental update control and retrieval operation efficiency. In this paper, the efficient retrieval algorithm and technology of massive data information are taken as the research object, and a set of construction scheme of big data storage and retrieval system is proposed for unstructured data, which promotes the organic combination of distributed technology and full-text retrieval technology and realizes the optimization of fast retrieval processing mode of large-scale data. The system is based on Hadoop framework, with Hbase as the data storage module, and combined with ElasticSearch engine, IKAnalyzer word breaker and Redis cache to complete real-time and efficient data retrieval. Finally, based on Java web technology, a network application program convenient for users to operate online is formed. Practice has proved that the system has solved many problems in the process of collecting, storing and retrieving massive unstructured text data. At the same time, it improves the sharing transmission efficiency and concurrent access control ability of data information, and opens up a brand-new big data retrieval service model.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Pan Gao and Shuhua Shao "Research and implementation of efficient retrieval algorithm in big data environment", Proc. SPIE 12969, International Conference on Algorithm, Imaging Processing, and Machine Vision (AIPMV 2023), 129690H (9 January 2024); https://doi.org/10.1117/12.3014436

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available