Research on real-time log analysis system based on elastic stack and Flink

Taizhi Lv; Li Zhang; Peiyi Tang

doi:10.1117/12.3004643

16 August 2023 Research on real-time log analysis system based on elastic stack and Flink

Taizhi Lv, Li Zhang, Peiyi Tang

Proceedings Volume 12787, Sixth International Conference on Advanced Electronic Materials, Computers, and Software Engineering (AEMCSE 2023); 127870X (2023) https://doi.org/10.1117/12.3004643
Event: 6th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE 2023), 2023, Shenyang, China

Abstract

The log data records the operation information of a software platform. The analysis of the log helps the manager to master the platform operation and ensure the normal operation of the platform. The rapid development of information technology has led to an explosive growth of log size. The traditional log processing architecture cannot meet the processing of massive logs. In order to process massive log data in real-time, a real-time log analysis system based on Elastic stack and Flink is designed and implemented. Based on Elastic stack, massive log data is acquired, transferred and stored in real-time. Based on Flink which is a streaming computing framework, log information is analyzed in realtime. Log analysis and statistical results are stored in MySQL database. Visualization is realized through front-end and back-end separation technology. The system has been deployed on a business platform. It can run stably against the daily average log of more than 10G, which proves the stability and reliability of the system.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Taizhi Lv, Li Zhang, and Peiyi Tang "Research on real-time log analysis system based on elastic stack and Flink", Proc. SPIE 12787, Sixth International Conference on Advanced Electronic Materials, Computers, and Software Engineering (AEMCSE 2023), 127870X (16 August 2023); https://doi.org/10.1117/12.3004643

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available