An efficient parallel optimization method for large model based on cloud computing

Zilong Xu

doi:10.1117/12.3033037

5 July 2024 An efficient parallel optimization method for large model based on cloud computing

Zilong Xu

Proceedings Volume 13184, Third International Conference on Electronic Information Engineering and Data Processing (EIEDP 2024); 131842T (2024) https://doi.org/10.1117/12.3033037
Event: 3rd International Conference on Electronic Information Engineering and Data Processing (EIEDP 2024), 2024, Kuala Lumpur, Malaysia

Abstract

Recent advancements in large-scale language models, as exemplified by ChatGPT, have undergone rapid and substantial development. Their efficacy in natural language processing (NLP) and related domains has notably surpassed that of traditional models. The pivotal role of cloud computing technology in supporting these advancements cannot be overstated. In this paper, we propose an effective and scalable 3D parallel optimization method for large-scale models, leveraging cloud computing capabilities to combine data, pipelines, and tensor slice-based parallelism, where the amalgamation of tensor slicing and pipeline parallelism operates with optimal efficiency in targeted areas. Specially, we curate a highquality natural language training corpus comprising hundreds of billions of tags and collaboratively develop training methodologies to enhance optimization efficiency and stability. Furthermore, a data parallelization strategy is proposed to expedite the training of large models, utilizing specialized hardware and software tailored for deep learning to augment training speed. To further reduce training time delays, we advocate for the use of more efficient optimization algorithms. This comprehensive approach addresses the intricate technical requisites of large language models in cloud computing, adapting to the burgeoning resource demands anticipated in future AIGC applications, thereby reinforcing the advancement of artificial intelligence. Experimentation has underscored the effectiveness and potential of our system.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Zilong Xu "An efficient parallel optimization method for large model based on cloud computing", Proc. SPIE 13184, Third International Conference on Electronic Information Engineering and Data Processing (EIEDP 2024), 131842T (5 July 2024); https://doi.org/10.1117/12.3033037

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Data modeling

Education and training

3D modeling

Mathematical optimization

Cloud computing

Systems modeling

Transformers

Show All Keywords

Keywords/Phrases

Search In:

Publication Years