ZeRO Memory Optimizations: Toward Training Trillion Parameter Models

Failed to add items

Sorry, we are unable to add the item because your shopping basket is already at capacity.

Add to cart failed.

Please try again later

Add to wishlist failed.

Please try again later

Remove from wishlist failed.

Please try again later

Follow podcast failed

Unfollow podcast failed

ZeRO Memory Optimizations: Toward Training Trillion Parameter Models

Listen for free

View show details

About this listen

The paper introduces ZeRO, a novel approach to optimize memory usage when training massive language models. ZeRO-DP and ZeRO-R components effectively reduce memory redundancy and allow for training models with up to 170 billion parameters efficiently. The technique shows superlinear scalability, user-friendly implementation, and has the potential to democratize large model training in AI research. Read full paper: https://arxiv.org/abs/1910.02054 Tags: Systems and Performance, Deep Learning, Natural Language Processing

No reviews yet