Module 2: The Transformer Architecture: History - The Bottleneck That Broke Language Models
Failed to add items
Sorry, we are unable to add the item because your shopping basket is already at capacity.
Add to cart failed.
Please try again later
Add to wishlist failed.
Please try again later
Remove from wishlist failed.
Please try again later
Follow podcast failed
Unfollow podcast failed
Module 2: The Transformer Architecture: History - The Bottleneck That Broke Language Models
-
Narrated by:
-
Written by:
About this listen
Shay breaks down why recurrent neural networks (RNNs) struggled with long-range dependencies in language: fixed-size hidden states and the vanishing gradient caused models to forget early context in long texts.
He explains how LSTMs added gates (forget, input, output) to manage memory and improve short-term performance but remained serial, creating a training and scaling bottleneck that prevented using massive parallel compute.
The episode frames this fundamental bottleneck in NLP and sets up the next episode on attention, ending with a brief reflection on persistence and steady effort.
No reviews yet