Transformers Explained

With the release of "Attention is All You Need" paper in 2017, the translation of text from one language to another took a giant leap forward. The Neural Networks which originated with an aim to translate texts, started writting code in just few years. The addition of one layer "Attention", sparked this revolution, so I dove into the paper and summed up my understanding here.

Why was even Transformers were introduced, what did it solved?

The Transformer model was introduced to overcome the major limitations of recurrent and convolutional sequence models (RNNs and CNNs), particularly in handling long-range dependencies and achieving efficient parallelization during training.

RNNs process input tokens one step at a time, making it impossible to parallelize computations across sequence elements. This results in slow training and inference, especially for long sequences. Even with gating mechanisms like LSTMs and GRUs, they struggle to connect distant positions effectively. Their training involves backpropagation through time, which often suffers from vanishing or exploding gradients, further limiting their ability to learn long-term dependencies.

CNN-based models, while more parallelizable, require many layers or large convolutional kernels to capture long-range relationships. This increases computational cost and still fails to model sequential order dynamically, as CNNs rely on fixed-size convolutional windows rather than a true temporal process.

Transformers were introduced to eliminate this dependence on sequential computation. By relying entirely on the attention mechanism, they enable direct connections between any two tokens in a sequence, regardless of their distance, allowing for massive parallelization and more effective modeling of long-range dependencies.

How Transformers Work, The Architecture Explained

In progress...