Do I need to train my own deep learning model?

Usually not. Most business use cases are served better by adapting or prompting a pre-trained model and evaluating it carefully on real cases.

What problem did residual networks solve?

Very deep networks were hard to train. Residual connections let each block learn a correction to its input, which made much deeper networks trainable.

Why did transformers replace recurrent networks for language?

Self-attention lets every word draw on every other word directly and allows parallel training, which scales far better than processing text one step at a time.

Deep learning architectures: from backpropagation to residual networks and transformers

1Introduction

Deep learning is often presented as a single breakthrough. In the literature it is a sequence of specific answers to specific obstacles: how to train many layers at once, how to remember across a sequence, how to learn from images at scale, how to go deeper without training collapsing, and how to process long text in parallel.

This review follows that sequence through primary sources and ends with what the history implies for teams building products today.

2Training layered networks

Backpropagation made it practical to adjust the weights of every layer of a network from a single error signal, so hidden layers could learn useful internal representations instead of hand-designed features [1]. This is the mechanism underneath almost every model in use today [2].

For sequences, plain recurrent networks struggled to carry information across many steps. Long short-term memory cells added gates that decide what to keep, forget and output, which allowed learning over much longer spans [3].

δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾ᵀ δ⁽ˡ⁺¹⁾) ⊙ σ′(z⁽ˡ⁾), ∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾ (a⁽ˡ⁻¹⁾)ᵀ(1)

Backpropagation: the error signal of each layer is computed from the layer above, then gives the gradient for that layer's weights.

Algorithm 1

Backpropagation for one training step

Input:: A batch of examples, the network weights and a loss function
Output:: Updated weights

1Forward pass: compute each layer's pre-activation z and activation a, from input to output.
2Compute the loss L from the output and the targets.
3At the output layer, compute the error δ as the derivative of L with respect to z.
4Moving backwards, compute each layer's δ from the layer above and the derivative of its activation.
5For every layer, the weight gradient is δ times the previous layer's activations.
6Update every weight by a small step against its gradient.

Cost: about twice the cost of the forward pass

3A worked example: learning XOR from scratch

XOR is the classic function a single linear layer cannot represent, because no straight line separates its outputs. One hidden layer trained with backpropagation learns it, which is the smallest demonstration of why hidden representations matter [1].

The listing implements the forward pass and the backward pass by hand, using the chain rule exactly as written in the equation above. The cross-entropy loss with a sigmoid output gives the simple output error p minus y.

xor_backprop.py

python

import numpy as np

rng = np.random.default_rng(0)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

W1, b1 = rng.normal(0, 1, (2, 4)), np.zeros((1, 4))
W2, b2 = rng.normal(0, 1, (4, 1)), np.zeros((1, 1))
sigmoid = lambda z: 1 / (1 + np.exp(-z))

for step in range(5000):
    # Forward pass
    h = np.tanh(X @ W1 + b1)
    p = sigmoid(h @ W2 + b2)
    # Backward pass: the chain rule, layer by layer
    d2 = (p - y) / len(X)             # output error for cross-entropy with a sigmoid
    d1 = (d2 @ W2.T) * (1 - h ** 2)   # tanh'(z) = 1 - tanh(z)^2
    W2 -= 0.5 * h.T @ d2
    b2 -= 0.5 * d2.sum(0, keepdims=True)
    W1 -= 0.5 * X.T @ d1
    b1 -= 0.5 * d1.sum(0, keepdims=True)

print(np.round(p.ravel(), 2))  # approaches [0, 1, 1, 0]

Listing 1. A two-layer network trained on XOR with hand-written backpropagation.

4Scale and depth

Convolutional networks trained on a large labelled image collection with GPU computing marked the moment deep learning overtook earlier approaches in image classification [4]. The broader account by LeCun, Bengio and Hinton frames this as representation learning: each layer turns the output of the previous one into a slightly more abstract description [5].

Going deeper, however, initially made networks harder to train. Residual connections, which let each block learn a correction added to its input, made networks with many more layers trainable and became a standard building block [6].

y = F(x, {Wᵢ}) + x(2)

A residual block learns a correction F that is added to its own input.

residual_block.py

python

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, width: int):
        super().__init__()
        self.f = nn.Sequential(nn.Linear(width, width), nn.ReLU(), nn.Linear(width, width))
        self.act = nn.ReLU()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.act(self.f(x) + x)   # y = F(x) + x

deep = nn.Sequential(*[ResidualBlock(64) for _ in range(32)])
print(deep(torch.randn(8, 64)).shape)  # torch.Size([8, 64])

Listing 2. A residual block in PyTorch: the block learns F(x) and returns F(x) + x.

5Attention replaces recurrence

The transformer removed recurrence entirely and relied on self-attention, where every position in a sequence can draw information directly from every other position [7]. Because positions are processed in parallel, transformers train efficiently on very large text collections, which is what made today's large language models possible [8] [9].

6Implications for product teams

Each of these advances depended on very large datasets and computing budgets. For a typical business, the practical question is therefore not which architecture to train, but which pre-trained model to adapt and how to evaluate it on the business's own cases.

In iBOY projects such as GreenSignal AI, the deep learning work sits inside a pre-trained language model, and the studio's effort goes into the persona inputs, the structure of the simulation and how predictions are presented and checked. This note reports design practice; it does not include a controlled comparison of training strategies.

7Conclusion

Backpropagation, gated memory, convolution at scale, residual connections and attention each removed one obstacle. Together they explain why deep learning now works well and why its benefits reach most teams through pre-trained models.

Key findings

01Backpropagation trains every layer from one error signal.
02Residual connections made very deep networks trainable.
03Attention allowed parallel training on long text and led to large language models.
04Most teams should adapt pre-trained models and invest in data and evaluation.

Questions and answers

Do I need to train my own deep learning model?: Usually not. Most business use cases are served better by adapting or prompting a pre-trained model and evaluating it carefully on real cases.
What problem did residual networks solve?: Very deep networks were hard to train. Residual connections let each block learn a correction to its input, which made much deeper networks trainable.
Why did transformers replace recurrent networks for language?: Self-attention lets every word draw on every other word directly and allows parallel training, which scales far better than processing text one step at a time.

References

[1]Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533 to 536. doi.org/10.1038/323533a0
[2]Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. www.deeplearningbook.org/
[3]Hochreiter, S., and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735 to 1780. doi.org/10.1162/neco.1997.9.8.1735
[4]Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25. papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
[5]LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature, 521, 436 to 444. doi.org/10.1038/nature14539
[6]He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition. arxiv.org/abs/1512.03385
[7]Vaswani, A., Shazeer, N., Parmar, N., and others (2017). Attention is all you need. Advances in Neural Information Processing Systems 30. arxiv.org/abs/1706.03762
[8]Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 2019. arxiv.org/abs/1810.04805
[9]Brown, T. B., Mann, B., Ryder, N., and others (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems 33. arxiv.org/abs/2005.14165

Cite this paper

Jaiganesh D (2026). Deep learning architectures: from backpropagation to residual networks and transformers. iBOY Research Notes, Paper 02. https://www.iboyhub.com/blog/deep-learning-architectures-from-backpropagation-to-residual-networks

Projects examined

Jaiganesh D

Founder of iBOY Innovation Hub, a creative technology studio from Tamil Nadu, India, established in 2025. Writes about what the studio learns while designing and building real websites, products and AI systems.