Data science & ML
Neural Networks from First Principles
An MLP and a CNN without autograd, checked against PyTorch
A working MLP and CNN written without autograd, each checked gradient by gradient against PyTorch and trained to the same accuracy as PyTorch's own version.
Key result The hand-written MLP and CNN match their PyTorch twins to within half a point of test accuracy. A training-free digit-addition method scored exactly the same as its simple baseline: no gain, and the analysis explains why.
Technical lab Status: Completed Academic · team of 2
Workflow diagram
The problem
Autograd hides how a network learns. Writing every backward pass by hand, and checking each against PyTorch, shows that the mathematics is right, not only that training seems to work.
My Role & Leadership
Project Owner · Technical Lead · Developer. Owned and implemented the layers, losses and optimisers of two course assignments: linear, ReLU and softmax cross-entropy forward and backward passes, the layers, models and SGD; then convolution and max-pooling forward and backward with unfold and fold, and momentum SGD. Both assignments were done in a course team of two; this implementation is my hands-on part.
- Set the test discipline: every gradient checked against PyTorch on the same inputs before anything was trained.
The algorithm
Every operation records itself on a tape during the forward pass; the backward pass replays the tape in reverse and accumulates gradients, which are then compared with PyTorch's for the same inputs.
What the team built
- Forward and backward passes for linear, ReLU and combined softmax cross-entropy, each asserted equal to PyTorch's gradients.
- Convolution and max pooling forward and backward, matching PyTorch to about 1e-6.
- An MLP trained on MNIST and a three-block CNN trained on CIFAR-10, both compared with an identical PyTorch model.
- A training-free digit-addition method: the distribution of the sum as the convolution of the two digits' softmax outputs.
- The backward pass of a matrix-inverse layer, derived two independent ways.
Result
- MLP on MNIST, hand-written
- 97.16%
- MLP on MNIST, PyTorch twin
- 97.20%
- CNN on CIFAR-10, hand-written
- 73.42%
- CNN on CIFAR-10, PyTorch twin
- 73.84%
- Digit addition, convolution method
- 96.14%
- Digit addition, simple baseline
- 96.14%
Digit addition: the method and the baseline disagree on 0 of 10,000 pairs.
Source: Saved notebook outputs: MNIST after 8 epochs, CIFAR-10 after 5 epochs, 10,000 digit pairs.
Both hand-written networks land within half a point of their PyTorch twins (MNIST after 8 epochs, CIFAR-10 after 5), as a correct implementation should. The digit-addition method and the simple baseline, adding the two most likely digits, disagree on none of the 10,000 pairs: no gain.
Implementation decisions
- Check every gradient against PyTorch on the same inputs before training anything, so a training curve is never the only test.
- Report the digit-addition result as no gain: the classifier is almost always certain (median top-1 probability 1.0000), so combining full distributions cannot change the answer.
What was provided
- The backward-tape pattern, a module base class and the training utilities came with the assignments.
- The data loaders and the digit-pair dataset were provided.
Key technologies
Known limits
Coursework with provided scaffolding. Accuracies are single notebook runs, not tuned benchmarks.