|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- MuzzammilShah/people-names |
|
|
language: |
|
|
- en |
|
|
model_name: Manual Backpropagation through BatchNorm |
|
|
library_name: pytorch |
|
|
tags: |
|
|
- makemore |
|
|
- backpropagation |
|
|
- batchnorm |
|
|
- neural-networks |
|
|
- andrej-karpathy |
|
|
--- |
|
|
|
|
|
# Manual Backpropagation through BatchNorm: Makemore (Part 4) |
|
|
|
|
|
This repository explores manual backpropagation through a 2-layer MLP (with BatchNorm) without using PyTorch autograd's loss.backward(). It involves manually backpropagating through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. |
|
|
|
|
|
## Documentation |
|
|
For a better reading experience and detailed notes, visit my **[Road to GPT Documentation Site](https://muzzammilshah.github.io/Road-to-GPT/Makemore-part4/)**. |
|
|
|
|
|
## Acknowledgments |
|
|
Notes and implementations inspired by the **Makemore - Part 4** video by [Andrej Karpathy](https://karpathy.ai/). |
|
|
|
|
|
For more of my projects, visit my [Portfolio Site](https://muhammedshah.com). |
