Neural Ninjas · Deep Learning Lab
BatchNorm1d vs BatchNorm2d: Bookshelf vs Chessboard
One shelf per channel, one board per channel. Same math, different furniture.
📚 Every shelf averages its own books
(N=4, C=3, L=5) — 4 groups of 5 books per shelf
nn.BatchNorm1d(C) → dim=(0, 2) · 20 books per shelf, one average height per shelf
IdlePress "Run normalization" to watch it happen.
♟️ Every board averages all its squares
(N=4, C=3, H=3, W=3) — 4 boards of 9 squares per channel
nn.BatchNorm2d(C) → dim=(0, 2, 3) · 36 squares per channel, one average tint per channel
IdlePress "Run normalization" to watch it happen.
Run both at once
Same channel means, different furniture
♟️ Chessboard — 36/channel
| Question | 📚 BatchNorm1d | ♟️ BatchNorm2d |
|---|
| One channel is | One shelf of books | A set of boards, one per sample |
| Reduced over | dim=(0) if (N,C) · dim=(0,2) if (N,C,L) | dim=(0,2,3) — always |
| Averages | every book on the shelf | every square on every board |
| Used after | Linear, Conv1d, RNN/Transformer blocks | Conv2d layers in CNNs |