Every stalk lands in a tight, similar range. Signal variance stays roughly constant layer to layer — nothing explodes, nothing vanishes.
🌲 He — the dense forest
Taller than the wheat field, still evenly distributed
std = √(2 / fan_in) = √(2/64) ≈ 0.177
—
Why it matters
ReLU zeroes out half of every layer's output, so He compensates with roughly double Xavier's variance — trees grow taller to make up for the half that gets cut down.
🌴 Random Normal (unscaled) — the wild jungle
Some bonsai, some redwoods breaking through the canopy
std = 1.0 — not scaled to layer size at all
—
Why it matters
Ignore fan-in and fan-out and you get chaos: some weights near zero (vanishing), some huge (exploding) — the 🔥 flags mark redwoods breaking the canopy.
🏜️ Zero init — barren land
Nothing grows. Nothing can.
std = 0 — every weight starts identical
—
Why it matters
Identical weights get identical gradients — every neuron in the layer learns the exact same thing forever. Symmetry never breaks.
All four, same soil
Same seed count per field — only the formula changes