Every layer multiplies the spread of its input by some factor. Over twenty layers a factor of 2 becomes a million and a factor of 0.5 a millionth: the signal explodes or vanishes before training has started, and the gradients do the same on the way back.

import torch

torch.manual_seed(0)
for std in (1.0, 0.05, (2 / 256) ** 0.5):
    x = torch.randn(512, 256)
    for _ in range(20):
        w = torch.randn(256, 256) * std
        x = torch.relu(x @ w)
    print(f"{std:.3f} {x.std():.3g}")

He (or Kaiming) initialisation draws weights with standard deviation sqrt(2 / fan_in), where fan_in is the number of inputs each unit sums. A sum of fan_in products has fan_in times their variance, and ReLU zeroes half the signal, which the 2 makes up for.

nn.init.kaiming_normal_(w, nonlinearity="relu") applies it in place. PyTorch's default for nn.Linear is smaller, so in a deep ReLU stack set it yourself.

Write activation_stds(model, x): run x through the nn.Sequential model layer by layer without recording gradients, and return the standard deviation (a float) of the output of every nn.ReLU, in order. Write init_he(model): give every nn.Linear in model weights from nn.init.kaiming_normal_ with nonlinearity="relu" and zero biases, and return the model.

The explanation opens here once every check passes.