Every activation has a simple derivative, best written in terms of its own output:

• sigmoid: s (1 - s), at most 0.25, at z = 0
• tanh: 1 - t ** 2, at most 1
• ReLU: 1 where the input is positive, 0 elsewhere

In a deep network, backpropagation multiplies one such slope per layer. Ten sigmoid layers multiply ten numbers no larger than 0.25, so the gradient reaching the first layer is at most 0.25 ** 10, about one in a million. That is the vanishing gradient problem.

import numpy as np

z = np.linspace(-4, 4, 5)
s = 1 / (1 + np.exp(-z))
t = np.tanh(z)
print(np.round(s * (1 - s), 3))
print(np.round(1 - t ** 2, 3))
print((z > 0).astype(float))
print(0.25 ** 10)

ReLU's slope is exactly 1 for active units, so it passes gradients through deep stacks unchanged. That, more than anything, is why it replaced the sigmoid in hidden layers.

Write sigmoid_grad(z), tanh_grad(z) and relu_grad(z) returning the element-wise derivatives for a NumPy array z (ReLU's slope at exactly 0 is 0). Write stacked_sigmoid_slope(z, depth): the derivative of the sigmoid applied depth times in a row, at z, as a float, by the chain rule.

The explanation opens here once every check passes.