The derivative of f at x is the slope of f there: how much the output changes per unit change of the input, for a very small change. Training a network is nothing more than following such slopes downhill.

You can estimate a derivative from two values of f. The forward difference (f(x + h) - f(x)) / h is the slope of a short line to the right. The central difference (f(x + h) - f(x - h)) / (2 h) looks both ways, and its error shrinks with h squared instead of h.

def f(x):
    return x ** 3

x, h = 2.0, 1e-3
forward = (f(x + h) - f(x)) / h
central = (f(x + h) - f(x - h)) / (2 * h)
print(forward - 12)  # about 6e-3
print(central - 12)  # about 1e-6

Do not make h too small: below about 1e-8, rounding in floating point swamps the difference. Around 1e-5 the central difference is accurate to many digits, which is why it is the standard way to test hand-written gradients.

Write forward_difference(f, x, h=1e-5) and central_difference(f, x, h=1e-5) returning the two estimates of the derivative of f at x as floats.

The explanation opens here once every check passes.