For most working engineers, the word “derivative” calls up a memory of memorised rules — power rule, chain rule, the derivative of sine is cosine — and then a vague feeling that this all matters because of something to do with optimisation. The “something” is the entire engine of deep learning: every weight in every model gets updated by following derivatives. Before you can think clearly about that, you need to re-own the geometric and operational picture of what a derivative is. The viz below is that picture; the kernel below it is what happens when you ask the floating-point hardware to compute one.

Derivative as sensitivity

For a function f : ℝ → ℝ, the derivative core term The instantaneous rate of change of a function: f'(x) = lim h→0 (f(x+h) − f(x)) / h. Geometrically, the slope of the tangent line at x. Operationally, 'sensitivity' — how much f's output responds to a tiny change in x. at a point x₀ is defined as a limit:

The expression on the right is a slope: rise over run, with the run being the small number h. As h shrinks toward zero, the secant line between the points (x₀, f(x₀)) and (x₀ + h, f(x₀ + h)) approaches the unique tangent line at x₀. The derivative is that tangent’s slope.

Three operational readings of the same number, all worth having ready:

• Slope. The angle the tangent makes with the x-axis. Steep curve → big derivative.
• Sensitivity. “If I move x by a tiny amount, f(x) moves by roughly f’(x₀) · Δx.” This is the framing that matters for gradient descent.
• Linear approximation. Near x₀, f(x) ≈ f(x₀) + f’(x₀)(x − x₀). The right-hand side is a straight line; near x₀ the curve is well-approximated by it. This is what you’ll see in attention: every “Q is a linear projection of X” is, at the kernel level, a linear approximation of something — the projection just happens to be the entire thing.

Toggle to secant, slide h down, watch the secant line converge to the tangent. That’s the limit, made tactile. Try |x| at x₀ = 0 — the limit doesn’t converge (the two-sided slopes disagree), which is the geometric fact that |x| has no derivative at 0. ReLU has the same kink and the same non-derivability, and gradient-based training handles it by convention (use the left- or right-derivative arbitrarily — the cases where it bites are measure-zero).

Numerical differentiation — the U-curve

If you’re doing math, you compute derivatives analytically: d/dx sin(x) = cos(x) and so on, rule by rule. If you’re a kernel and someone hands you a black-box function, you can also approximate the derivative by evaluating f at two nearby points and taking the slope. Two estimators:

The truncation-error orders come from Taylor expansion: forward keeps a first-order error in h, central cancels it and keeps only the second-order term. So in exact arithmetic, shrinking h always improves the estimate.

In floating-point arithmetic, it doesn’t. As h shrinks, f(x+h) and f(x) become almost equal, and their difference loses bits to catastrophic cancellation numerical hazard When subtracting two nearly-equal floats, the leading bits cancel and the result is dominated by the trailing bits — which carry the rounding error from earlier operations. The relative error of the result explodes even though both inputs were precise. . Each rounded subtraction introduces error on the order of ε · |f(x)|, which is then divided by h. So the roundoff component of the error grows as 1/h. The total error is a sum:

Below those optimal h values, roundoff wins; above them, truncation wins. The classical U-curve.

The kernel sweeps h across 15 orders of magnitude and prints both estimators. The minima land exactly where theory says they should:

best forward error: 1.407e-08 at h ≈ 1e-08   (theory: h ≈ √ε   ≈ 1.5e-8)
best central error: 1.114e-11 at h ≈ 1e-05   (theory: h ≈ ε^(1/3) ≈ 6.1e-6)

The “best” choices of h for sin(x) at x = 1 match the analytical predictions to within an order of magnitude — close enough to be confident the model is right.

int main(void) {
    const double x = 1.0;
    const double exact = dtrue(x);

    printf("numerical differentiation of sin(x) at x = 1\n");
    printf("exact f'(x) = cos(1) = %.15g\n\n", exact);
    printf("%-12s %-22s %-22s %-12s\n",
           "h", "forward diff", "central diff", "best so far");

    double best_fwd = 1, best_ctr = 1;
    double best_fwd_h = 0, best_ctr_h = 0;
    for (int e = -1; e >= -15; e--) {
        double h = pow(10.0, (double)e);
        double fwd = (f(x + h) - f(x)) / h;
        double ctr = (f(x + h) - f(x - h)) / (2.0 * h);
        double err_fwd = fabs(fwd - exact);
        double err_ctr = fabs(ctr - exact);
        if (err_fwd < best_fwd) { best_fwd = err_fwd; best_fwd_h = h; }
        if (err_ctr < best_ctr) { best_ctr = err_ctr; best_ctr_h = h; }
        printf("1e%-3d        fwd %.6e        ctr %.6e\n", e, err_fwd, err_ctr);
    }
    printf("\nbest forward error: %.3e at h ≈ %.0e   (theory: h ≈ sqrt(eps) ≈ 1.5e-8)\n",
           best_fwd, best_fwd_h);
    printf("best central error: %.3e at h ≈ %.0e   (theory: h ≈ eps^(1/3) ≈ 6.1e-6)\n",

Why backprop doesn’t use numerical gradients

A neural network has on the order of 10⁸–10¹² parameters. To compute the gradient of the loss with respect to each one numerically, you would need:

• Two forward passes per parameter (one for f(θ_i + h), one for f(θ_i)) — call it ~10⁹ forward passes for a 5×10⁸-parameter model.
• The forward-difference floor of ~10⁻⁸ relative error per parameter, which is fine but you’d need a chip with 10⁹× the compute of an ordinary forward pass to actually train.

That’s clearly not how it works. Backprop forward ref The algorithm at the heart of neural-network training: compute the gradient of the loss with respect to every parameter by one forward pass and one backward pass through the computation graph, using the chain rule to combine local derivatives. Cost: ~2–3× a single forward pass. Discovered/popularised in 1986 (Rumelhart, Hinton, Williams); the modern incarnation runs on every GPU on Earth. exploits the chain rule (Ch.4 §3) to compute all derivatives in a single backward pass — total cost roughly 2–3× a forward pass, independent of parameter count. Numerical gradients survive only in two corners:

1. Gradient checking. A unit-test trick: pick a random parameter, perturb it, compare the numerical derivative against the backprop result. Catches bugs in custom autograd code that the unit-test author wrote.
2. Black-box surrogates. When the function is genuinely opaque (calling a simulator, a remote API, a non-differentiable rendering pipeline), numerical or evolutionary gradients are the only option.

For everything else — i.e., the entire content of Chapters 5–24 — we use the analytical chain rule applied to a known computation graph. The next two sections (Jacobians and the chain rule) are the algebra you need to do that.

END OF CH.4 §1 — Derivatives as sensitivity.
Built: TangentLine viz (drag a point along five functions, toggle tangent vs secant, watch the limit); numdiff.c demonstrates the U-curve in error vs h, with empirical minima matching analytical predictions; three recall items.
Coming next: §4.2 — Partials, the gradient, and the Jacobian.