Numeric Function Derivative

Numerical derivative of a function

A numerical derivative estimates f'(x) from sampled function values, which is what you reach for when there's no analytic form or it costs too much to compute. The forward difference f'(x) ≈ (f(x+h) − f(x))/h has error O(h). The central difference f'(x) ≈ (f(x+h) − f(x−h))/(2h) brings that down to O(h²), and a 5-point stencil (−f(x+2h)+8f(x+h)−8f(x−h)+f(x−2h))/(12h) reaches O(h⁴). Take f(x) = x² at x = 2 with h = 0.01: central gives (4.0401 − 3.9601)/0.02 = 4.0000, which matches the exact 2x = 4. Pick h around √ε for forward or ∛ε for central, with ε being machine epsilon. Go too small and subtractive cancellation wrecks your precision.

Applications and context

You'll find numerical derivatives behind finite-element analysis (FEM), numerical ODE/PDE solvers, optimization without analytic gradients, signal processing (rate-of-change detection) and economic sensitivity studies. In ML the story is different. Automatic differentiation (PyTorch autograd, JAX, TensorFlow) usually wins over finite differences, since it hands back gradients exact up to round-off and scales to millions of parameters.

FAQ

Forward vs central, which one? Go central whenever you can evaluate both sides of x. That one extra call buys you a whole order of accuracy.

Can I just use h = 10⁻¹⁵? No. When you subtract two near-equal floats you throw away significant digits, and the total error traces a U-shape with one optimal h sitting at the bottom.

How does autodiff differ? It walks the chain rule symbolically through your source code, so you get exact derivatives and never have to tune an h.