Numerical Derivative

Numerical differentiation by finite differences

A finite-difference approximation estimates the derivative of f(x) by sampling the function at nearby points. The plainest one is the forward difference: f'(x) ≈ (f(x+h) − f(x))/h, and the backward form is its mirror image. Both carry error O(h). Step up to the central difference f'(x) ≈ (f(x+h) − f(x−h))/(2h) and the error drops to O(h²). A 5-point stencil (−f(x+2h)+8f(x+h)−8f(x−h)+f(x−2h))/(12h) pushes that down to O(h⁴). The second derivative has its own formula: f''(x) ≈ (f(x+h)−2f(x)+f(x−h))/h². Keep h small, but resist going microscopic, because once it gets too tiny round-off cancellation takes over. Automatic differentiation (autograd in PyTorch, JAX) sidesteps all of this by applying the chain rule symbolically at runtime, so the numerical errors never show up.

Applications and context

Numerical derivatives turn up across physics simulation (CFD, FEM, electromagnetics), machine learning (gradient descent, though autodiff usually takes over from FD here), economics (marginal sensitivity), engineering control and computer graphics. Even a black-box function with no known analytic derivative can be differentiated this way.

FAQ

Which h should I use? A handy rule of thumb is h ≈ √ε for forward differences and h ≈ ∛ε for the central one, where ε is machine epsilon (about 10⁻⁸ and 10⁻⁵ in double precision).

Why not just shrink h indefinitely? When you subtract two numbers that are nearly equal, you lose significant digits. Below a certain h the error starts climbing again.

When is autodiff better? Any time you have the source code for f. It returns derivatives that are exact up to round-off and handles thousands of inputs at once in reverse mode, and that's exactly why every major ML framework relies on it.