Neural Network Fundamentals

What it is

A neural network is layers of weighted sums followed by nonlinear activation functions. The forward pass computes predictions; backpropagation uses the chain rule to compute how much each weight contributed to the error, so gradient descent can update it.

Key points

• Without a nonlinear activation function (ReLU, sigmoid, tanh), stacking linear layers collapses to a single linear transform — depth would be pointless.
• ReLU is the default hidden-layer activation in most modern architectures — cheap, and avoids the vanishing-gradient problem sigmoid/tanh have at extreme values.
• Backpropagation is just the chain rule applied layer by layer, computed efficiently via reverse-mode autodiff.
• Vanishing/exploding gradients in deep networks are addressed by good initialization, normalization layers (batch/layer norm), and residual connections.