Neural networks

In plain English

A neural network is a stack of simple units (neurons) connected by weights. Each unit mixes its inputs, adds a bias, and passes the result through a nonlinear activation so the whole system can curve and bend to fit complex patterns.

Early layers might detect edges in an image; deeper layers combine them into shapes and objects. For text, layers build up word and phrase patterns.

Training adjusts weights so outputs match labels. The same blueprint scales from tiny tutorials to billion-parameter language models.

How layers compute

Fully connected layers multiply input vectors by weight matrices, add biases, then apply activations (ReLU, GELU, sigmoid). Convolutional layers slide small filters across spatial data; attention layers mix tokens with learned relevance scores.

Depth adds representational power; width adds capacity per layer. Skip connections (residual nets) help gradients flow in very deep stacks.

Inference is just forward math—matrix multiplies and activations. Learning adds backward passes to update weights (see backpropagation).

import math

def relu(x):
    return max(0.0, x)

# inputs, weights, bias
x = [1.0, -2.0, 0.5]
w = [0.3, -0.1, 0.8]
b = 0.1

# weighted sum z, then activation
z = sum(xi * wi for xi, wi in zip(x, w)) + b
a = relu(z)
print("pre-activation z:", z)
print("output a:", a)

Going deeper

Architectures specialize: CNNs for grids (images), RNNs/Transformers for sequences, MLPs for tabular after embedding categorical fields.

Initialization and normalization prevent vanishing or exploding signals in deep stacks. Training stability is engineering plus theory.

Neural nets are universal approximators in theory; in practice data, compute, and inductive biases determine what actually trains.

Common misconceptions

Neural networks copy the brain accurately.

They are inspired by neurons but use simplified math optimized for GPUs and gradient descent, not biological fidelity.

More layers always help.

Depth helps when data and compute support it. Shallow nets or classical models may win on small datasets.

Key facts

• Neural networks compose layers of differentiable nonlinear transformations.
• Weights and biases are the learned parameters adjusted during training.
• Nonlinear activations let networks represent non-linear decision boundaries.
• Architectures match data structure: convolutions for images, attention for sequences.
• Forward pass computes predictions; training uses gradients to update weights.