Perceptron

The simplest neural network — a single linear classifier.

Forward Pass

Given input vector $\mathbf{x} \in \mathbb{R}^n$, weight vector $\mathbf{w} \in \mathbb{R}^n$, and bias $b$:

Learning Rule

For a training sample $(\mathbf{x}, y)$ with learning rate $\eta$:

Convergence Theorem

If the training data is linearly separable with margin $\gamma = \min_i \frac{y_i(\mathbf{w}^{*\top}\mathbf{x}_i)}{\|\mathbf{w}^*\|}$, the perceptron converges in at most $\left(\frac{R}{\gamma}\right)^2$ updates, where $R = \max_i \|\mathbf{x}_i\|$.

Multi-Layer Perceptron (MLP)

Feedforward network with one or more hidden layers — a universal function approximator.

Architecture

An MLP with $L$ layers maps input $\mathbf{x}$ through a series of affine transformations and nonlinearities:

Where $f$ is a hidden activation (e.g. ReLU) and $g$ is the output activation (e.g. softmax for classification, identity for regression).

Universal Approximation Theorem

A feedforward network with a single hidden layer containing a finite number of neurons can approximate any continuous function on a compact subset of $\mathbb{R}^n$, given a non-polynomial activation function.

Loss Functions

Mean Squared Error (Regression)

Cross-Entropy (Classification)

Activation Functions

Nonlinearities that give neural networks their expressive power.

GELU (Gaussian Error Linear Unit)

Backpropagation

The chain rule applied layer-by-layer to compute gradients efficiently.

Chain Rule (Vector Form)

For loss $\mathcal{L}$ with respect to parameters in layer $l$:

Parameter Gradients

Computational Complexity

For a network with $L$ layers and $n$ neurons per layer, backpropagation has $O(Ln^2)$ time complexity — the same as the forward pass — making it highly efficient compared to numerical differentiation.

Optimization Algorithms

Methods for traversing the loss landscape to find good minima.

Stochastic Gradient Descent (SGD)

SGD with Momentum

Nesterov Accelerated Gradient

AdaGrad

RMSProp

Adam

AdamW (Decoupled Weight Decay)

Regularization Techniques

Methods to prevent overfitting and improve generalization.

L1 Regularization (Lasso)

L2 Regularization (Ridge / Weight Decay)

Dropout

During training, each neuron is independently set to zero with probability $p$:

Batch Normalization

Layer Normalization

RMSNorm

Convolutional Neural Network (CNN)

Networks exploiting spatial structure through shared local filters.

2D Convolution

For input $\mathbf{X} \in \mathbb{R}^{C_{in} \times H \times W}$ and filter $\mathbf{K} \in \mathbb{R}^{C_{in} \times k \times k}$:

Output Dimensions

Where $p$ is padding, $s$ is stride, $k$ is kernel size.

Depthwise Separable Convolution

Factorizes a standard convolution into a depthwise and pointwise step:

Dilated (Atrous) Convolution

Effective receptive field: $k + (k-1)(d-1)$, where $d$ is the dilation rate.

Pooling Operations

Transposed Convolution

Used for upsampling. Equivalent to convolving with fractional strides or padding the input:

U-Net

Encoder-decoder with skip connections for dense prediction tasks. Backbone of all modern diffusion models.

Architecture

Symmetric encoder (contracting) and decoder (expanding) path with skip connections concatenating encoder features to decoder features at each resolution:

Where $[\cdot;\cdot]$ denotes channel-wise concatenation (skip connection).

ConvBlock

U-Net in Diffusion Models

In DDPM / Stable Diffusion, the U-Net is conditioned on timestep $t$ and optional conditioning $c$:

Time embeddings are injected via addition/FiLM layers; text conditioning via cross-attention at each resolution level.

Recurrent Neural Network (Vanilla RNN)

Networks with temporal memory via recurrent connections.

Hidden State Dynamics

Backpropagation Through Time (BPTT)

Vanishing/Exploding Gradient Problem

The product of Jacobians $\prod_j \frac{\partial \mathbf{h}_j}{\partial \mathbf{h}_{j-1}}$ can shrink or grow exponentially:

Where $\gamma = \max |f'(z)|$. If $\|\mathbf{W}_{hh}\| \cdot \gamma < 1$, gradients vanish; if $> 1$, they explode.

Long Short-Term Memory (LSTM)

Gated RNN architecture solving the vanishing gradient problem.

Gate Equations

Gradient Flow through Cell State

The cell state provides a highway for gradients:

When $\mathbf{f}_t \approx 1$, gradients flow unattenuated over many timesteps.

Parameter Count

Gated Recurrent Unit (GRU)

A simplified gating mechanism merging cell and hidden state.

Gate Equations

Parameter Count

GRU uses 25% fewer parameters than LSTM (3 gates vs 4).

Extended LSTM (xLSTM)

Modernized LSTM with exponential gating and matrix memory for LLM-scale performance.

sLSTM (Scalar Memory)

Extends LSTM with exponential gating and a normalizer state for numerical stability:

mLSTM (Matrix Memory)

Replaces the scalar cell state with a matrix $\mathbf{C}_t \in \mathbb{R}^{d \times d}$, enabling key-value storage:

mLSTM is fully parallelizable (no hidden-to-hidden recurrence) and can be viewed as a linearized self-attention with a decay factor.

Bidirectional RNN

Processing sequences in both forward and backward directions.

Architecture

Attention Mechanism

Learning to focus on relevant parts of the input.

Additive (Bahdanau) Attention

Multiplicative (Luong) Attention

Scaled Dot-Product Attention

The $\sqrt{d_k}$ scaling prevents softmax saturation when dot products grow large.

Transformer

Attention-only architecture that revolutionized NLP and beyond.

Multi-Head Attention

Where $\mathbf{W}_i^Q, \mathbf{W}_i^K \in \mathbb{R}^{d \times d_k}$, $\mathbf{W}_i^V \in \mathbb{R}^{d \times d_v}$, $\mathbf{W}^O \in \mathbb{R}^{hd_v \times d}$.

Sinusoidal Positional Encoding

Rotary Positional Embedding (RoPE)

Feed-Forward Network (per position)

Encoder Block

Decoder Block (with causal mask)

The causal mask $\mathbf{M}$ sets future positions to $-\infty$ before softmax:

Grouped-Query Attention (GQA)

Shares key-value heads across groups of query heads to reduce memory:

Computational Complexity

BERT (Encoder-Only Transformer)

Bidirectional encoder pre-trained with masked language modeling — the foundation for NLU tasks.

Masked Language Modeling (MLM)

Randomly mask 15% of input tokens and predict the originals:

Of the 15% selected: 80% are replaced with [MASK], 10% with a random token, 10% unchanged.

Next Sentence Prediction (NSP)

Input Representation

Segment embeddings distinguish sentence A from B. The [CLS] token representation is used for classification tasks.

Fine-Tuning

Sequence-to-Sequence (Encoder-Decoder)

Mapping variable-length input sequences to variable-length output sequences.

RNN-Based Seq2Seq

Seq2Seq with Attention

Transformer Encoder-Decoder (T5 / BART)

Teacher Forcing

Vision Transformer (ViT)

Applying the transformer architecture directly to image patches.

Patch Embedding

An image $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$ is split into $N$ patches of size $P \times P$:

CLS Token

Full Forward Pass

Variants

RWKV (Receptance Weighted Key Value)

Linear-complexity RNN that matches transformer quality — trainable like a transformer, runs like an RNN.

Time Mixing (Attention Replacement)

WKV Mechanism (Linear Attention)

Where $w$ is a learned decay vector and $u$ is a learned bonus for the current token. This can be computed recurrently in $O(1)$ per step.

Channel Mixing (FFN Replacement)

Complexity

Autoencoder

Learning compressed representations via reconstruction.

Architecture

Denoising Autoencoder

Sparse Autoencoder

Variational Autoencoder (VAE)

Probabilistic generative model with a learned latent space.

Generative Model

Evidence Lower Bound (ELBO)

Reparameterization Trick

KL Divergence (Closed Form for Gaussians)

Loss Function

Generative Adversarial Network (GAN)

Two networks competing in a minimax game to generate realistic data.

Minimax Objective

Optimal Discriminator

Global Optimum

At the Nash equilibrium, $p_g = p_{\text{data}}$ and $D^*(\mathbf{x}) = \frac{1}{2}$:

Wasserstein GAN (WGAN)

The critic $f$ (replacing the discriminator) is enforced to be 1-Lipschitz via gradient penalty:

Diffusion Models (DDPM)

Generating data by learning to reverse a gradual noising process.

Forward Process (Diffusion)

Where $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$.

Reverse Process

Training Objective (Simplified)

Score-Based Formulation

Classifier-Free Guidance

Normalizing Flows

Exact likelihood models using invertible transformations.

Change of Variables

Composition of Flows

Coupling Layer (RealNVP)

The Jacobian is triangular, so $\det = \prod \exp(s_i) = \exp(\sum s_i)$, computed in $O(D)$.

Energy-Based Models

Defining probability distributions via scalar energy functions.

Energy Function

Score Matching

Contrastive Divergence

Where $\tilde{\mathbf{x}}$ is obtained from a few steps of MCMC starting from data.

Siamese Networks & Contrastive Learning

Learning representations by comparing pairs or groups of inputs — foundation of CLIP, SimCLR, and modern self-supervised vision.

Siamese Network

Two identical networks sharing weights process two inputs and compare their embeddings:

Contrastive Loss

Where $y=0$ for similar pairs, $y=1$ for dissimilar, and $m$ is the margin.

Triplet Loss

NT-Xent Loss (SimCLR)

Normalized temperature-scaled cross-entropy over a batch of $2N$ augmented pairs:

CLIP (Contrastive Language-Image Pre-training)

Aligns image and text embeddings using a symmetric contrastive loss over a batch of $N$ image-text pairs:

BYOL / SimSiam (No Negatives)

Where $\text{sg}(\cdot)$ is stop-gradient, $\mathbf{z}_2'$ comes from an EMA target encoder, and $q_\theta$ is a predictor MLP.

JEPA (Joint Embedding Predictive Architecture)

Yann LeCun's proposed path to human-level AI — predicting in latent space rather than pixel space.

Core Principle

Unlike generative models (which predict pixels) or contrastive models (which compare positive/negative pairs), JEPA predicts the representation of a target from a context — entirely in embedding space:

Architecture

Loss Function

Where $\text{sg}(\cdot)$ is stop-gradient. The target encoder $\bar{f}$ is updated via exponential moving average (EMA):

I-JEPA (Image JEPA)

The context encoder sees a partial view of the image (with masked patches), and the predictor must predict target block representations in latent space:

V-JEPA (Video JEPA)

Extends to video by masking spacetime tubes and predicting their latent representations:

JEPA vs Other Paradigms

Graph Neural Networks

Neural networks operating on graph-structured data.

Message Passing Framework

Graph Convolutional Network (GCN)

Where $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$ (adjacency with self-loops), $\tilde{\mathbf{D}}_{ii} = \sum_j \tilde{A}_{ij}$.

Graph Attention Network (GAT)

GraphSAGE

Graph Readout

Capsule Networks

Encoding part-whole relationships with vector-valued capsules.

Squash Function

Dynamic Routing

Margin Loss

Hopfield Network

Associative memory via energy minimization in a fully connected network.

Energy Function

Hebbian Learning (Storage)

Update Rule (Asynchronous)

Storage Capacity

Modern Hopfield Network (2020)

This update rule is equivalent to the attention mechanism in transformers.

Boltzmann Machine

Stochastic neural network based on statistical mechanics.

Energy Function

Probability Distribution

Stochastic Update

Restricted Boltzmann Machine (RBM)

A bipartite Boltzmann machine enabling efficient training via Gibbs sampling.

Energy

Conditional Distributions

Contrastive Divergence (CD-k)

Free Energy

Radial Basis Function Network

Using radial basis functions as activation in a single hidden layer.

Architecture

Training

Typically a two-phase process: (1) find centers $\boldsymbol{\mu}_j$ via k-means clustering; (2) solve for weights $\mathbf{w}$ via least squares:

Where $\Phi_{ij} = \phi_j(\mathbf{x}_i)$ is the interpolation matrix.

Self-Organizing Map (SOM)

Unsupervised learning that maps high-dimensional data to a low-dimensional grid preserving topology.

Best Matching Unit (BMU)

Weight Update

Neighborhood Function

Both $\eta(t)$ and $\sigma(t)$ decrease monotonically over training.

Residual Networks (ResNet)

Skip connections enabling training of very deep networks.

Residual Block

The network learns the residual $\mathcal{F}(\mathbf{x}) = \mathbf{y} - \mathbf{x}$ rather than the full mapping.

Bottleneck Block

$\mathbf{W}_1$ reduces channels (1×1), $\mathbf{W}_2$ is 3×3 conv, $\mathbf{W}_3$ expands channels (1×1).

Gradient Flow

The "1 +" term ensures gradients can flow directly to any layer without attenuation.

Pre-Activation ResNet

Neural Ordinary Differential Equations

Continuous-depth networks defined by differential equations.

Continuous Dynamics

Adjoint Method (Memory-Efficient Backprop)

Memory cost is $O(1)$ regardless of depth, since states are recomputed during the backward ODE solve.

Connection to ResNets

Echo State Network (Reservoir Computing)

A fixed random recurrent reservoir with only output weights trained.

Reservoir Dynamics

$\mathbf{W}_{\text{res}}$ and $\mathbf{W}_{\text{in}}$ are random and fixed. $\alpha$ is the leaking rate.

Output (Readout)

Echo State Property

The reservoir must satisfy the echo state property: the spectral radius $\rho(\mathbf{W}_{\text{res}}) < 1$ ensures that the effect of initial conditions fades over time.

Spiking Neural Network

Biologically plausible networks where neurons communicate via discrete spikes.

Leaky Integrate-and-Fire (LIF) Model

Discrete LIF

Where $\beta = \exp(-\Delta t / \tau_m)$ is the decay factor and $\Theta$ is the Heaviside step function.

Surrogate Gradient

Since $\Theta'(x) = \delta(x)$ is not useful for backprop, replace with a smooth surrogate:

Spike-Timing-Dependent Plasticity (STDP)

Kolmogorov-Arnold Network (KAN)

Learnable activation functions on edges, based on the Kolmogorov-Arnold representation theorem.

Kolmogorov-Arnold Representation Theorem

KAN Layer

Each edge $(i, j)$ has a learnable univariate function $\phi_{ij}$, parameterized by B-splines:

Layer Computation

Compared to MLPs which have fixed activations on nodes and learnable linear weights on edges, KANs have learnable nonlinear functions on edges and summation on nodes.

State Space Models (S4 / Mamba)

Sequence models based on continuous-time state space representations with efficient linear-time computation.

Continuous State Space

Discretization (Zero-Order Hold)

Discrete Recurrence

Convolution Form

Computed in $O(L \log L)$ via FFT during training.

HiPPO Initialization

Selective SSM (Mamba)

Makes parameters input-dependent for content-aware reasoning:

Hypernetworks

Networks that generate the weights of another network.

Formulation

The hypernetwork $h_\psi$ maps an embedding $\mathbf{z}$ (which can be task-specific, layer-specific, or input-dependent) to the parameters of the main network $f$.

Training

Neural Cellular Automata

Learned local update rules that produce global emergent behavior.

Cell State Update

All cells share the same neural network $f_\theta$, and the stochastic update mask enforces asynchrony for robustness.

Training via Differentiable Simulation

Gradients are backpropagated through time across the simulation steps.

Neural Turing Machine / Differentiable Neural Computer

Neural networks augmented with external differentiable memory — capable of learning algorithms.

Architecture

A controller network (LSTM or MLP) interacts with an external memory matrix $\mathbf{M} \in \mathbb{R}^{N \times M}$ via differentiable read/write heads:

Addressing — Content-Based

Addressing — Location-Based

Read & Write

$\mathbf{e}_t$ is the erase vector and $\mathbf{a}_t$ is the add vector.

Bayesian Neural Network

Placing probability distributions over weights for principled uncertainty quantification.

Bayesian Inference over Weights

Predictive Distribution

Variational Inference (Bayes by Backprop)

Approximate the intractable posterior with $q_\phi(\boldsymbol{\theta})$:

With the reparameterization trick: $\theta_i = \mu_i + \sigma_i\,\epsilon$, $\epsilon\sim\mathcal{N}(0,1)$.

MC Dropout as Approximate BNN

Running $T$ forward passes with dropout enabled at test time provides uncertainty estimates.

Liquid Neural Network

Continuous-time neural networks with input-dependent dynamics — inspired by C. elegans neuroscience.

Liquid Time-Constant (LTC) Neuron

The key insight: the time constant $\tau$ is modulated by the input, making dynamics input-dependent.

Neural Circuit Policy

Closed-Form Continuous-Depth (CfC)

An analytical solution avoiding ODE solvers:

Where $f_\infty = A\,\sigma(\mathbf{W}[\mathbf{h}_0;\,\mathbf{x}] + \mathbf{b})$ is the steady-state.

Properties

Liquid networks are remarkably compact (19 neurons can drive a car) and inherently interpretable due to their neuroscience-inspired wiring.

Mixture Density Network

Predicting full conditional probability distributions using a mixture of Gaussians.

Output Parameterization

A neural network outputs the parameters of a Gaussian mixture model:

Network Outputs

Loss (Negative Log-Likelihood)

MDNs can model one-to-many mappings (e.g., inverse kinematics, handwriting generation) where a single input maps to multiple valid outputs.

WaveNet

Autoregressive generative model with dilated causal convolutions for raw audio synthesis.

Autoregressive Formulation

Dilated Causal Convolutions

Stack convolutions with exponentially increasing dilation rates to grow the receptive field efficiently:

Gated Activation

Conditional WaveNet

Where $\mathbf{c}$ is a conditioning signal (e.g., mel spectrogram, speaker ID, linguistic features).

μ-Law Quantization

Compresses the 16-bit audio range into 256 values for categorical output via softmax.