Multilayered networks learn hierarchical representations automatically from raw data, eliminating the need for hand-engineered features. End-to-end training, architectural inductive biases, and scaling effects distinguish deep learning from classical machine learning approaches.
Deep learning is the subfield of machine learning concerned with training parametric function approximators composed of many layers of learned transformations. That description is deceptively simple. What makes deep learning remarkable — and what distinguishes it sharply from the classical machine learning that preceded it — is not any single algorithm, but a thesis: that the right way to solve hard problems with data is to learn hierarchical representations automatically, rather than to engineer features by hand.
This document is a map of that thesis — where it came from, what it means, how it is realized in practice, and what the open questions are.
Classical machine learning is, at its core, a two-stage process. First, a human expert designs features — measurements, transforms, or descriptors that capture what is relevant about raw data. Second, a learning algorithm (logistic regression, an SVM, a random forest) fits a function from those features to outputs. The feature engineering stage is expensive, domain-specific, and often the binding constraint on performance.
Deep learning dissolves this boundary. A deep network takes raw data — pixels, waveforms, tokens — and learns, jointly with the predictive head, the features that are most useful. Each layer in the network computes a new representation of its input, and because layers are stacked, the network can build representations of increasing abstraction: edges give way to shapes, shapes to objects, objects to scenes. This is the hierarchical representation hypothesis, and it is the organizing idea of the field.
The representations learned by deep networks are not designed — they are discovered from data. This is both their power and the source of the deepest open questions about the field: what, precisely, do these representations capture, and why do they generalize?
Three distinctions define the gulf between classical and deep approaches:
End-to-end learning. In classical pipelines, the feature extractor and the predictor are trained separately, or not at all (fixed hand-crafted features). In deep learning, every component — from raw input to final output — is differentiable, and gradients flow through the entire system. The loss function shapes not just the final classifier but the representations on which it operates.
Inductive biases from architecture. Classical methods encode domain knowledge through features. Deep networks encode it through architecture: convolutional layers assume spatial locality and translation equivariance; recurrent networks assume temporal ordering; transformers assume pairwise relational structure. Choosing an architecture is choosing a prior over functions.
Scale as a first-class variable. Classical methods typically plateau quickly with more data; their expressiveness is fixed. Deep networks can absorb more data and more compute, and empirically their performance improves predictably as both are scaled up. This scaling behavior — now studied formally through scaling laws — is one of the most consequential empirical findings in the field.
The ideas behind neural networks are old. The perceptron was invented in 1958. Backpropagation was formalized in the 1980s. What changed was not the algorithm — it was the data and the compute.
The first wave (1950s–1980s) established the basic framework: artificial neurons, layered networks, and gradient-based training. The XOR problem revealed the limits of single-layer networks; backpropagation offered a solution for multilayer ones. But training deep networks remained brittle.
The AI winter (1990s) saw interest collapse as SVMs and kernel methods, with better theoretical guarantees and more reliable training, displaced neural approaches. Networks deeper than two or three layers were generally considered untrainable.
The deep learning renaissance (2006–2012) began with Hinton et al.'s work on deep belief networks and restricted Boltzmann machines — layer-by-layer pretraining offered a path to initializing deep networks in a useful part of parameter space. But the decisive moment was 2012: AlexNet won ImageNet by a margin that made the field impossible to ignore. The key ingredients were depth, ReLU activations, dropout, and GPUs — none individually novel, but together transformative.
The modern era (2012–present) has seen a relentless expansion: residual networks enabled training of networks hundreds of layers deep; the attention mechanism and Transformers displaced recurrent architectures for sequential data; self-supervised learning reduced dependence on labeled data; and large language models trained at previously unimaginable scale have produced systems with emergent capabilities that were not anticipated from theory.
Any deep learning system has several interlocking components. Understanding them individually is necessary; understanding how they interact is the work of this course.
The core components of any deep learning system and how they interact. Data and architecture define the function class; the loss and optimizer determine how parameters are updated; the training loop ties these together into learned parameters and intermediate representations.
Data is the substrate of learning. Deep learning is data-hungry in a qualitatively different way from classical ML: not just more labeled examples, but increasingly, unlabeled data used through self-supervised objectives. Data curation, augmentation, and the distribution from which data is drawn all profoundly shape what a model learns.
Architecture encodes prior knowledge about the structure of the problem. A convolutional network knows that nearby pixels are more related than distant ones. A transformer knows that any input token may be relevant to any other. Architecture design is one of the most consequential engineering decisions in a deep learning project.
Loss function specifies what the model should optimize. Cross-entropy loss for classification; mean squared error for regression; reconstruction loss for autoencoders; contrastive loss for representation learning. The choice of loss determines which properties of the data distribution the model is incentivized to capture.
Optimizer is the algorithm that uses gradients to update parameters. Stochastic gradient descent in its modern forms — Adam, AdaFactor, Lion — navigates the complex high-dimensional landscape of the loss function. Learning rate schedules, momentum, and adaptive step sizes all matter enormously in practice.
Regularization constrains or penalizes the model to prevent overfitting. Dropout, weight decay, batch normalization, data augmentation, and early stopping all play this role, often with theoretical motivations that are only partially understood.
Why do deep networks work? This question is harder than it sounds, and the field has not fully answered it.
The universal approximation theorem guarantees that a sufficiently wide network with a nonlinear activation can approximate any continuous function to arbitrary precision. This is reassuring but insufficient: it says nothing about whether the function can be found by gradient descent, or whether it will generalize to unseen data.
Optimization in deep networks is nonconvex, yet in practice gradient descent reliably finds solutions that work. Understanding why requires thinking about the geometry of the loss landscape — the distribution of saddle points versus local minima, the role of overparameterization, and the implicit biases that gradient descent introduces.
Generalization is the deepest open question. Classical learning theory (VC dimension, Rademacher complexity) predicts that models with more parameters than training examples should overfit catastrophically. Deep networks routinely violate this prediction and generalize well. The phenomenon of double descent — where test error decreases again after model capacity passes a critical threshold — has forced a rethinking of the classical bias-variance tradeoff.
Deep learning in 2025 is a mature engineering discipline with genuinely profound theoretical open questions. We can train language models with hundreds of billions of parameters that exhibit emergent capabilities not predicted from smaller-scale experiments. We do not have a complete theory of why they generalize, what they learn, or how their capabilities scale with compute and data — though scaling laws give us partial empirical answers.
This course treats deep learning as what it is: a coherent field with deep mathematical roots, powerful practical tools, and fundamental questions that remain contested. Your job is not just to learn the tools but to understand the theory behind them, the assumptions those theories make, and where those assumptions break down. That is the only way to be a practitioner who can reason about novel problems rather than one who merely applies recipes.
Every subsequent concept in this course is a piece of this larger picture. The architecture choices, the optimization algorithms, the regularization strategies, the representation learning objectives — they all connect back to the central question: what does a network learn, and why does it work?