Math for Machine Learning
Probability, linear algebra, calculus and optimization, developed as a single continuous argument.
We organize the mathematics of machine learning around one problem: given data, fit a model to it. Two instances of this problem are carried in parallel throughout: the prediction of a number (regression) and the prediction of a label (classification). We show that probability supplies the loss function for both, that linear algebra supplies the model, that calculus supplies the means of searching for its parameters, and that constrained optimization yields ridge regression, PCA and the support vector machine. No concept is introduced before the problem that requires it, and every result is derived by hand.
Modules
Probability: Where Loss Functions Come From
We develop the language of probability from the axioms through conditioning, Bayes' formula, random variables and the principal distributions. We then introduce maximum likelihood estimation and show that the loss functions used throughout machine learning are not arbitrary design choices but consequences of an assumed noise model: assuming Gaussian noise about a continuous target yields squared error, while assuming a Bernoulli outcome over a label yields cross-entropy. We thereby obtain, from a single principle, the two objectives that the remaining three modules are devoted to minimizing.
Linear Algebra: One Model, Two Readings
We represent a dataset as a matrix of row vectors and derive the dot product as a consequence of matrix multiplication rather than as a definition. We then show that the single expression wᵀx + w₀ admits two distinct readings. Interpreted as a predicted value, it gives linear regression, for which we derive the normal equations and a closed-form solution. Interpreted through its sign, it partitions space into halfspaces and gives a classifier, with its magnitude measuring distance from the decision boundary. We conclude by composing it with the sigmoid to obtain logistic regression, recovering the cross-entropy objective of Module 1.
Calculus & Optimization: Finding the Best w
We observe that the closed-form solution of Module 2 requires a matrix inversion that becomes intractable at scale and unavailable once the model is no longer linear, and that logistic regression admits no closed form at all. Motivated by this, we develop limits, continuity and differentiation, and introduce the gradient as a vector of partial derivatives indicating the direction of steepest ascent. We show that descending along its negative yields a single algorithm that solves linear regression, logistic regression and the perceptron alike, and we derive each update rule by hand from its corresponding loss. We further address the learning rate, convexity and the stochastic variant used at scale.
Constrained Optimization: Ridge, PCA and SVM
We extend the preceding development to the case in which the parameter vector is not free but subject to constraint, and present the method of Lagrange multipliers together with its geometric interpretation. We then derive three algorithms as direct consequences. Ridge regression is shown to be least squares under a budget on ‖w‖, which accounts for the shrinkage induced by regularization. Principal component analysis is formulated as the maximization of projected variance subject to ‖u‖ = 1, whose solution we show to be an eigenvector of the covariance matrix, and we apply it to the compression of high-dimensional image data. Support vector machines are obtained by maximizing the margin among all separating hyperplanes, and we treat the hard- and soft-margin formulations, the dual, and its extension to kernels.
Algorithms You'll Derive From Scratch
Probability
Maximum Likelihood
Probability
Naive Bayes
Regression
Linear Regression
Regression
Least Squares & Normal Equations
Classification
Logistic Regression
Classification
The Perceptron
Optimization
Gradient Descent
Optimization
Stochastic Gradient Descent
Constrained Optimization
Lagrange Multipliers
Constrained Optimization
Ridge Regression
Constrained Optimization
PCA
Constrained Optimization
Support Vector Machines
What you'll learn
Every Sunday, 6:00 – 8:00 PM IST · 6 September – 25 October 2026