CS 294-34: Practical Machine LearningTutorial

Ariel Kleiner

Content inspired by Fall 2006tutorial lecture by Alexandre Bouchard-Cote and Alex Simma

August 27, 2009

Machine Learning Draws Heavily On. . .

Probability and StatisticsOptimizationAlgorithms and Data Structures

Probability: Foundations

A probability space (Ω,F ,P) consists ofa set Ω of "possible outcomes"a set1 F of events, which are subsets of Ω

a probability measure P : F → [0,1] which assignsprobabilities to events in F

Example: Rolling a DieConsider rolling a fair six-sided die. In this case,

Ω = 1,2,3,4,5,6F = ∅, 1, 2, . . . , 1,2, 1,3, . . .

P(∅) = 0,P(1) =16,P(3,6) =

13, . . .

1Actually, F is a σ-field. See Durrett’s Probability: Theory and Examplesfor thorough coverage of the measure-theoretic basis for probability theory.

Probability: Random Variables

A random variable is an assignment of (often numeric)values to outcomes in Ω.For a set A in the range of a random variable X , theinduced probability that X falls in A is written as P(X ∈ A).

Example Continued: Rolling a DieSuppose that we bet $5 that our die roll will yield a 2. LetX : 1,2,3,4,5,6 → −5,5 be a random variable denotingour winnings: X = 5 if the die shows 2, and X = −5 if not.Furthermore,

P(X ∈ 5) =16

and P(X ∈ −5) =56.

Probability: Common Discrete Distributions

Common discrete distributions for a random variable X :Bernoulli(p): p ∈ [0,1]; X ∈ 0,1

P(X = 1) = p,P(X = 0) = 1− p

Binomial(p,n): p ∈ [0,1],n ∈ N; X ∈ 0, . . . ,n

P(X = x) =

(nx

)px (1− p)n−x

The multinomial distribution generalizes the Bernoulli andthe Binomial beyond binary outcomes for individualexperiments.Poisson(λ): λ ∈ (0,∞); X ∈ N

P(X = x) =e−λλx

x!

Probability: More on Random Variables

Notation: X ∼ P means "X has the distribution given by P"The cumulative distribution function (cdf) of a randomvariable X ∈ Rm is defined for x ∈ Rm as F (x) = P(X ≤ x).We say that X has a density function p if we can writeP(X ≤ x) =

∫ x−∞ p(y)dy .

In practice, the continuous random variables with which wewill work will have densities.For convenience, in the remainder of this lecture we willassume that all random variables take values in somecountable numeric set, R, or a real vector space.

Probability: Common Continuous Distributions

Common continuous distributions for a random variable X :Uniform(a,b): a,b ∈ R, a < b; X ∈ [a,b]

p(x) =1

b − a

Normal(µ, σ2): µ ∈ R, σ ∈ R++; X ∈ R

p(x) =1

σ√

2πexp

(−(x − µ)2

2σ2

)

Normal distribution can be easily generalized to themultivariate case, in which X ∈ Rm. In this context, µbecomes a real vector and σ is replaced by a covariancematrix.Beta, Gamma, and Dirichlet distributions also frequentlyarise.

Probability: DistributionsOther Distribution Types

Exponential Familyencompasses distributions of the form

P(X = x) = h(x) exp(η(θ)T (x)− A(θ))

includes many commonly encountered distributionswell-studied and has various nice analytical propertieswhile being fairly general

Graphical ModelsGraphical models provide a flexible framework for buildingcomplex models involving many random variables whileallowing us to leverage conditional independence relationshipsamong them to control computational tractability.

Probability: Expectation

Intuition: the expection of a random variable is its"average" value under its distribution.Formally, the expectation of a random variable X , denotedE [X ], is its Lebesgue integral with respect to itsdistribution.If X takes values in some countable numeric set X , then

E [X ] =∑x∈X

xP(X = x)

If X ∈ Rm has a density p, then

E [X ] =

∫Rm

xp(x)dx

Probability: More on Expectation

Expection is linear: E [aX + b] = aE [X ] + b. Also, if Y isalso a random variable, then E [X + Y ] = E [X ] + E [Y ].Expectation is monotone: if X ≥ Y , then E [X ] ≥ E [Y ].Expectations also obey various inequalities, includingJensen’s, Cauchy-Schwarz, and Chebyshev’s.

VarianceThe variance of a random variable X is defined as

Var(X ) = E [(X − E [X ])2] = E [X 2]− (E [X ])2

and obeys the following for a,b ∈ R:

Var(aX + b) = a2Var(X ).

Probability: Independence

Intuition: two random variables are independent if knowingthe value of one yields no knowledge about the value ofthe other.

Formally, two random variables X and Y are independentiff P(X ∈ A,Y ∈ B) = P(X ∈ A)P(Y ∈ B) for all(measurable) subsets A and B in the ranges of X and Y .If X ,Y have densities pX (x),pY (y), then they areindependent if pX ,Y (x , y) = pX (x)pY (y).

Probability: Conditioning

Intuition: conditioning allows us to capture the probabilisticrelationships between different random variables.For events A and B, P(A|B) is the probability that A willoccur given that we know that event B has occurred. IfP(B) > 0, then

P(A|B) =P(A ∩ B)

P(B).

In terms of densities,

p(y |x) =p(x , y)

p(x), for p(x) > 0

where p(x) =∫

p(x , y)dy .If X and Y are independent, thenP(Y = y |X = x) = P(Y = y) andP(X = x |Y = y) = P(X = x).

Probability: More on Conditional Probability

For any events A and B (e.g., we might have A = Y ≤ 5),

P(A ∩ B) = P(A|B)P(B)

Bayes’ Theorem:

P(A|B)P(B) = P(A ∩ B) = P(B ∩ A) = P(B|A)P(A)

Equivalently, if P(B) > 0,

P(A|B) =P(B|A)P(A)

P(B)

Bayes’ Theorem provides a means of inverting the "order"of conditioning.

Probability: Law of Large Numbers

Strong Law of Large NumbersLet X1,X2,X3, . . . be independent identically distributed (i.i.d.)random variables with E |Xi | <∞. Then

1n

n∑i=1

Xi → E [X1]

with probability 1 as n→∞.

Application: Monte Carlo MethodsHow can we compute an (approximation of) an expectationE [f (X )] with respect to some distribution P of X? (assume thatwe can draw independent samples from P).A Solution: Draw a large number of samples x1, . . . , xn from P.Compute E [f (X )] ≈ f (x1)+···+f (xn)

n .

Probability: Central Limit Theorem

The Central Limit Theorem provides insight into thedistribution of a normalized sum of independent randomvariables. In contrast, the law of large numbers onlyprovides a single limiting value.Intuition: The sum of a large number of small, independent,random terms is asymptotically normally distributed.This theorem is heavily used in statistics.

Central Limit TheoremLet X1,X2,X3, . . . be i.i.d. random variables with E [Xi ] = µ,Var(Xi) = σ2 ∈ (0,∞). Then, as n→∞,

1√n

n∑i=1

Xi − µσ

d−→ N(0,1)

Statistics: Frequentist Basics

Given data (i.e., realizations of random variables)x1, x2, . . . , xn which is generally assumed to be i.i.d.Based on this data, we would like to estimate some(unknown) value θ associated with the distribution fromwhich the data was generated.In general, our estimate will be a function θ(x1, . . . , xn) ofthe data (i.e., a statistic).

ExamplesGiven the results of n independent flips of a coin,determine the probability p with which it lands on heads.Simply determine whether or not the coin is fair.Find a function that distinguishes digital images of fivesfrom those of other handwritten digits.

Statistics: Parameter Estimation

In practice, we often seek to select from some class ofdistributions a single distribution corresponding to our data.If our model class is parametrized by some (possiblyuncountable) set of values, then this problem is that ofparameter estimation.That is, from a set of distributions pθ(x) : θ ∈ Θ, we willselect that corresponding to our estimate θ(x1, . . . , xn) ofthe parameter.How can we obtain estimators in general?One answer: maximize the likelihoodl(θ; x1, . . . , xn) = pθ(x1, . . . , xn) =

∏ni=1 pθ(xi) (or,

equivalently, log likelihood) of the data.

Maximum Likelihood Estimation

θ(x1, . . . , xn) = argmaxθ∈Θ

n∏i=1

pθ(xi) = argmaxθ∈Θ

n∑i=1

ln pθ(xi)

Statistics: Maximum Likelihood EstimationExample: Normal Mean

Suppose that our data is real-valued and known to bedrawn i.i.d. from a normal distribution with variance 1 butunknown mean.Goal: estimate the mean θ of the distribution.Recall that a univariate N(θ,1) distribution has densitypθ(x) = 1√

2πexp(−1

2(x − θ)2).

Given data x1, . . . , xn, we can obtain the maximumlikelihood estimate by maximizing the log likelihood w.r.t. θ:

ddθ

n∑i=1

ln pθ(xi) =n∑

i=1

ddθ

[−1

2(xi − θ)2

]=

n∑i=1

(xi − θ) = 0

⇒ θ(x1, . . . , xn) = argmaxθ∈Θ

n∑i=1

ln pθ(xi) =1n

n∑i=1

xi

Statistics: Criteria for Estimator Evaluation

Bias: B(θ) = Eθ[θ(X1, . . . ,Xn)]− θVariance: Varθ(θ(X1, . . . ,Xn)) = Eθ[(θ − Eθ[θ])2]

Loss/RiskA loss function L(θ, θ(X1, . . . ,Xn)) assigns a penalty to anestimate θ when the true value of interest is θ.The risk is the expectation of the loss function:R(θ) = Eθ[L(θ, θ(X1, . . . ,Xn))].Example: squared loss is given by L(θ, θ) = (θ − θ)2.

Bias-Variance DecompositionUnder squared loss,

Eθ[L(θ, θ)] = Eθ[(θ − θ)2] = [B(θ)]2 + Varθ(θ)

Consistency: Does θ(X1, . . . ,Xn)p−→ θ as n→∞?

Statistics: Criteria for Estimator EvaluationExample: Evaluation of Maximum Likelihood Normal Mean Estimator

Recall that, in this example, X1, . . . ,Xni.i.d .∼ N(θ,1) and the

maximum likelihood estimator for θ is

θ(X1, . . . ,Xn) =1n

n∑i=1

Xi

Therefore, we have the following:Bias:

B(θ) = Eθ[θ(X1, . . . ,Xn)]− θ = E

[1n

n∑i=1

Xi

]− θ

=1n

n∑i=1

E [Xi ]− θ =1n

n∑i=1

θ − θ = 0

Variance: Var(θ) = Var(1

n∑n

i=1 Xi)

= 1n2

∑ni=1 Var(Xi) = 1

n

Consistency: 1n∑n

i=1 Xi → E [X1] = θ with probability 1 asn→∞, by the strong law of large numbers.

Statistics: Bayesian Basics

The Bayesian approach treats statistical problems bymaintaining probability distributions over possibleparameter values.That is, we treat the parameters themselves as randomvariables having distributions:

1 We have some beliefs about our parameter values θ beforewe see any data. These beliefs are encoded in the priordistribution P(θ).

2 Treating the parameters θ as random variables, we canwrite the likelihood of the data X as a conditionalprobability: P(X |θ).

3 We would like to update our beliefs about θ based on thedata by obtaining P(θ|X ), the posterior distribution.Solution: by Bayes’ theorem,

P(θ|X ) =P(X |θ)P(θ)

P(X )

whereP(X ) =

∫P(X |θ)P(θ)dθ

Statistics: More on the Bayesian Approach

Within the Bayesian framework, estimation and predictionsimply reduce to probabilistic inference. This inferencecan, however, be analytically and computationallychallenging.It is possible to obtain point estimates from the posterior invarious ways, such as by taking the posterior mean

Eθ|X [θ] =

∫θP(θ|X )dθ

or the mode of the posterior:

argmaxθ

P(θ|X )

Alternatively, we can directly compute the predictivedistribution of a new data point Xnew, having already seendata X :

P(Xnew|X ) =

∫P(Xnew|θ)P(θ|X )dθ

Statistics: Bayesian Approach for the Normal Mean

Suppose that X |θ ∼ N(θ,1) and we place a prior N(0,1) over θ(i.e., θ ∼ N(0,1)):

P(X = x |θ) =1√2π

exp(−(x − θ)2

2

)P(θ) =

1√2π

exp(−θ

2

2

)Then, if we observe X = 1,

P(θ|X = 1) =P(X = 1|θ)P(θ)

P(X = 1)

∝ P(X = 1|θ)P(θ)

=

[1√2π

exp(−(1− θ)2

2

)][1√2π

exp(−θ

2

2

)]∝ N(0.5,0.5)

Statistics: Bayesian Prior Distributions

Important Question: How do we select our prior distribution?Different possible approaches:

based on actual prior knowledge about the system or datageneration mechanismtarget analytical and computational tractability; e.g., useconjugate priors (those which yield posterior distributionsin the same family)allow the data to have "maximal impact" on the posterior

Statistics: Parametric vs. Non-Parametric Models

All of the models considered above are parametric models,in that they are determined by a fixed, finite number ofparameters.This can limit the flexibility of the model.Instead, can permit a potentially infinite number ofparameters which is allowed to grow as we see more data.Such models are called non-parametric.Although non-parametric models yield greater modelingflexibility, they are generally statistically andcomputationally less efficient.

Statistics: Generative vs. Discriminative Models

Suppose that, based on data (x1, y1), . . . , (xn, yn), wewould like to obtain a model whereby we can predict thevalue of Y based on an always-observed random variableX .Generative Approach: model the full joint distributionP(X ,Y ), which fully characterizes the relationship betweenthe random variables.Discriminative Approach: only model the conditionaldistribution P(Y |X )

Both approaches have strengths and weaknesses and areuseful in different contexts.

Linear Algebra: Basics

Matrix TransposeFor an m × n matrix A with (A)ij = aij , its transpose is ann ×m matrix AT with (AT )ij = aji .(AB)T = BT AT

Matrix InverseThe inverse of a square matrix A ∈ Rn×n is the matrix A−1

such that A−1A = I.This notion generalizes to non-square matrices via left-and right-inverses.Not all matrices have inverses.If A and B are invertible, then (AB)−1 = B−1A−1.Computation of inverses generally requires O(n3) time.However, given a matrix A and a vector b, we can computea vector x such that Ax = b in O(n2) time.

Linear Algebra: Basics

TraceFor a square matrix A ∈ Rn×n, its trace is defined astr(A) =

∑ni=1(A)ii .

tr(AB) = tr(BA)

Eigenvectors and EigenvaluesGiven a matrix A ∈ Rn×n, u ∈ Rn\0 is called aneigenvector of A with λ ∈ R the corresponding eigenvalue if

Au = λu

An n × n matrix can have no more than n distincteigenvector/eigenvalue pairs.

Linear Algebra: Basics

More definitionsA matrix A is called symmetric if it is square and(A)ij = (A)ji ,∀i , j .A symmetric matrix A is positive semi-definite (PSD) if allof its eigenvalues are greater than or equal to 0.Changing the above inequality to >, ≤, or < yields thedefinitions of positive definite, negative semi-definite, andnegative definite matrices, respectively.A positive definite matrix is guaranteed to have an inverse.

Linear Algebra: Matrix Decompositions

Eigenvalue DecompositionAny symmetric matrix A ∈ Rn×n can be decomposed as follows:

A = UΛUT

where Λ is a diagonal matrix with the eigenvalues of A on itsdiagonal, U has the corresponding eigenvectors of A as itscolumns, and UUT = I.

Singular Value DecompositionAny matrix A ∈ Rm×n can be decomposed as follows:

A = UΣV T

where UUT = VV T = I and Σ is diagonal.

Other Decompositions: LU (into lower and upper triangularmatrices); QR; Cholesky (only for PSD matrices)

Optimization: Basics

We often seek to find optima (minima or maxima) of somereal-valued vector function f : Rn → R. For example, wemight have f (x) = xT x .Furthermore, we often constrain the value of x in someway: for example, we might require that x ≥ 0.In standard notation, we write

minx∈X

f (x)

s.t. gi(x) ≤ 0, i = 1, . . . ,Nhi(x) = 0, i = 1, . . . ,M

Every such problem has a (frequently useful)corresponding Lagrange dual problem which lower-boundsthe original, primal problem and, under certain conditions,has the same solution.It is only possible to solve these optimization problemsanalytically in special cases, though we can often findsolutions numerically.

Optimization: A Simple Example

Consider the following unconstrained optimization problem:

minx∈Rn

‖Ax − b‖22 = minx∈Rn

(Ax − b)T (Ax − b)

In fact, this is the optimization problem that we must solveto perform least-squares regression.To solve it, we can simply set the gradient of the objectivefunction equal to 0.The gradient of a function f (x) : Rn → R is the vector ofpartial derivatives with respect to the components of x :

∇x f (x) =

(∂f∂x1

, . . .∂f∂xn

)

Optimization: A Simple Example

Thus, we have

∇x‖Ax − b‖22 = ∇x

[(Ax − b)T (Ax − b)

]= ∇x

[xT AT Ax − 2xT AT b + bT b

]= 2AT Ax − 2AT b= 0

and so the solution is

x = (AT A)−1AT b

(if (AT A)−1 exists).

Optimization: Convexity

In the previous example, we were guaranteed to obtain aglobal minimum because the objective function wasconvex.A differentiable function f : Rn → R is convex if its Hessian(matrix of second derivatives) is everywhere PSD (if n = 1,then this corresponds to the second derivative beingeverywhere non-negative)2.An optimization problem is called convex if its objectivefunction f and inequality constraint functions g1, . . . ,gN areall convex, and its equality constraint functions h1, . . . ,hMare linear.For a convex problem, all minima are in fact global minima.In practice, we can efficiently compute minima for problemsin a number of large, useful classes of convex problems.

2This definition is in fact a special case of the general definition forarbitrary vector functions.