Data Mining and Machine Learning

Madhavan Mukund

Lecture 18, Jan–Apr 2020https://www.cmi.ac.in/~madhavan/courses/dmml2020jan/

Loss functions (costs) for neural networks

So far, we have assumed mean sum-squared error as the loss function.

Consider single neuron, two inputs x = (x1, x2)

C =1

n

n∑i=1

(yi − ai )2, where ai = σ(zi ) = σ(w1x

i1 + w2x

i2 + b)

For gradient descent, we compute∂C

∂w1,∂C

∂w2,∂C

∂bI For j = 1, 2,

∂C

∂wj=

2

n

n∑i=1

(yi − ai ) · −∂ai∂wj

=2

n

n∑i=1

(ai − yi )∂ai∂zi

∂zi∂wj

=2

n

n∑i=1

(ai − yi )σ′(zi )x

ij

I∂C

∂b=

2

n

n∑i=1

(ai − yi )∂ai∂zi

∂zi∂b

=2

n

n∑i=1

(ai − yi )σ′(zi )

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 2 / 17

Loss functions . . .

∂C

∂wj=

2

n

n∑i=1

(ai − yi )σ′(zi )x

ij ,∂C

∂b=

2

n

n∑i=1

(ai − yi )σ′(zi )

Each term in∂C

∂w1,∂C

∂w2,∂C

∂bis proportional to σ′(zi )

Ideally, gradient descent should take large steps when a− y is large

I σ(z) is flat at bothextremes

I If a is completely wrong,a ≈ (1− y), we still haveσ′(z) ≈ 0

I Learning is slow even whencurrent model is far fromoptimal

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 3 / 17

Cross entropy

A better loss function

C (a, y) =

{− ln(a), if y = 1

− ln(1− a), if y = 0

I If a ≈ y , C (a, y) ≈ 0 in both casesI If a ≈ 1− y , C (a, y)→∞ in both cases

Combine into a single equation

C (a, y) = −[y ln(a) + (1− y) ln(1− a)]

I y = 1 ⇒ second term vanishes, C = − ln(a)I y = 0 ⇒ first term vanishes, C = − ln(1− a)

This is called cross entropy

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 4 / 17

Cross entropy and gradient descent

C = −[y ln(σ(z)) + (1− y) ln(1− σ(z))]

∂C

∂wj=∂C

∂σ

∂σ

∂wj= −

[y

σ(z)− 1− y

1− σ(z)

]∂σ

∂wj

= −[

y

σ(z)− 1− y

1− σ(z)

]∂σ

∂z

∂z

∂wj

= −[

y

σ(z)− 1− y

1− σ(z)

]σ′(z)xj

= −[y(1− σ(z))− (1− y)σ(z)

σ(z)(1− σ(z))

]σ′(z)xj

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 5 / 17

Cross entropy and gradient descent . . .

∂C

∂wj= −

[y(1− σ(z))− (1− y)σ(z)

σ(z)(1− σ(z))

]σ′(z)xj

Recall that σ′(z) = σ(z)(1− σ(z))

Therefore,∂C

∂wj= −[y(1− σ(z))− (1− y)σ(z)]xj

= −[y − yσ(z)− σ(z) + yσ(z)]xj

= (σ(z)− y)xj

= (a− y)xj

Similarly,∂C

∂b= (a− y)

Thus, as we wanted, the gradient is proportional to a− y

The greater the error, the faster the learning rate

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 6 / 17

Cross entropy . . .

Overall,

I∂C

∂wj=

1

n

n∑i=1

(ai − yi )xij

I∂C

∂b=

1

n

n∑i=1

(ai − yi )

Cross entropy allows the network to learn faster when the model is farfrom the true one

Other theoretical justifications to justify using cross entropy

I Derive from goal of maximizing log-likelihood of model

I Will be addressed in advanced ML course

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 7 / 17

Case study: Handwritten digits

MNIST database has 1000 samples of handwritten digits {0,1,. . . ,9}

Assume input segmented as individual digits

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 8 / 17

Handwritten digits . . .

Each image is 28× 28 = 784 pixels

Each pixel is a grayscale value from 0.0 (white) to 1.0 (black)

Building a neural network

I Linearize the image row-wise, inputs are x1, x2, . . . , x784

I Single hidden layer, with 15 nodes

I Output layer has 10 nodes, a decision aj for each digit j ∈ {0, 1, . . . , 9}I Final output is the maximum among these

F Naıvely, arg maxj

aj

F Softmax : arg maxj

eaj∑j e

aj

Smooth approximation to arg max

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 9 / 17

Handwritten digits . . .

Neural network to recognize handwritten digits

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 10 / 17

Handwritten digits . . .

Intuitively, the internal node recognize features

Combinations of features identify digits

Hypothetically, suppose first four hidden neurons focus on fourquadrants of image.

This combination favours the verdict 0

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 11 / 17

Identifying images

Suppose we have a network that can recognize hand-written{0, 1, . . . , 9} of size 28× 28

We want to find these images in a larger image

Slide a window of size 28× 28 over the larger image

Apply the original network to each window

Pool the results to see if the digit occurs anywhere in the image

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 12 / 17

Convolutional neural network

Each “window” connects to a different node in the first hidden layer.

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 13 / 17

Convolutional neural network . . .

Sliding window performs a convolution of the window networkfunction across the entire input

I Convolutional Neural Network (CNN)

Combine these appropriately — e.g., max-pool partitions features intosmall regions, say 2× 2, and takes the maximum across each region

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 14 / 17

Convolutional neural network . . .

Example

Input is 28× 28 MNIST image

Three features, each examines a 5× 5 window

I Construct a separate hidden layer for each feature

I Each feature produces a hidden layer with 24× 24 nodes

Max-pool of size 2× 2 partitions each feature layer into 12× 12

I Second hidden layer

Finally, three max-pool layers from three hidden features arecombined into 10 indicator output nodes for {0, 1, . . . , 9}

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 15 / 17

Convolutional neural network . . .

Network structure for this example

Input layer is 28× 28

Three hidden feature layers, each 24× 24

Three max-pool layers, each 12× 12

10 output nodes

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 16 / 17

Deep learning

Hidden layers extract “features” from the input

Individual features can be combined into more complex ones

Networks with multiple hidden layers are called deep neural networks

I How deep is deep?

I Originally even 4 layers was deep.

I Recently, 100+ layers have been used

The main challenge is learning the weights

I Vanishing gradients

I Enormously large training data

Madhavan Mukund Data Mining and Machine Learning Lecture 18, Jan–Apr 2020 17 / 17