Self-induced regularization: From linear regression to ...

Self-induced regularization:From linear regression to neural networks

Andrea Montanari

Stanford University

August 10, 2020

1 / 67

Song Mei Yiqiao Zhong

2 / 67

Supervised learning

I Data

f(yi ; x i )gi�n �iid P :

P 2 P(R� Rd) unknown.

I Want

f : Rd ! R

I Objective: Given loss ` : R� R! R�0, minimize

R(f ) := E�`(ynew; f (x new))

; (ynew; x new) � P :

3 / 67

Classical theory

1. Empirical Risk Minimization

min bRn(f ) :=1n

nXi=1

`(yi ; f (x i )) ; subj. to f 2 F :

2. Uniform convergence

supf 2F

�� bRn(f )�R(f )�� "(F ;n) :

3. Convex optimizationChoose `, F = ff ( � ;θ) : θ 2 �g so that ERM is convex

4 / 67

Why constrain f 2 F? Baby example

0.0 0.2 0.4 0.6 0.8 1.0

x

0.0

0.2

0.4

0.6

0.8y

5 / 67

Classical viewpoint

I Since �2010, none of the three pillars seems to hold anymore1.

1For many applications6 / 67

Multi-layer (fully connected) neural network

θ = (W 1;W 2; : : : ;W L)

θ 2 Θ := RN1�N0 � � � � � RNL�NL�1 ; N0 = d ;NL = 1 ;

f ( � ;θ) := W L � � �W L�1 � � � � � � �W 1 :

where

W `(x ) := W `x ;

�(x ) := (�(x1); : : : ; �(xN )) ;

Examples: �(x ) = tanh(x ), �(x ) = max(x ; 0),. . .7 / 67

Pillar #3: Convex optimization

f ( � ;θ) = W L � � � � � � � � �W 1 ;

bRn(θ) =1n

nXi=1

�yi � f (x i ;θ)

�2:

I Highly nonconvex!

8 / 67

Pillar #2: Uniform convergence

[Zhang, Bengio, Hardt, Recht, Vinyals, 2016]

I F rich enough to ‘interpolate’ data points

I Test error � Train error � 0

9 / 67

Pillar #2: Uniform convergence

I MNIST (subset): 4,000 images in 10 different classes.I 2-layers Neural Net. Square loss.

Belkin, Hsu, Ma, Mandal, 2018; Spigler, Geiger, d’Ascoli, Sagun, Biroli, Wyart, 2018;. . .10 / 67

Pillar #1: Empirical Risk Minimization

GD/SGD

θt+1 = θt � "trθbRn(θ

t )

I Nonconvex optimization

I Many global optima ( bRn(θ) � 0)

I Output depends on

I InitializationI Step-size schedule "t

I . . .

11 / 67

Can we understand all of this mathematically?

1 The big picture

2 A toy model

3 Results: The infinite width limit

4 Results: Random features model

5 Results: Neural tangent model

6 Conclusion and current directions

Ghorbani, Mei, Misiakiewicz, M, arXiv:1904.12191, 1906.08899Mei, M, arXiv:1908.05355

M, Ruan, Sohn, Yan, arXiv:1911.01544M, Zhong, arXiv:2007.12826

12 / 67

Related work

I Belkin, Rakhlin, Tsybakov, 2018

I Liang, Rakhlin, 2018

I Hastie, Montanari, Rosset, Tibshirani, 2019

I Belkin, Hsu, Xu,2019

I Bartlett, Long, Lugosi, Tsigler, 2019

I Muthukumar, Vodrahalli, Sahai, 2019

I Many papers in 2020

This work: Sharp asymptotics in the lazy regime of 2-layers nnets(random features models)

13 / 67

The big picture

14 / 67

The big picture: Classical view

Learning Empirical risk minimization

Tractability Convexity

Regularization Penalty in the cost function

15 / 67

The big picture

Learning Empirical risk minim. Algorithmic selection

Tractability Convexity Overparametrization

Regularization Penalty in the cost ‘Self-induced regularization’

16 / 67

Can we understand this rigorously?

Two-layers neural networks

FNNN �

nf (x ;a ;W ) =

NXi=1

ai �(hw i ; x i) : ai 2 R;w i 2 Rd 8i � No:

Remark

f ( � ) 2 FNNN ; � 2 R ) �f ( � ) 2 FN

NN

17 / 67

Can we understand this rigorously?

Lazy regime (linearize around a random initialization)

1"f (x ;a0 + "a ;W 0 + "W ) �

� 1"f (x ;a0;W 0) + ha ;ra f (x ;a0;W 0)i+ hW ;rW f (x ;a0;W 0)i

� 1"f (x ;a0;W 0) +

NXi=1

aiσ(hw0;i ; x i) +NX

i=1

a0;i hw i ; x iσ(hw0;i ; x i)

Jacot, Gabriel, Hongler, 2018; Du, Zhai, Poczos, Singh 2018; Allen-Zhu, Li, Song 2018; Chizat,

Bach, 2019; Ghorbani, Mei, Misiakiewicz, M, 2019; Arora, Du, Hu, Li, Salakhutdinov, Wang, 2019;

Oymak, Soltanolkotabi, 2019; . . .18 / 67

1"f (x ;a0 + "a ;W 0 + "W )

�NX

i=1

aiσ(hw0;i ; x i)| {z }FN

RF(W 0)

+NX

i=1

hb i ; x iσ(hw0;i ; x i)| {z }FN

NT(W 0)

FNRF(W ) :=

nf (x ;a) =

NXi=1

ai �(hw i ; x i) : ai 2 R 8i � No;

FNNT(W ) :=

nf (x ;a) =

NXi=1

ha i ; x i�0(hw i ; x i) : a i 2 Rd 8i � No;

W = [w1; : : : ;wN ] w i �iid Unif(Sd�1(1))

19 / 67

Training: Ridge regression

baRR(�) := arg mina2RN

(1n

nXi=1

(yi � fRF=NT(x i ;a))2 + �kak22

):

20 / 67

Why (ridgeless) ridge regression?

2 0 2 4 6 20

24

6

510152025

21 / 67

Why (ridgeless) ridge regression?

Gradient descent

bak+1 = bak � tkra bRn(bak ) ;bRn(a) := Empirical square loss :

Remark: In the overparametrized regime

limk!1

bak = lim�!0

baRR(�) :

22 / 67

Self-induced regularization: A toy model

23 / 67

Linear regression

I Data (y ;X ) = f(yi ; x i )gi�n , y 2 Rn , X 2 Rn�d

I Ridge regularization

bβ( ) := arg minβ2Rd

n 1nky �Xθk2

2 + kβk22

o;

=1d( I d + Σ̂X )�1XTy

I Σ̂X := XTX =n

24 / 67

Linear regression with a twist

I Data (y ;X ) = f(yi ; x i )gi�n

I Add noise to the covariates z i = x i + �g i , g i � N(0; I d);Z = (z i )i�n

I Ridge regularization

bβ( ;�) := arg minβ2Rd

n 1nky � Zβk2

2 + kβk22

o;

=1d( I d + Σ̂Z )

�1ZTy

Kobak, Lomond, Sanchez, 201825 / 67

Linear regression with a twist

Σ̂Z =1nZTZ

=1nXTX +

�

nXTG +

�

nGTX +

�2

nGTG

� 1nXTX + �2I d :

bβ( ;�) = 1d( I d + Σ̂Z )

�1ZTy

� 1d(( + �2)I d + Σ̂X )�1XTy

� bβ( + �2; 0)

Covariates noise � Ridge regularization

26 / 67

Self-induced regularization

Nonlinear ridgeless high-dimensional model

+

Simpler model with positive ridge regularization

The role of covariate noise is played by nonlinearity

27 / 67

Results: The infinite width limit

28 / 67

Connection with kernels

baRR(�) := arg mina2RN

(1n

nXi=1

(yi � fRF=NT(x i ;a))2 +N�

dkak2

2

);

fRF(x ;a) :=NX

i=1

ai�(hw i ; x i) ;

fNT(x ;a) :=NX

i=1

ha i ; x i�0(hw i ; x i) :

29 / 67

Function space formulation

f̂RR;� := arg minf̂ :Rd!R

(1n

nXi=1

(yi � f̂ (x i ))2 +

�

dkf̂ k2

KN

);

KRF;N (x 1; x 2) :=1N

NXi=1

�(hw i ; x 1i)�(hw i ; x 2i) ;

KNT;N (x 1; x 2) :=1N

NXi=1

hx 1; x 2i�0(hw i ; x 1i)�0(hw i ; x 2i) :

(random kernel!)

30 / 67

Connection with kernels

Very wide limit

KRF;N (x 1; x 2)! KRF(x 1; x 2) := Ew��(hw ; x 1i)�(hw ; x 2i)

KNT;N (x 1; x 2)! KNT(x 1; x 2) := hx 1; x 2iEw

��0(hw ; x 1i)�0(hw ; x 2i)

Rahimi, Recht; 2008; Bach, 2016; Daniely, Frostig, Gupta, Singer, 2017; Jacot,Gabriel, Hongler, 2018;. . .

31 / 67

Setting

I f(yi ; x i )gi�n iid

I x i � Unif(Sd�1(p

d)) or x i � N(0; I d) , d � 1

I

yi = f�(x i ) + "i ; "i � N(0; � 2) :

32 / 67

Prediction error of Kernel Ridge Regression

Theorem (Ghorbani, Mei, Misiakiewicz, M. 2019)

Assume � continuous, j�(x )j � c0 exp(c1jx j). Let ` 2 Z, and assumed `+" � n � d `+1�", " > 0. Then, for any � 2 [0; ��(�)],

RKRR(f�;�) = kP>`f�k2L2 + od(1)(kf�k2

L2 + � 2) ;

P>`f� = Projection of f� onto deg. > ` polynomials

Further, no kernel method can do better.

I Optimal error ! interpolants (� = 0)

Generalizes El Karoui, 201033 / 67

Results: Random features model

34 / 67

Random features model

FNRF(W ) :=

nf (x ;a) =

NXi=1

ai �(hw i ; x i) : ai 2 R 8i � No;


I Number of parameters: NI Number of samples: nI Degrees of freedom in the target (polynomial of degree `): d `.

Neal, 1996; Balcan, Blum, Vempala 2006; Rahimi, Recht; 2008; Bach, 201635 / 67

Proportional asymptotics

I n � d

I N � d

36 / 67

Focus on linear targets (d degrees of freedom)

I Target function

f�(x ) = hβ0; x i+ f NL� (x )

f NL� non-linear isotropic.

I kβ0k2 = F1, kf NL� kL2 = F�

I n ;N ; d !1: N=d ! 1, n=d ! 2.

I R(f̂�) � prediction error

37 / 67

Precise asymptotics

Theorem (Mei, M. 2019)

Decompose �(x ) = �0 + �1x + �NL(x ) where (for G � N(0; 1))

E[G�NL(G)] = E[�NL(G)] = 0; �2 :=�2

1E[�NL(G)2]

:

Then, for any � = �=b2� > 0

R(f̂�) = F 21 B(�; 1; 2; �) + (� 2 + F 2

� )V (�; 1; 2; �) + F 2� + od(1) ;

where B(�; 1; 2; �), V (�; 1; 2; �) are explicitly given below.

Variance computed in [Hastie, M, Rosset, Tibshirani, 2019]38 / 67

Explicit formulaeLet (�1(�); �2(�)) be the unique solution of

�1 = 1�� 2 �

�2�2

1� �2�1�2

��1

;

�2 = 2�� 1 �

�2�1

1� �2�1�2

��1

;

Let� � �1(i( 1 2�)

1=2) � �2(i( 1 2�)

1=2);

and

E0(�; 1; 2; �) � � �5�6+ 3�4

�4+ ( 1 2 � 2 � 1 + 1)�3

�6� 2�3

�4� 3�3

�2

+ ( 1 + 2 � 3 1 2 + 1)�2�4+ 2�2

�2+ �

2+ 3 1 2��

2� 1 2 ;

E1(�; 1; 2; �) � 2�3�4� 2�

2�2+ 1 2��

2� 1 2 ;

E2(�; 1; 2; �) � �5�6� 3�4

�4+ ( 1 � 1)�3

�6+ 2�3

�4+ 3�3

�2+ (� 1 � 1)�2

�4� 2�2

�2� �

2:

We then have

B(�; 1; 2; �) �E1(�; 1; 2; �)

E0(�; 1; 2; �); V (�; 1; 2; �) �

E2(�; 1; 2; �)

E0(�; 1; 2; �):

39 / 67

I Kernel inner product random matrices

Cheng, Singer, 2016; Do, Vu, 2017; Fan, M, 2017; Pennington Wohra, 2018;. . .40 / 67

What do these formulae mean?

41 / 67

‘Noisy linear features model’Nonlinear features

f̂ (x i ;a) = ha ; x i i ;uij = �(hw j ; x i i) = �1hw j ; x i i+ �NL(hw j ; x i i)

Noisy linear features

f̂a (x i ) = ha ; ~ui ;~uij = �1hw j ; x i i+ �� zij ; (zij ) �iid N(0; 1)

�� := k�NLkL2

Gaussian, correlated

42 / 67

Conceptual version of our theorem

Theorem (Mei, M, 2019)

Consider random-features ridge regression in the proportionalasymptotics

d !1; N=d ! 1; n=d ! 2:

Then the nonlinear features model and noisy linear features modelare ‘asymptotically equivalent.’

43 / 67

Simulations vs theory

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

0

0.5

1

1.5

2

2.5

3

Te

st

err

or

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Te

st

err

or

� = 0+ � > 0

44 / 67

Insigths

45 / 67

Insight #1: Optimum at N=n !1

10-4

10-2

100

102

104

106

108

1010

10-4

10-2

100

102

104

Te

st

err

or

10-4

10-2

100

102

104

106

108

1010

10-4

10-2

100

102

104

Te

st

err

or

� = 0+ � = 0+

46 / 67

Insight #2: No double descent for optimal �

10-2

100

102

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Te

st

err

or

10-2

100

102

0

0.5

1

1.5

2

2.5

3

Te

st

err

or

SNR= 5 SNR= 1=5

47 / 67

Insight #3: � = 0+ optimal at high SNR

10-3

10-2

10-1

100

101

102

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Te

st

err

or

10-3

10-2

10-1

100

101

102

0

0.5

1

1.5

2

2.5

3

Te

st

err

or

SNR= 5 SNR= 1=10

I High SNR: Minimum at � = 0+.I Low SNR: Minimum at � > 0.

48 / 67

Insight #4: Self-induced regularization

I Wide limit 1 = N=d !1, 2 = n=d <1

49 / 67


0.0 0.5 1.0 1.5 2.0 2.50.0

0.5

1.0

1.5

2.0

2.5

2

0.0 0.5 1.0 1.5 2.0 2.50.0

0.5

1.0

1.5

2.0

2.5

2

Bias Variance

50 / 67


0.0 0.5 1.0 1.5 2.0 2.50.0

0.5

1.0

1.5

2.0

2.5

2

0.0 0.5 1.0 1.5 2.0 2.50.0

0.5

1.0

1.5

2.0

2.5

2

Bias Variance

Decreasing �2 := Ef�(G)Gg2

E[�NL(G)2], Increasing �

51 / 67


I Normalized regularization

r =�+ �1

2�2 ; �2 :=

Ef�(G)Gg2

E[�NL(G)2]

I Bias, Variance

B(�;1; 2; �) =1

(1+ !)2 � !2= 2;

V (�; 1; 2; �) =!2= 2

(1+ !)2 � !2= 2;

! = !(1=r) increasing in 1=r .

52 / 67

Results: Neural tangent model

53 / 67

Neural tangent model

FNNT(W ) :=

nf (x ;a) =

NXi=1

ha i ; x i�0(hw i ; x i) : ai 2 R 8i � No;


I Number of parameters: NdI Number of samples: nI Degrees of freedom in the target (polynomial of degree `): d `.

Jacot, Gabriel, Hongler, 201854 / 67

Polynomial asymptotics

I N ;n ; d !1

I d" � N , for some " > 0

I N � Cd for some C > 0

55 / 67

The interpolation phase transition

Theorem (M, Zhong, 2020)

There exists C <1 such that, if Nd=(log d)C � n, then an NTinterpolator exists with high probability for any choice of theresponses (yi )i�n .

I Interpoaltion requires Nd � nI Recent related results:

Daniely 2020; Bubeck, Eldan, Lee, Mikulincer, 2020

56 / 67

Key technical result

Empirical kernel

K =

1

Nd

NX`=1

hx i ; x j i�0(hw `; x i i)�0(hw `; x j i)!

i ;j�N


There exists a matrix E � 0, rank(E) � N, such that

K &�v(�)� o(1)

�I n +E ; v(�) := VarG�N(0;1)(�

0(G)) :

57 / 67

Effective linear model

bβ( ) := arg minβ2Rd

n 1d

nXi=1

(yi � hβ; x i i�2

+ kβk22

o;

Rlin( ) := Exnew

�(hβ0; x newi � hbβ( ); x newi)2

�:

Very well understood!

58 / 67

Very well understood!

Rlin(�; ) = kβ�k22Blin(�; ) + �2

"Vlin(�; ); ;

When n ; d !1, n=d ! � 2 (0;1):

Blin(�; ) =12

(1� � +

p(� � 1+ )2 + 4 �

(1+ � + )p(� � 1+ )2 + 4

)+ o(1) ;

Vlin(�; ) =12

(�1+

� + + 1p(� � 1+ )2 + 4

)+ o(1) :

Hastie, M, Rosset, Tibshirani, 201959 / 67

Generalization


Assume that n � "0d for a constant "0 > 0, and that E[�0(G)] 6= 0.Recall v(�) = Var(�0(G)).If Nd=(logC (Nd)) � n, then for any � � 0,

RNT(�) = Rlin( eff(�; �)) + Od ;P�sn(log d)C

Nd

�; where

eff(�; �) :=�+ v(�)

fE[�0(G)]g2 :

In particular, ridgeless NT � linear regression with = v(�)=fE[�0(G)]g2.

60 / 67

NT vs linear regression

0 200 400 600 800 1000N

0.1

0.2

0.5

Test

erro

r

NT KRR, LRR, and theoretical prediction NT is : 0.0Lin is : 1.0theoreticalNT is : 0.05Lin is : 1.2theoreticalNT is : 0.1Lin is : 1.4theoreticalNT is : 0.3Lin is : 2.2theoreticalNT is : 0.5Lin is : 3.0theoreticalNT is : 1.0Lin is : 5.0theoretical

61 / 67

NT vs linear regression vs Random matrix theory

0 500 1000 1500 2000 2500 3000 3500 4000n

0.1

0.2

0.5

Test

erro

r

NT KRR, LRR, and theoretical prediction NT is : 0.0Lin is : 1.0theoreticalNT is : 0.05Lin is : 1.2theoreticalNT is : 0.1Lin is : 1.4theoreticalNT is : 0.3Lin is : 2.2theoreticalNT is : 0.5Lin is : 3.0theoreticalNT is : 1.0Lin is : 5.0theoretical

62 / 67

Intuitive picture

+

+

+

+

+

+

++

++

++

+

+

+

-

--

-

-

-

-

-

-

-

-

-

-

-

-

+

+

+

-

-

63 / 67

Conclusion and current directions

64 / 67

Conclusion

I Do not need to carefully trade model complexity vs sample size.

I Optimal generalization with no/minimal regularization

I Self-induced regularization

65 / 67

Open problems

I Other losses (classification)[M, Ruan, Sohn, Yan, 2019]

I Anisotropic data distributions[Ghorbani, Mei, Misiakiewicz, M, 2020]

I Sharp results for NT models

I Sharp results under polynomial scalings

66 / 67

Thanks!

67 / 67

Self-induced regularization: From linear regression to ...

Documents

Transcript of Self-induced regularization: From linear regression to ...

Reflexive Modernity’s Reflexive Actors - Beyond self-moitoring and self-interpetation

RosettaLigand Docking - Rosetta Design Grouprosettadesigngroup.org/workshops/RCW2008/presentations...Training set (Meiler & Baker 2006)receptor self cross self cross self cross self

Classi cation 3: Logistic regression (continued); model ...ryantibs/datamining/lectures/22-clas3.pdf1Kohonen (1989), \Self-Organization and Associative Memory" 18. 2.Choose a training

Statistik :: Multiple Regression Model

Testing Parametric Regression Speciﬁcations with ...cshalizi/uADA/12/lectures/ch10.pdfChapter 10 Testing Parametric Regression Speciﬁcations with Nonparametric Regression 10.1

Regularization Parameter Estimation forrosie/mypresentations/prague.pdf · Regularization Parameter Estimation for ... Rao, C. R., 1973, Linear Statistical Inference and its applications,

Simple Linear Regression

Logistic Regression - legacydirs.umiacs.umd.edulegacydirs.umiacs.umd.edu/~jbg/teaching/CSCI_3022/10c.pdf · Logistic Regression: Regression for outputting Probabilities Intuitions

SELF AND CONSCIOUSNESS - uni-muenchen.de · Self and Consciousness in Rämänuja's Perspestive John B. Chethimattam 44 ... Self as Consciousness in Classical Samkhya Gerhard Oberhammer

Multiple Linear Regression

Regression Discontinuity Design

SIMPLE REGRESSION MODEL - Yıldız Teknik Üniversitesiyildiz.edu.tr/~tastan/teaching/Handout 02 Simple Regression Model.pdf · 5 Simple Regression Model: Examples Agricultural production

TOTAL VARIATION REGULARIZATION FOR IMAGE ...wka/66261.pdfKey words. total variation, regularization, image denoising AMS subject classiﬁcations. 49Q20, 58E30 DOI. 10.1137/060662617

Regression Models

Regularization Parameter Estimation forrosie/mypapers/NotesMathPaper_v8.pdf · Regularization Parameter Estimation for Underdetermined problems by the ˜2 principle with application

Ridge regression in R

Regression inference confidence intervals

Neural Networks: Backpropagation & Regularization

Simple Linear Regression: Misc. Topics › slr_misc.pdf · Simple Linear Regression: Misc. Topics Sleuth3 Chapters 7, 8 Simple Linear Regression Model and Conditions ...

Bayesian Regression & Classiﬁcation · Bayesian Regression & Classiﬁcation learning as inference, Bayesian Kernel Ridge regression & Gaussian Processes, Bayesian Kernel Logistic