On the number of variables to use in principal component regressionJi Xu, Daniel Hsu

Computer Science Department and Data Science Institute, Columbia University

Principal Component Regression

Model: Suppose the data consists of n i.i.d. observations (x1, y1), . . . ,(xn, yn)from RN × R, where

yi = x>i θ + wi,

andxi ∼ N (0,Σ), wi ∼ N (0, σ2).

Principal component regression: Let λ1 ≥ . . . ≥ λp be the eigenvalues of Σ.

Let v1, v2, . . . , vp be the corresponding eigenvectors. The PCR estimator θ̂ for θis defined by (minimum `2 norm solution for p > n regime)

θ̂P :=

{(X>

PXP)−1X>Py if p ≤ n,

X>P(XPX

>P)−1y if p > n,

where XP = [x1| · · · |xn]>[v1| . . . |vp]. The prediction error is given by

Errorp := Ex,y[(y − x>θ̂P)2].

Question: What is the optimal value of p that minimizes the prediction error?

Double descent phenomenon

• First descent: classic U-shaped risk curve arising from a bias-variance trade-off.

• Second descent: behavior of models in H that interpolate training data.

• This particular shape is observed in many learning problems, such as neural networks,decision trees and ensemble methods.

Figure: [BHMM19] (a) The classical U-shaped risk curve arising from the bias-variance trade-off. (b)The double descent risk curve including both the U-shaped risk curve and the observed behavior fromusing high capacity function classes.

For principal component regression (PCR):

Assumption: We assume Eθ[θ] = 0 and Eθ[θθ>] = I.

Question: Does double descent phenomenon happen in PCR?

Question: When the second descent achieve error smaller than the first descent?

Case of Polynomial Decay

We first analyze a special case when the eigenvalues of Σ decay to zero at a polynomial rate.Specifically, we assume

A.1 There exists a constant κ > 0 such that λj = j−κ for all j = 1, . . . , N .

A.2 There exist constants α ∈ [0, 1] and β ∈ (0, 1) such that p/N → α andn/N → β as p, n,N →∞.

Define mκ(z) for z ≤ 0 to be the smallest positive solution to the equation

−z =1

mκ(z)−

1

β

∫ ∞α−κ

1

κt1/κ(1 + t ·mκ(z))dt, (1)

and let m′κ(·) denote the derivative of mκ(·).

Remark: mκ(z) is the Stieltjes transform of the limiting distribution of the empiricaldistribution of the eigenvalues of NκΣ.

Theorem 1. Assume A.1 with constant κ and A.2 with constants α and β.

(i) Risk characterization at α < β: For all α < β, we have

Ew,θ[Error]p→(N1−κ

∫ 1

α

t−κ dt+ σ2

)·

β

β − α=: Rκ(α, σ), ∀α < β.

(ii) Optimal risk at α < β: When κ > 1, the minimum of Rκ(α, σ) is achieved atα = 0 and the minimum risk is given by

minα<βRκ(α, σ) = σ2.

When κ ≤ 1, the minimum of Rκ(α, σ) is achieved at α∗ which is the uniquesolution of the equation hκ(α) = 0 on (0, β), where hκ(α) is given by

hκ(α) :=β

α−∫ 1

α

tκ−2 dt− 1− σ21{κ = 1}.

The minimum risk is therefore given by

minα<βRκ(α, σ) = N1−κ β

(α∗)κ.

(iii) Risk characterization at α > β: For all α > β, the function mκ defined in (1) and itsderivative m′κ are well-defined and positive at z = 0, and

Ew,θ[Error]p→ N1−κ β

mκ(0)+

(N1−κ

∫ 1

α

t−κ dt+ σ2

)m′κ(0)

m2κ(0)

=: Rκ(α, σ).

(iv) Comparison between two regimes: When κ > 1, the minimum risk for all α < 1 andα 6= β is achieved at α = 0, i.e., p = o(n). When κ < 1, let α∗ be the minimizerof Rκ(α, σ) over the interval [0, β). Then

lim supN

Rκ(1, σ)

Rκ(α∗, σ)< 1.

Case of General Decay

Results from Theorem 1 can be extended to the eigenvalues of Σ with otherdecay rate when the following assumptions hold:

B.1 ‖Σ‖2 ≤ C for some constant C > 0. Also, there exists a positivesequence (cN)N≥1 such that the empirical eigenvalue distribution of cNΣconverges as N →∞ to F = (1− δ)F0 + δF1, where δ ∈ (0, 1], F0

is a point mass of 0, and F1 has a continuous probability density fsupported on either [η1, η2] or [η1,∞) for some constants η1, η2 > 0.

B.2 There exist constants ν > 0 and β ∈ (0, δ) s.t. p =∑Ni=1 I(λi ≥ νN),

νNcN → ν and n/N → β as n,N →∞.

Remark: B.1 is the extension of A.1 with cN = Nκ. B.2 is the extension ofA.2 where νN is the threshold parameter that determines the number of selectedprincipal components.

Summaries and Discussions

• We confirm the “double descent” in a natural setting with Gaussian design.

• Optimal p depends on noise level and decay rate of the eigenvalues of Σ.

◦ When κ < 1, a smaller risk is achieved after the interpolation threshold(p > n) than any point before (p < n).

◦ When κ > 1, a smaller risk is achieved after the interpolation threshold(p > n) only in the noiseless setting.

• When Σ is unknown

◦ Estimate Σ via unlabeled data.

◦ Since the dominance of the p > n regime is always established at p = N(full model), we believe same results hold for standard PCR as well.

R(↵)<latexit sha1_base64="G3+rf6RqL6ugJ3aBEV4KtdSPiRA=">AAACBXicbVBNS8NAEN34WetX1KMegkWol5KIoMeiF49V7Ac0IUy2m3bpJll2N0IJuXjxr3jxoIhX/4M3/42bNgdtfTDweG+GmXkBZ1Qq2/42lpZXVtfWKxvVza3tnV1zb78jk1Rg0sYJS0QvAEkYjUlbUcVIjwsCUcBINxhfF373gQhJk/heTTjxIhjGNKQYlJZ888iNQI0wsOwu9zN3DJxDXneB8RGc+mbNbthTWIvEKUkNlWj55pc7SHAakVhhBlL2HZsrLwOhKGYkr7qpJBzwGIakr2kMEZFeNv0it060MrDCROiKlTVVf09kEEk5iQLdWdws571C/M/rpyq89DIa81SRGM8WhSmzVGIVkVgDKghWbKIJYEH1rRYegQCsdHBVHYIz//Ii6Zw1HLvh3J7XmldlHBV0iI5RHTnoAjXRDWqhNsLoET2jV/RmPBkvxrvxMWtdMsqZA/QHxucPwoGYtg==</latexit><latexit sha1_base64="G3+rf6RqL6ugJ3aBEV4KtdSPiRA=">AAACBXicbVBNS8NAEN34WetX1KMegkWol5KIoMeiF49V7Ac0IUy2m3bpJll2N0IJuXjxr3jxoIhX/4M3/42bNgdtfTDweG+GmXkBZ1Qq2/42lpZXVtfWKxvVza3tnV1zb78jk1Rg0sYJS0QvAEkYjUlbUcVIjwsCUcBINxhfF373gQhJk/heTTjxIhjGNKQYlJZ888iNQI0wsOwu9zN3DJxDXneB8RGc+mbNbthTWIvEKUkNlWj55pc7SHAakVhhBlL2HZsrLwOhKGYkr7qpJBzwGIakr2kMEZFeNv0it060MrDCROiKlTVVf09kEEk5iQLdWdws571C/M/rpyq89DIa81SRGM8WhSmzVGIVkVgDKghWbKIJYEH1rRYegQCsdHBVHYIz//Ii6Zw1HLvh3J7XmldlHBV0iI5RHTnoAjXRDWqhNsLoET2jV/RmPBkvxrvxMWtdMsqZA/QHxucPwoGYtg==</latexit><latexit sha1_base64="G3+rf6RqL6ugJ3aBEV4KtdSPiRA=">AAACBXicbVBNS8NAEN34WetX1KMegkWol5KIoMeiF49V7Ac0IUy2m3bpJll2N0IJuXjxr3jxoIhX/4M3/42bNgdtfTDweG+GmXkBZ1Qq2/42lpZXVtfWKxvVza3tnV1zb78jk1Rg0sYJS0QvAEkYjUlbUcVIjwsCUcBINxhfF373gQhJk/heTTjxIhjGNKQYlJZ888iNQI0wsOwu9zN3DJxDXneB8RGc+mbNbthTWIvEKUkNlWj55pc7SHAakVhhBlL2HZsrLwOhKGYkr7qpJBzwGIakr2kMEZFeNv0it060MrDCROiKlTVVf09kEEk5iQLdWdws571C/M/rpyq89DIa81SRGM8WhSmzVGIVkVgDKghWbKIJYEH1rRYegQCsdHBVHYIz//Ii6Zw1HLvh3J7XmldlHBV0iI5RHTnoAjXRDWqhNsLoET2jV/RmPBkvxrvxMWtdMsqZA/QHxucPwoGYtg==</latexit><latexit sha1_base64="G3+rf6RqL6ugJ3aBEV4KtdSPiRA=">AAACBXicbVBNS8NAEN34WetX1KMegkWol5KIoMeiF49V7Ac0IUy2m3bpJll2N0IJuXjxr3jxoIhX/4M3/42bNgdtfTDweG+GmXkBZ1Qq2/42lpZXVtfWKxvVza3tnV1zb78jk1Rg0sYJS0QvAEkYjUlbUcVIjwsCUcBINxhfF373gQhJk/heTTjxIhjGNKQYlJZ888iNQI0wsOwu9zN3DJxDXneB8RGc+mbNbthTWIvEKUkNlWj55pc7SHAakVhhBlL2HZsrLwOhKGYkr7qpJBzwGIakr2kMEZFeNv0it060MrDCROiKlTVVf09kEEk5iQLdWdws571C/M/rpyq89DIa81SRGM8WhSmzVGIVkVgDKghWbKIJYEH1rRYegQCsdHBVHYIz//Ii6Zw1HLvh3J7XmldlHBV0iI5RHTnoAjXRDWqhNsLoET2jV/RmPBkvxrvxMWtdMsqZA/QHxucPwoGYtg==</latexit>

Ew,✓⇤ Error<latexit sha1_base64="b+Ytr3XWSnr8N38ZMyOL/kC6yrU=">AAACD3icbVDLSsNAFJ34rPUVdekmWBQRKYkIuixKwWUF+4Amhsl02g6dPJi5UUqIX+DGX3HjQhG3bt35N07SLLT1wIXDOfdy7z1exJkE0/zW5uYXFpeWSyvl1bX1jU19a7slw1gQ2iQhD0XHw5JyFtAmMOC0EwmKfY/Ttje6zPz2HRWShcENjCPq+HgQsD4jGJTk6ge2j2HoeUk9dZP7YxuGFPDtUfqQ68JP6kKEInX1ilk1cxizxCpIBRVouPqX3QtJ7NMACMdSdi0zAifBAhjhNC3bsaQRJiM8oF1FA+xT6ST5P6mxr5Se0Q+FqgCMXP09kWBfyrHvqc7sSjntZeJ/XjeG/rmTsCCKgQZksqgfcwNCIwvH6DFBCfCxIpgIpm41yBALTEBFWFYhWNMvz5LWSdUyq9b1aaV2UcRRQrtoDx0iC52hGrpCDdREBD2iZ/SK3rQn7UV71z4mrXNaMbOD/kD7/AFrvZ2H</latexit><latexit sha1_base64="b+Ytr3XWSnr8N38ZMyOL/kC6yrU=">AAACD3icbVDLSsNAFJ34rPUVdekmWBQRKYkIuixKwWUF+4Amhsl02g6dPJi5UUqIX+DGX3HjQhG3bt35N07SLLT1wIXDOfdy7z1exJkE0/zW5uYXFpeWSyvl1bX1jU19a7slw1gQ2iQhD0XHw5JyFtAmMOC0EwmKfY/Ttje6zPz2HRWShcENjCPq+HgQsD4jGJTk6ge2j2HoeUk9dZP7YxuGFPDtUfqQ68JP6kKEInX1ilk1cxizxCpIBRVouPqX3QtJ7NMACMdSdi0zAifBAhjhNC3bsaQRJiM8oF1FA+xT6ST5P6mxr5Se0Q+FqgCMXP09kWBfyrHvqc7sSjntZeJ/XjeG/rmTsCCKgQZksqgfcwNCIwvH6DFBCfCxIpgIpm41yBALTEBFWFYhWNMvz5LWSdUyq9b1aaV2UcRRQrtoDx0iC52hGrpCDdREBD2iZ/SK3rQn7UV71z4mrXNaMbOD/kD7/AFrvZ2H</latexit><latexit sha1_base64="b+Ytr3XWSnr8N38ZMyOL/kC6yrU=">AAACD3icbVDLSsNAFJ34rPUVdekmWBQRKYkIuixKwWUF+4Amhsl02g6dPJi5UUqIX+DGX3HjQhG3bt35N07SLLT1wIXDOfdy7z1exJkE0/zW5uYXFpeWSyvl1bX1jU19a7slw1gQ2iQhD0XHw5JyFtAmMOC0EwmKfY/Ttje6zPz2HRWShcENjCPq+HgQsD4jGJTk6ge2j2HoeUk9dZP7YxuGFPDtUfqQ68JP6kKEInX1ilk1cxizxCpIBRVouPqX3QtJ7NMACMdSdi0zAifBAhjhNC3bsaQRJiM8oF1FA+xT6ST5P6mxr5Se0Q+FqgCMXP09kWBfyrHvqc7sSjntZeJ/XjeG/rmTsCCKgQZksqgfcwNCIwvH6DFBCfCxIpgIpm41yBALTEBFWFYhWNMvz5LWSdUyq9b1aaV2UcRRQrtoDx0iC52hGrpCDdREBD2iZ/SK3rQn7UV71z4mrXNaMbOD/kD7/AFrvZ2H</latexit><latexit sha1_base64="b+Ytr3XWSnr8N38ZMyOL/kC6yrU=">AAACD3icbVDLSsNAFJ34rPUVdekmWBQRKYkIuixKwWUF+4Amhsl02g6dPJi5UUqIX+DGX3HjQhG3bt35N07SLLT1wIXDOfdy7z1exJkE0/zW5uYXFpeWSyvl1bX1jU19a7slw1gQ2iQhD0XHw5JyFtAmMOC0EwmKfY/Ttje6zPz2HRWShcENjCPq+HgQsD4jGJTk6ge2j2HoeUk9dZP7YxuGFPDtUfqQ68JP6kKEInX1ilk1cxizxCpIBRVouPqX3QtJ7NMACMdSdi0zAifBAhjhNC3bsaQRJiM8oF1FA+xT6ST5P6mxr5Se0Q+FqgCMXP09kWBfyrHvqc7sSjntZeJ/XjeG/rmTsCCKgQZksqgfcwNCIwvH6DFBCfCxIpgIpm41yBALTEBFWFYhWNMvz5LWSdUyq9b1aaV2UcRRQrtoDx0iC52hGrpCDdREBD2iZ/SK3rQn7UV71z4mrXNaMbOD/kD7/AFrvZ2H</latexit>

0.0 0.2 0.4 0.6 0.8 1.0

05

1015

p/N

risk

E ErrorR(alpha)

0.0 0.2 0.4 0.6 0.8 1.0

02

46

810

↵ = p/N<latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit><latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit><latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit><latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit>

risk

= 1<latexit sha1_base64="A0YchN0cVRV7/kKcxtDLjLttdgA=">AAAB73icbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0EYI2lhGMDGQHGFus0mW7O2tu3tCOPInbCwUsfXv2Plv3CRXaOKDgcd7M8zMi5Tgxvr+t1dYWV1b3yhulra2d3b3yvsHTZOkmrIGTUSiWxEaJrhkDcutYC2lGcaRYA/R6GbqPzwxbXgi7+1YsTDGgeR9TtE6qdUZoVJ4FXTLFb/qz0CWSZCTCuSod8tfnV5C05hJSwUa0w58ZcMMteVUsEmpkxqmkI5wwNqOSoyZCbPZvRNy4pQe6SfalbRkpv6eyDA2ZhxHrjNGOzSL3lT8z2untn8ZZlyq1DJJ54v6qSA2IdPnSY9rRq0YO4JUc3croUPUSK2LqORCCBZfXibNs2rgV4O780rtOo+jCEdwDKcQwAXU4Bbq0AAKAp7hFd68R+/Fe/c+5q0FL585hD/wPn8Ai+CPoQ==</latexit><latexit sha1_base64="A0YchN0cVRV7/kKcxtDLjLttdgA=">AAAB73icbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0EYI2lhGMDGQHGFus0mW7O2tu3tCOPInbCwUsfXv2Plv3CRXaOKDgcd7M8zMi5Tgxvr+t1dYWV1b3yhulra2d3b3yvsHTZOkmrIGTUSiWxEaJrhkDcutYC2lGcaRYA/R6GbqPzwxbXgi7+1YsTDGgeR9TtE6qdUZoVJ4FXTLFb/qz0CWSZCTCuSod8tfnV5C05hJSwUa0w58ZcMMteVUsEmpkxqmkI5wwNqOSoyZCbPZvRNy4pQe6SfalbRkpv6eyDA2ZhxHrjNGOzSL3lT8z2untn8ZZlyq1DJJ54v6qSA2IdPnSY9rRq0YO4JUc3croUPUSK2LqORCCBZfXibNs2rgV4O780rtOo+jCEdwDKcQwAXU4Bbq0AAKAp7hFd68R+/Fe/c+5q0FL585hD/wPn8Ai+CPoQ==</latexit><latexit sha1_base64="A0YchN0cVRV7/kKcxtDLjLttdgA=">AAAB73icbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0EYI2lhGMDGQHGFus0mW7O2tu3tCOPInbCwUsfXv2Plv3CRXaOKDgcd7M8zMi5Tgxvr+t1dYWV1b3yhulra2d3b3yvsHTZOkmrIGTUSiWxEaJrhkDcutYC2lGcaRYA/R6GbqPzwxbXgi7+1YsTDGgeR9TtE6qdUZoVJ4FXTLFb/qz0CWSZCTCuSod8tfnV5C05hJSwUa0w58ZcMMteVUsEmpkxqmkI5wwNqOSoyZCbPZvRNy4pQe6SfalbRkpv6eyDA2ZhxHrjNGOzSL3lT8z2untn8ZZlyq1DJJ54v6qSA2IdPnSY9rRq0YO4JUc3croUPUSK2LqORCCBZfXibNs2rgV4O780rtOo+jCEdwDKcQwAXU4Bbq0AAKAp7hFd68R+/Fe/c+5q0FL585hD/wPn8Ai+CPoQ==</latexit><latexit sha1_base64="A0YchN0cVRV7/kKcxtDLjLttdgA=">AAAB73icbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0EYI2lhGMDGQHGFus0mW7O2tu3tCOPInbCwUsfXv2Plv3CRXaOKDgcd7M8zMi5Tgxvr+t1dYWV1b3yhulra2d3b3yvsHTZOkmrIGTUSiWxEaJrhkDcutYC2lGcaRYA/R6GbqPzwxbXgi7+1YsTDGgeR9TtE6qdUZoVJ4FXTLFb/qz0CWSZCTCuSod8tfnV5C05hJSwUa0w58ZcMMteVUsEmpkxqmkI5wwNqOSoyZCbPZvRNy4pQe6SfalbRkpv6eyDA2ZhxHrjNGOzSL3lT8z2untn8ZZlyq1DJJ54v6qSA2IdPnSY9rRq0YO4JUc3croUPUSK2LqORCCBZfXibNs2rgV4O780rtOo+jCEdwDKcQwAXU4Bbq0AAKAp7hFd68R+/Fe/c+5q0FL585hD/wPn8Ai+CPoQ==</latexit>

0.0 0.2 0.4 0.6 0.8 1.0

0.00

60.

008

0.01

00.

012 E Error

R(alpha)

0.0 0.2 0.4 0.6 0.8 1.0

0.006

0.008

0.010

0.012

↵ = p/N<latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit><latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit><latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit><latexit sha1_base64="dadLJSVMX+K2Z5x9P6fR+QzKsK0=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSYi6EUoevEkFewHtqFMtpt26Waz7G6EEvovvHhQxKv/xpv/xm2bg1YfDDzem2FmXig508bzvpzC0vLK6lpxvbSxubW9U97da+okVYQ2SMIT1Q5RU84EbRhmOG1LRTEOOW2Fo+up33qkSrNE3JuxpEGMA8EiRtBY6aGLXA7xUp7c9soVr+rN4P4lfk4qkKPeK392+wlJYyoM4ah1x/ekCTJUhhFOJ6VuqqlEMsIB7VgqMKY6yGYXT9wjq/TdKFG2hHFn6s+JDGOtx3FoO2M0Q73oTcX/vE5qoosgY0KmhgoyXxSl3DWJO33f7TNFieFjS5AoZm91yRAVEmNDKtkQ/MWX/5LmadX3qv7dWaV2lcdRhAM4hGPw4RxqcAN1aAABAU/wAq+Odp6dN+d93lpw8pl9+AXn4xvuWZBq</latexit>

risk

= 2<latexit sha1_base64="YL1ePw28IevvBzwsoV26KDEGt2I=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoBeh6MVjBfuBbSiT7aZdutmE3Y1QQv+FFw+KePXfePPfuG1z0NYHA4/3ZpiZFySCa+O6305hbX1jc6u4XdrZ3ds/KB8etXScKsqaNBax6gSomeCSNQ03gnUSxTAKBGsH49uZ335iSvNYPphJwvwIh5KHnKKx0mNvjEmC5JrU+uWKW3XnIKvEy0kFcjT65a/eIKZpxKShArXuem5i/AyV4VSwaamXapYgHeOQdS2VGDHtZ/OLp+TMKgMSxsqWNGSu/p7IMNJ6EgW2M0Iz0sveTPzP66YmvPIzLpPUMEkXi8JUEBOT2ftkwBWjRkwsQaq4vZXQESqkxoZUsiF4yy+vklat6rlV7/6iUr/J4yjCCZzCOXhwCXW4gwY0gYKEZ3iFN0c7L86787FoLTj5zDH8gfP5Az3oj/Y=</latexit><latexit sha1_base64="YL1ePw28IevvBzwsoV26KDEGt2I=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoBeh6MVjBfuBbSiT7aZdutmE3Y1QQv+FFw+KePXfePPfuG1z0NYHA4/3ZpiZFySCa+O6305hbX1jc6u4XdrZ3ds/KB8etXScKsqaNBax6gSomeCSNQ03gnUSxTAKBGsH49uZ335iSvNYPphJwvwIh5KHnKKx0mNvjEmC5JrU+uWKW3XnIKvEy0kFcjT65a/eIKZpxKShArXuem5i/AyV4VSwaamXapYgHeOQdS2VGDHtZ/OLp+TMKgMSxsqWNGSu/p7IMNJ6EgW2M0Iz0sveTPzP66YmvPIzLpPUMEkXi8JUEBOT2ftkwBWjRkwsQaq4vZXQESqkxoZUsiF4yy+vklat6rlV7/6iUr/J4yjCCZzCOXhwCXW4gwY0gYKEZ3iFN0c7L86787FoLTj5zDH8gfP5Az3oj/Y=</latexit><latexit sha1_base64="YL1ePw28IevvBzwsoV26KDEGt2I=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoBeh6MVjBfuBbSiT7aZdutmE3Y1QQv+FFw+KePXfePPfuG1z0NYHA4/3ZpiZFySCa+O6305hbX1jc6u4XdrZ3ds/KB8etXScKsqaNBax6gSomeCSNQ03gnUSxTAKBGsH49uZ335iSvNYPphJwvwIh5KHnKKx0mNvjEmC5JrU+uWKW3XnIKvEy0kFcjT65a/eIKZpxKShArXuem5i/AyV4VSwaamXapYgHeOQdS2VGDHtZ/OLp+TMKgMSxsqWNGSu/p7IMNJ6EgW2M0Iz0sveTPzP66YmvPIzLpPUMEkXi8JUEBOT2ftkwBWjRkwsQaq4vZXQESqkxoZUsiF4yy+vklat6rlV7/6iUr/J4yjCCZzCOXhwCXW4gwY0gYKEZ3iFN0c7L86787FoLTj5zDH8gfP5Az3oj/Y=</latexit><latexit sha1_base64="YL1ePw28IevvBzwsoV26KDEGt2I=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoBeh6MVjBfuBbSiT7aZdutmE3Y1QQv+FFw+KePXfePPfuG1z0NYHA4/3ZpiZFySCa+O6305hbX1jc6u4XdrZ3ds/KB8etXScKsqaNBax6gSomeCSNQ03gnUSxTAKBGsH49uZ335iSvNYPphJwvwIh5KHnKKx0mNvjEmC5JrU+uWKW3XnIKvEy0kFcjT65a/eIKZpxKShArXuem5i/AyV4VSwaamXapYgHeOQdS2VGDHtZ/OLp+TMKgMSxsqWNGSu/p7IMNJ6EgW2M0Iz0sveTPzP66YmvPIzLpPUMEkXi8JUEBOT2ftkwBWjRkwsQaq4vZXQESqkxoZUsiF4yy+vklat6rlV7/6iUr/J4yjCCZzCOXhwCXW4gwY0gYKEZ3iFN0c7L86787FoLTj5zDH8gfP5Az3oj/Y=</latexit>

Figure: The asymptotic risk function Rκ as a function of α (with σ = 0, n = 300,N = 1000, β = n/N = 0.3 and κ = 1, 2 respectively). The location of α∗ from Theorem1 is marked with a black circle. In both cases, the asymptotic risk at α = 1 is lower than theasymptotic risk at α∗.

https://arxiv.org/abs/1906.01139 jx2193@columbia.edu, djhsu@cs.columbia.edu