Robustness and Complex Data Structures || Least Squares Estimation in High Dimensional Sparse...

Chapter 9Least Squares Estimation in High DimensionalSparse Heteroscedastic Models

Holger Dette and Jens Wagener

9.1 Introduction

Consider a linear regression model of the form

Yi = xTi β0 + εi, i = 1, . . . , n, (9.1)

where Yi ∈ R, xi is a p-dimensional covariate, β0 = (β0,1, . . . , β0,p)T the unknownvector of parameters and the εi are i.i.d. random variables. We assume that p isfixed and most of the components of the parameter vector β0 vanish. This scenariois called sparse linear regression model in the literature. The problem of identify-ing significant components of the vector x and estimating the corresponding com-ponents of the parameter β0 has found considerable interest in the last 20 years.A very well-known tool are penalized least squares methods because they providean attractive methodology to select variables and estimate parameters in sparse lin-ear models. We mention bridge regression (Frank and Friedman 1993), Lasso (Tib-shirani 1996), the nonnegative Garotte (Breiman 1995), SCAD (Fan and Li 2001),least angle regression (Efron et al. 2004), the elastic net (Zou and Hastie 2005),the adaptive Lasso (Zou 2006), the Dantzig selector (Candes and Tao 2007) andBayesian Lasso (see the contribution by Konrath, Fahrmeir and Kneib, Chap. 10)among others. Some of these meanwhile “classical” concepts are briefly explainedin Sect. 9.2.

Much of the aforementioned literature concentrates on the case of linear modelswith independent identical distributed errors. To our best knowledge, there has beenno attempt to investigate bridge estimators and the adaptive Lasso in high dimen-sional linear models with heteroscedastic errors.

H. Dette (B) · J. WagenerFakultät für Mathematik, Ruhr-Universität Bochum, 44780 Bochum, Germanye-mail: [email protected]

J. Wagenere-mail: [email protected]

C. Becker et al. (eds.), Robustness and Complex Data Structures,DOI 10.1007/978-3-642-35494-6_9, © Springer-Verlag Berlin Heidelberg 2013

135

mailto:[email protected]

mailto:[email protected]

http://dx.doi.org/10.1007/978-3-642-35494-6_9

136 H. Dette, J. Wagener

The purpose of this chapter is to present some of the commonly used methodsfor variable selection in high dimensional sparse linear regression methods witha special focus on heteroscedastic models. We give a careful explanation on howthe choice of the regularizing parameter affects the quality of the statistical infer-ence (such as conservative or consistent model selection). In Sect. 9.3, it is demon-strated that (under suitable assumption on the regularizing parameter) the modelselection properties of the commonly used estimators in sparse linear regressionmodels still persist under heteroscedasticity. However, the bridge estimators and theadaptive Lasso estimators of the important parameters are asymptotically normallydistributed with the optimal rate, but with a suboptimal variance. In this section wealso present an approach introduced in Wagener and Dette (2012) which is espe-cially designed to address heteroscedasticity. For the case where the dimension p

is fixed and the sample size n is increasing, we present some results regarding con-sistency and asymptotic normality. In particular, the new methods possess an oracleproperty in the sense of Fan and Li (2001), which means that they perform consistentmodel selection (see Sect. 9.3 for a precise definition) and estimate the nonvanish-ing parameters with the optimal rate and variance. The final Sect. 9.4 is devoted toa small numerical study, which investigates some of the theoretical properties forrealistic sample sizes.

9.2 Penalized Least Squares Estimators

9.2.1 Bridge Regression

Frank and Friedman (1993) introduced the so called ‘bridge regression’, which isdefined as the minimizer of an objective function penalized by the Lq -norm (q > 0),that is

L(β) =n∑

i=1

(Yi − xT

i β)2 + λn

p∑

j=1

|βj |q, (9.2)

where λn is a regularization parameter which converges to infinity with increasingsample size. The procedure shrinks the estimates of the parameters in the model(9.1) towards 0. Throughout this paper, we will always assume that 0 < q ≤ 1. Inthis case, it was shown by Knight and Fu (2000) that this procedure identifies thenon vanishing components of the parameter β with positive probability and that thecorresponding estimators are asympotically normal distributed (after an appropriatestandardization). However, the corresponding optimization problem is not convexif 0 < q < 1 due to the non-convex penalty. Therefore, the problem of determin-ing the estimator minimizing (9.2) is a challenging one. Standard tools of convexoptimization theory are not applicable and multiple local minima might exist.

9 Least Squares Estimation in High Dimensional Sparse Heteroscedastic Models 137

9.2.2 Lasso and Adaptive Lasso

The celebrated Lasso (Tibshirani 1996) corresponds to a bridge estimator withq = 1. The asymptotic behavior of these estimators was investigated by Knight andFu (2000), who established asymptotic normality of the estimators of the non-zerocomponents of the parameter vector and showed that the estimator sets some pa-rameters exactly to 0. Thereby the Lasso performs model selection and parameterestimation in a single step with computational cost growing polynomially with thesample size. Fan and Li (2001) argued that any reasonable estimator should be un-biased, continuous in the data, should estimate zero parameters as exactly zero withprobability converging to one (consistency for model selection) and should have thesame asymptotic variance as the ideal estimator in the correct model. They calledthis the ‘oracle property’ of an estimator, because such an estimator is asymptoti-cally (point-wise) as efficient as an estimator which is assisted by a model selectionoracle, that is the estimator which uses the precise knowledge of the non-vanishingcomponents of β0. Knight and Fu (2000) showed that for 0 < q < 1 the bridge es-timator has the oracle property using a particular tuning parameter λn, while Zou(2006) demonstrated that the Lasso can not have it. This author also showed theoracle property for the adaptive Lasso, which determines the estimator minimizingthe objective function

L(β) =n∑

i=1

(Yi − xT

i β)2 + λn

p∑

j=1

|βj ||β̃j |γ

. (9.3)

Here β = (β1, . . . , βp)T and β̃ = (β̃1, . . . , β̃p)T denotes a preliminary estimate ofβ0 which satisfies certain regularity assumptions. Fan and Peng (2004), Kim et al.(2008), Huang et al. (2008a) and Huang et al. (2008b) derived generalizations ofthe aforementioned results in the case where the number of parameters is increasingwith the sample size, in particular for the case where p > n, which will not beconsidered in this paper.

9.3 Penalizing Estimation Under Heteroscedasticity

In order to investigate the behaviour of the commonly used penalized estimators un-der heteroscedasticity, we consider the following (heteroscedastic) linear regressionmodel

Y = Xβ0 + Σ(β0)ε, (9.4)

where Y = (Y1, . . . , Yn)T is an n-dimensional vector of observed random variables,

X is a n × p-matrix of covariates, β0 is a p-dimensional vector of unknown param-eters,

Σ(β0) = diag(σ(x1, β0), . . . , σ (xn,β0)

)


is a positive definite matrix and ε = (ε1, . . . , εn) is a vector of independent randomvariables with E(εi) = 0 and Var(εi) = 1 for i = 1, . . . , n. Note that σ 2(xi, β0) is thevariance of the observation Yi (i = 1, . . . , n). We assume that the model is sparsein the sense that the parameter β0 can be decomposed as β0 = (β0(1)T ,β0(2)T )T ,where β0(1) ∈ R

k and β0(2) = 0 ∈ Rp−k , but it is not known which components of

the vector β0 vanish (this knowledge is called the oracle). Without loss of generality,it is assumed that the k nonzero components are given by the vector β0(1)T . In thefollowing discussion, xT

1 , . . . , xTn denote the rows of the matrix X. Moreover, the

matrix of covariates is partitioned according to β0, that is

X = (X(1),X(2)

),

where X(1) ∈Rn×k and X(2) ∈ R

n×(p−k). The rows of the matrix X(j) are denotedby x1(j)T , . . . , xn(j)T for j = 1,2. We assume that the matrix X is not randombut note that for random covariates all results presented in this contribution holdconditionally on the predictor X.

We will also investigate estimators, which take information regarding the scalefunction into account. For the sake of brevity, we restrict ourselves to estimators ofthe form

β̂lse = argminβ

[n∑

i=1

(Yi − xT

i β)2 + λnP (β, β̃)

],

β̂wlse = argminβ

[n∑

i=1

(Yi − xT

i β

σ (xi, β̄)

)2

+ λnP (β, β̃)

](9.5)

in the linear heteroscedastic model (9.4). Here β̄ and β̃ denote preliminary es-timates of the parameter β0 and P(β, β̃) is a penalty function. The estimators

β̂lse = (β̂Tlse(1), β̂T

lse(2))T and β̂wlse = (β̂Twlse(1), β̂T

wlse(2))T are decomposed in thesame way as the parameter β0, where the first part corresponds to the k non vanish-ing coefficients of β0. We are particularly interested in the cases

P(β, β̃) = P(β) = ‖β‖qq (0 < q ≤ 1),

P (β, β̃) =p∑

j=1

|βj ||β̃j |−γ (γ > 0),

corresponding to bridge regression (with the special case of Lasso for q = 1) andthe adaptive Lasso, respectively, where ‖β‖q = (

∑p

j=1 |βj |q)1/q . The subscripts‘lse’ and ‘wlse’ correspond to ‘ordinary’ and ‘weighted’ least squares regression,respectively. We mention again that for bridge regression with q < 1 the objectivefunctions are not convex in β and there may exist multiple minimizing values. Inthat case, the argmin is understood as an arbitrary minimizing value, and all resultsstated here are valid for any such value.


9.3.1 Ordinary Penalized Least Squares Estimators

For regularizing parameters satisfying λn/√

n → λ0 > 0, it was shown in Wagenerand Dette (2012) that under heteroscedasticity the classical Lasso performs conser-vative model selection in the same way as in the homoscedastic case, that is

limn→∞P

({1, . . . , k} ⊂ {j | β̂j = 0}) = 1 (9.6)

and

limn→∞P(β̂j = 0) = cj for all j > k (9.7)

for some constants c1, . . . , cp−k ∈ (0,1), where β̂j denotes the j th component ofthe Lasso estimator β̂lse (see, e.g., Leeb and Pötscher 2005). This means that theestimator β̂lse = (β̂1, . . . , β̂p)T estimates a vanishing component of β0 as 0 withpositive probability. On the other hand, conditional on the event that βlse is of theform (δT ,0T ) for some vector δ ∈ R

k , the asymptotic covariance matrix of the Lassoestimator

√n(β̂lse(1) − β0(1)

)

of the nonzero parameters is given by C−111 B11C

−111 , where the matrices B11 and C11

are the upper k × k blocks of the limits

1

nXT X → C =

(C11 CT

21C21 C22

)> 0, (9.8)

and

1

nXT Σ(β0)

2X → B =(

B11 BT21

B21 B22

)> 0,

respectively. Note that results of this type require the existence of the limits and sev-eral other technical assumptions. We refer to Wagener and Dette (2012) for a preciseformulation of the necessary assumptions such that these statements are true. Theasymptotic covariance matrix of the standardized statistic

√n(β̂lse(1) − β0(1)) is

the same as for the ordinary least squares (OLS) estimator in heteroscedastic linearmodels. This covariance is not the best one achievable, as under heteroscedasticitythe OLS estimator is dominated by a generalized LS estimator. Additionally, theestimator is biased if λ0 = 0.

Similarly, if λn/nq/2 → λ0 > 0, the bridge estimator also performs conser-vative model selection. Again the asymptotic covariance matrix of the statistic√

n(β̂lse(1) − β0(1)) is given by C−111 B11C

−111 and is suboptimal. However, the esti-

mator is unbiased in contrast to the Lasso estimator.Finally, if p < n, the canonical choice of the preliminary estimate β̃ for the

adaptive Lasso is the ordinary least squares estimator. In this case, if λnn(γ−1)/2 →

λ0 > 0, the adaptive Lasso performs conservative model selection, is asymptoticallyunbiased, but again the asymptotic covariance matrix is not the optimal one.


If the constant cj in (9.7) is exactly 1, that is

limn→∞P(β̂j = 0) = 1 for all j > k (9.9)

and

limn→∞P(β̂j = 0) = 1 for all j ≤ k. (9.10)

the estimator β̂ of the parameter β0 in model (9.4) is called consistent for modelselection. This means that the estimator β̂ = (β̂1, . . . , β̂p)T estimates a vanishingcomponent of β0 as 0 with probability 1. Whether an estimator performs consistentor conservative model selection, respectively, depends on the choice of the tuningparameter λn. A ‘larger’ value of λn usually yields consistent model selection whilea ‘smaller’ value yields conservative model selection. It turns out that for the Lassothe goal of consistent model selection contradicts to the property of estimating thenon-zero coefficients of the parameter β at the optimal rate 1/

√n. For example, if

λn/n → 0 and λn/√

n → ∞, it can be shown that the estimator β̂lse satisfies

n

λn

(β̂lse − β0)P→ argmin(V ),

where the function V is given by

V (u) = uT Cu +k∑

j=1

uj sgn(β0,j ) +p∑

j=k+1

|uj |,

where β0(1) = (β0,1, . . . , β0,k)T and the matrix C is defined in (9.8). Moreover,

the Lasso performs consistent model selection (even under heteroscedasticity) if thestrong irrepresentable condition

∣∣C21C−111 sgn

(β0(1)

)∣∣ < 1p−k (9.11)

(compare, e.g., Zhao and Yu 2006) is satisfied. Here the inequality and the absolutevalue are understood component-wise and 1p−k denotes a p − k dimensional vectorwith all entries given by 1. However, the estimator of the non vanishing parametersdoes not converge with the optimal rate 1/

√n.

On the other hand, the bridge and adaptive estimator perform consistent modelselection and are simultaneously able to estimate the non-vanishing components ofthe vector β with the optimal rate, if the regularizing parameters satisfy λn/

√n → 0

and λn/nq/2 → ∞ (for the bridge estimator) and λn/√

n → 0, λn/n(1−γ )/2 → ∞(for the adaptive Lasso with a preliminary estimator satisfying

√n(β̃ − β0) =

Op(1)). Both estimators are unbiased but have a non-optimal variance, that is

√n(β̂lse(1) − β0(1)

) → N(0,C−1

11 B11C−111

).

Due to the lack of convexity of the optimization criterion it is extremely difficultto calculate the bridge estimator in the case 0 < q < 1. However, the optimization


problem corresponding to the adaptive Lasso is convex (but requires a preliminaryestimator).

9.3.2 Weighted Penalized Least Squares Estimators

In order to construct an oracle estimator, Wagener and Dette (2012) proposed touse a preliminary estimator, say β̄ , to estimate σ(xT

i , β0) and apply a weightedpenalized least squares regression to identify the non-zero components of β0. Thisestimator has to satisfy a mild consistency property. The corresponding objectivefunction is defined in (9.5) and the asymptotic properties of the resulting estima-tor of the nonzero components are described in the following paragraph. For thesake of brevity and transparency, we only mention the mathematical assumptionswhich are necessary to understand the differences between the different procedures.For detailed discussion of the asymptotic theory, we refer to the work of Wagenerand Dette (2012). Roughly speaking the properties of the estimators can be summa-rized as follows. The weighted bridge and the weighted adaptive Lasso estimatorcan be tuned to perform consistent model selection and estimation of the non zeroparameters with the optimal rate simultaneously. Moreover, the corresponding stan-dardized estimator of the non-vanishing components is asymptotically unbiased andnormal distributed with optimal covariance matrix. Different assumptions regardingthe tuning parameters and the initial estimators are necessary for statements of thistype and some details are given in the following discussion.

Weighted Lasso If

λn/√

n → λ0 ≥ 0

and the preliminary estimator β̄ (required for estimating the variance structure) sat-isfies

bn(β̄ − β0) = Op(1) (9.12)

for some sequence 0 < bn → ∞ such that bn/n1/4 → ∞, the weighted Lasso es-timator β̂wlse performs conservative model selection whenever λ0 = 0. The stan-dardized estimator

√n(β̂wlse(1) − β0(1)) of the non-zero components is asymp-

totically normal distributed with expectation D−111 λ0 sgn(β0(1))/2 and covariance

matrix D−111 , where the matrix D11 ∈R

k×k is the upper block in the limit matrix

1

nXT Σ(β0)

−2X → D =(

D11 DT21

D21 D22

)> 0.

This means that the estimator β̂wlse(1) has the optimal asymptotic variance but itsstandardized version is asymptotically biased.

For

λn/n → 0, λn/√

n → ∞,


the weighted Lasso estimator performs consistent model selection if the condition∣∣D21D

−111 sgn

(β0(1)

)∣∣ < 1p−k

is satisfied. Note that this condition is an analogue of the strong irrepresentablecondition (9.11). In this case, the estimator of the nonvanishing components doesnot converge at the optimal rate.

Weighted Adaptive Lasso Note that the weighted adaptive Lasso requires twopreliminary estimators which can but must not necessarily coincide. One estimatorβ̃ is used in the penalty and a further estimator β̄ is required for the parameters inthe variance function σ 2(x,β).

If the preliminary estimators β̄ and β̃ of β0 satisfy (9.12) and

an(β̃ − β0)D→ Z (9.13)

for some positive sequence an → ∞, such that P(Z = 0) = 0 and

λnaγn /

√n → λ0 ≥ 0,

the weighted adaptive Lasso estimator β̂wlse performs conservative model selectionand the standardized estimator

√n(β̂wlse(1) − β0(1)) of the nonzero components is

asymptotically normal distributed with mean 0 and covariance matrix D−111 .

If

λn/√

n → 0, λn/√

naγn → ∞

for some sequence an → ∞ and the preliminary estimators β̄ and β̃ of β0 satisfy(9.12) and an(β̃ −β0) = Op(1), respectively, the weighted adaptive Lasso estimatorβ̂wlse performs consistent model selection, is asymptotically unbiased and convergesweakly with the optimal rate and covariance matrix, that is

√n(β̂wlse(1) − β0(1)

) → N(0,D−1

11

).

Weighted Bridge Estimation We assume that q ∈ (0,1). If

λn/nq/2 → λ0 ≥ 0

and the preliminary estimator β̄ of β0 satisfies (9.12), the weighted bridge esti-mator β̂wlse performs conservative model selection and the standardized estimator√

n(β̂wlse(1) − β0(1)) of the nonzero components is asymptotically normal dis-tributed with mean 0 and covariance matrix D−1

11 .If

λn/√

n → 0, λn/nq/2 → ∞,

the weighted bridge estimator performs consistent model selection and estimates thenonzero parameters with the optimal rate and optimal covariance matrix D−1

11 .


9.4 Finite Sample Properties

In this section, we present a small simulation study and illustrate the differencesbetween the various estimators in a data example. We mention again that despiteof their appealing theoretical properties bridge estimators are extremely difficult tocalculate and therefore we concentrate on a comparison of the Lasso and adaptiveLasso. These estimators can be calculated by convex optimization, and we used thepackage “penalized” available for R on http://www.R-project.org (R DevelopmentCore Team 2008) to perform all computations. The numerical results presented hereare taken from Wagener and Dette (2012). Further finite sample results can be foundin this reference.

In all examples, the data were generated using a linear model (9.4). The errorsε were i.i.d. standard normal and the matrix Σ was a diagonal matrix with entriesσ(xi, β0) on the diagonal, where σ was given by one of the following functions:

(a) σ (xi, β0) = 1

2

√xTi β0,

(b) σ (xi, β0) = 1

4|xT

i β0|,

(c) σ (xi, β0) = 1

20exp |xT

i β0|,

(d) σ (xi, β0) = 1

50exp

(xTi β0

)2.

The different factors were chosen in order to generate data with comparable vari-ance in each of the four models. The tuning parameter λn was chosen by fivefoldgeneralized cross validation performed on a training data set. For the preliminaryestimator β̃ we used the OLS estimator and for β̄ the unweighted Lasso estimator.All reported results are based on 100 simulation runs. The design matrix was gener-ated having independent normally distributed rows and the covariance between thei-th and j -th entry in each row was 0.5|i−j |. The sample size was chosen by n = 60.We consider the model (9.4) with parameter β = (3,1.5,2,0,0,0,0,0) (see Zou2006). The model selection performance of the estimators is presented in Table 9.1,where we show the mean of the correctly zero and correctly non-zero estimatedparameters. In the ideal case these should be 5 and 3, respectively. It can be seenfrom Table 9.1 that the adaptive Lasso always performs better with respect to modelselection than the Lasso. This confirms the asymptotic properties described in theprevious section. The model selection performance of the weighted and unweightedadaptive Lasso are comparable. In Table 9.2, we present the mean squared errorof the estimates for the non-vanishing components β1, β2, β3. In terms of this cri-terion, the weighted versions of the estimators nearly always do a (in some casessubstantially) better job than their unweighted counterparts.

We conclude this chapter by illustrating the differences between the two esti-mators β̂lse and β̂wlse with reanalyzing the diabetes data considered in Efron et al.(2004). The data consist of a response variable Y which is a quantitative measure of

http://www.R-project.org


Table 9.1 Mean number ofcorrectly zero and correctlynonzero estimated parametersin model (9.4) withβ = (3,1.5,2,0,0,0,0,0)

σ

(a) (b) (c) (d)

Lasso = 0 1.67 3.33 1.58 2.64

= 0 3 3 2.99 3

Adaptive Lasso = 0 4.51 4.32 2.95 4.48

= 0 3 3 2.95 3

Weighted Lasso = 0 0.97 1.53 0.67 0.43

= 0 3 3 3 3

Weightedadaptive Lasso

= 0 3.97 4.09 3.29 3.91

= 0 3 3 3 3

Table 9.2 Mean squarederror of the estimators of thenonzero coefficients in model(9.4) with β = (3,1.5,

2,0,0,0,0,0)

σ

(a) (b) (c) (d)

Lasso β1 0.0308 0.0682 0.3480 0.0692

β2 0.0306 0.0374 0.2461 0.0784

β3 0.0322 0.0484 0.3483 0.1141

Adaptive Lasso β1 0.0293 0.0593 0.3514 0.0668

β2 0.0330 0.0393 0.3241 0.1027

β3 0.0285 0.0416 0.3871 0.1126

Weighted Lasso β1 0.0215 0.0424 0.1431 0.2004

β2 0.0171 0.0133 0.0458 0.0174

β3 0.0191 0.0202 0.1086 0.0780

Weightedadaptive Lasso

β1 0.0193 0.0152 0.0944 0.1953

β2 0.0168 0.0069 0.0293 0.0134

β3 0.0165 0.0080 0.0864 0.0763

diabetes progression one year after baseline and of ten covariates (age, sex, bodymass index, average blood pressure and six blood serum variables). It includesn = 442 observations.

First, we calculated the unweighted Lasso estimate β̂lse using a cross-validated(conservative) tuning parameter λn. This solution excluded three covariates from themodel (age and two of the six blood serum variables). In a next step we calculatedthe resulting residuals

ε = Y − Xβ̂lse,

which are plotted in the upper left panel of Fig. 9.1.This picture suggests a heteroscedastic nature of the residuals. In fact the hypoth-

esis of homoscedastic residuals was rejected by the test of Dette and Munk (1998)which had a p-value of 0.006. Next, we computed a local linear fit of the squared


Fig. 9.1 Upper left: Lasso residuals, upper right: Squared residuals together with local polynomialestimate, lower left: rescaled residuals

residuals in order to estimate the conditional variance σ(xTi β) of the residuals. The

upper right panel of Fig. 9.1 presents the squared residuals plotted against its abso-lute values |xT

i β̂lse| together with the local linear smoother, say σ̂ 2. In the lower leftpanel of Fig. 9.1 we present the rescaled residuals ε̃i = (Yi − xT

i β̂lse)/σ̂ (|xTi β̂lse|).

These look “more homoscedastic” than the unscaled residuals and the test of Detteand Munk (1998) has a p-value of 0.514, thus not rejecting the hypothesis of ho-moscedasticity. The weighted Lasso estimator β̂wlse was calculated by (9.5) on thebasis of the “nonparametric” weights σ̂ (xT

i β̂lse) and the results are depicted in Ta-ble 9.3. In contrast to β̂lse, the weighted Lasso only excludes one variable from themodel, namely the blood serum HDL if λn is chosen by cross-validation.

9.5 Conclusions

In this paper, we presented several penalized least squares methods for estimationin sparse high-dimensional linear regression models. We explain the terminology


Table 9.3 The Lasso estimators β̂lse and β̂wlse for the tuning parameter λn selected by cross-validation

Intercept Age Sex BMI BP TC LDL HDL TCH LTG GLU

β̂lse 152.1 0.0 −186.5 520.9 291.3 −90.3 0.0 −220.2 0.0 506.6 49.2

β̂wlse 183.8 −110.3 −271.3 673.3 408.3 84.1 −547.6 0.0 449.4 213.7 138.5

of conservative and consistent model selection. It is demonstrated that under het-eroscedasticity these estimators can be modified, such that they are as efficient asoracle estimators. On the other hand, classical methods like Lasso or adaptive Lassoyield consistent estimators but with sub-optimal rates.

Acknowledgements This work has been supported in part by the Collaborative Research Center“Statistical modeling of nonlinear dynamic processes” (SFB 823, Teilprojekt C1) of the GermanResearch Foundation (DFG). The authors would also like to thank Torsten Hothorn for pointingout some important references on the subject.

References

Breiman, L. (1995). Better subset regression using the nonnegative garrote. Technometrics, 37,373–384.

Candes, E., & Tao, T. (2007). The Dantzig selector: statistical estimation when p is much largerthan n. The Annals of Statistics, 35, 2313–2351.

Dette, H., & Munk, A. (1998). Testing heteroscedasticity in nonparametric regression. Journal ofthe Royal Statistical Society. Series B. Statistical Methodology, 60, 693–708.

Efron, B., Hastie, T., & Tibshirani, R. (2004). Least angle regression (with discussion). The Annalsof Statistics, 32, 407–451.

Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracleproperties. Journal of the American Statistical Association, 96, 1348–1360.

Fan, J., & Peng, H. (2004). Nonconcave penalized likelihood with a diverging number of parame-ters. The Annals of Statistics, 32, 928–961.

Frank, I. E., & Friedman, J. H. (1993). A statistical view of some chemometrics regression tools(with discussion). Technometrics, 35, 109–148.

Huang, J., Horowitz, J. L., & Ma, S. (2008a). Asymptotic properties of bridge estimators in sparsehigh dimensional regression models. The Annals of Statistics, 36, 587–613.

Huang, J., Ma, S., & Zhang, C. (2008b). Adaptive lasso for sparse high-dimensional regressionmodels. Statistica Sinica, 18, 1603–1618.

Kim, Y., Choi, H., & Oh, H.-S. (2008). Smoothly clipped absolute deviation on high dimensions.Journal of the American Statistical Association, 103, 1665–1673.

Knight, F., & Fu, W. (2000). Asymptotics for Lasso-type estimators. The Annals of Statistics, 28,1356–1378.

Leeb, H., & Pötscher, B. M. (2005). Model selection and inference: facts and fiction. EconometricTheory, 21, 21–59.

R Development Core Team (2008). R: A language and environment for statistical computing.R Foundation for Statistical Computing. Vienna, Austria. ISBN 3-900051-07-0. http://www.R-project.org

Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Sta-tistical Society. Series B. Statistical Methodology, 58, 267–288.




Wagener, J., & Dette, H. (2012). Bridge estimators and the adaptive Lasso under heteroscedasticity.Mathematical Methods of Statistics. doi:10.3103/S1066530712020032

Zhao, P., & Yu, B. (2006). On model selection consistency of Lasso. Journal of Machine LearningResearch, 7, 2541–2563.

Zou, H. (2006). The adaptive Lasso and its oracle properties. Journal of the American StatisticalAssociation, 101, 1418–1429.

Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal ofthe Royal Statistical Society. Series B. Statistical Methodology, 67, 301–320.

http://dx.doi.org/10.3103/S1066530712020032

Robustness and Complex Data Structures || Least Squares Estimation in High Dimensional Sparse...

Documents

Transcript of Robustness and Complex Data Structures || Least Squares Estimation in High Dimensional Sparse...