Contents lists available at ScienceDirect

Journal of Statistical Planning and Inference

Journal of Statistical Planning and Inference 147 (2014) 84–94

0378-37http://d

n Tel.:E-m

journal homepage: www.elsevier.com/locate/jspi

On a clustering criterion for dependent observations

Karthik Bharath n

Department of Statistics, The Ohio State University, 1958 Neil Avenue, Columbus, OH 43210, United States

a r t i c l e i n f o

Article history:Received 22 April 2013Received in revised form9 November 2013Accepted 15 November 2013Available online 25 November 2013

Keywords:Clusteringβ-MixingTruncated sumsLLNCLTFunctional CLT

58/$ - see front matter & 2013 Elsevier B.V.x.doi.org/10.1016/j.jspi.2013.11.005

þ1 6077274172.ail address: karthikbharath@gmail.com

a b s t r a c t

A univariate clustering criterion for stationary processes satisfying a β-mixing condition isproposed extending the work of Bharath et al. (2013) to the dependent setup. Theapproach is characterized by an alternative sample criterion function based on truncatedpartial sums which renders the framework amenable to various interesting extensions forwhich limit results for partial sums are available. Techniques from empirical processtheory for mixing sequences play a vital role in the arguments employed in the proofs ofthe limit theorems.

& 2013 Elsevier B.V. All rights reserved.

1. Introduction

The purpose of this paper is two-fold: we propose a clustering criterion for observations from a smooth, invertibledistribution function which is noticeably simpler than the one recently proposed by Bharath et al. (2013); we thendemonstrate the power of its simplicity by proving limit theorems for clustering constructs under a dependence setupextending their work from the i.i.d. case. A pleasant by-product of the proposed framework is that the proofs of the limitresults, in some cases, follow along similar lines to the ones in Bharath et al. (2013).

Hartigan (1978) considered the statistical underpinnings of the k-means clustering framework and derived asymptoticdistributions of a suitably defined criterion function and its maximum. Given a set of observations (data), he defined acluster to be the subset of the observations which are grouped together on the basis of a point which splits the data bymaximizing the between-group sums of squares; in other words, he considered a criterion function which was based onmaximizing the between cluster sums of squares as opposed to minimizing the within cluster sums of squares. Bharath et al.(2013) too resorted to maximizing the between cluster sums of squares but deviated from Hartigan's framework and insteadconsidered the zero of a certain function of the derivative of Hartigan's criterion function. Contrary to Hartigan's setup,which required the existence of a fourth moment for asymptotic results, they proved results for their clustering constructsunder a second moment assumption with an added smoothness condition on the criterion function. Both works consideredi.i.d. data and primarily the case k¼2, viz. two clusters.

We extend the approach used in Bharath et al. (2013) to the case of dependent data satisfying a β-mixing condition andreprove all results under a dependent setup using techniques from empirical processes. We achieve this by employing analternative sample version of the criterion function used in their paper which, by virtue of its definition, renders itself

All rights reserved.

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–94 85

amenable to a variety of interesting scenarios. The circumscription of the proposed framework to the β-mixing case is notnecessary and is artificial; our intention is to highlight the versatility of our definition of the sample criterion function andits deployment in investigating asymptotic behavior of the k-means clustering criterion for dependent observations. Theresults presented in this paper ought to be viewed as a demonstration of the framework's amenability to sequences having avariety of dependence structures for which limit results for partial sums exist. The results presented in this paper are alsovalid to sequences which satisfy a ϕ-mixing condition since it represents a stronger condition in the sense that a ϕ-mixingsequence is also β-mixing.

Interest in β-mixing sequences has increased in recent years with attempts to develop stability bounds for learningalgorithms. In many machine learning applications the i.i.d. assumption is not valid since the observations obtained from thelearning algorithms usually display inherent temporal dependence. In fact, Vidyasagar (2003) argues that β-mixing is “justthe right” assumption for the analysis of weakly dependent sample points in machine learning, in particular since severallearning results then carry over to the non-i.i.d. case. Another important application of β-mixing sequences is in modelingscenarios involving aperiodic and positively recurrent Markov chains; i.e. if fYigiZ1 is an aperiodic positive recurrent Markovchain, then it satisfies the β mixing condition (see Rio, 2000, p. 139). This fact has been employed in several econometricapplications; for an overview see Doukhan (1994). In a similar vein, the condition of ϕ-mixing is also a fairly commonassumption while analyzing stability of learning algorithms such as Support Vector Regression (SVR) (Vapnik, 1998), KernelRidge Regression (KRR) (Saunders et al., 1998) and Support Vector Machines (SVM) (Cortes and Vapnik, 1995). Sinceϕ-mixing represents a stronger condition than β-mixing, stability bounds for β-mixing usually lead to good bounds forϕ-mixing too. In a recent paper, Mohri and Rostamizadeh (2010) proved stability bounds for stationary ϕ-mixing andβ-mixing processes with applications in the above-mentioned algorithms. It is therefore of considerable interest, at least inmachine learning applications, to move over from the i.i.d. setup to the β-mixing or ϕ-mixing cases when weak dependencewhich decays over time is a reasonable modeling assumption. The results in this paper can conceivably be employed whileanalyzing clustering properties of such sequences.

In Section 2 we review some of the pertinent constructs from Bharath et al. (2013) and Hartigan (1978) and define thequantities of interest in this paper: the theoretical and sample criterion functions and their respective zeroes. In Section 3we state our assumptions regarding the conditions on the distribution function F and rate of decay of the β-mixingcoefficients for the rest of the paper. Then, in Section 4, we examine the nature of the criterion function as a statisticalfunctional which induces a functional on the space of cádlág functions and note some of the difficulties involved in directlyusing existing results. In Section 5, in anticipation of the arguments employed in the main results, we prove a fewpreparatory lemmas and state two results from the literature which play an important role in our proofs. Section 6comprises the main results of this paper. We first examine the sample criterion function and prove a weak convergenceresult; this then provides us with several useful corollaries which will then assist us in examining the sample split point.However, since the “zero” of our sample criterion function is similar to the one in Bharath et al. (2013), the proofs ofstatements regarding its asymptotic behavior follow along almost identical lines as the ones for the empirical split point inBharath et al. (2013); for ease of exposition and interests of brevity, we direct the interested reader to their paper for thedetails of the proofs. Finally, in Section 7, we provide numerical verification of the limit theorems via a simulation exercise.

2. Preliminaries

Let us first consider the criterion function that was introduced in Hartigan (1978) for partitioning a sample into twogroups. For a distribution function F, if Q is the quantile function associated with F, then the split function of Q at pAð0;1Þ isdefined as

B Q ; pð Þ ¼ p1p

Zqop

Q qð Þ dq� �2

þ 1�pð Þ 11�p

Zq4p

Q qð Þ dq� �2

�Z 1

0Q ðqÞ dq

" #2: ð2:1Þ

A value p0Að0;1Þ which maximizes the split function is called the split point. When F is invertible (Q is the unique inverse),Bharath et al. (2013) considered a different criterion function defined as, for 0opo1,

G pð Þ ¼ 1p

Z p

0Q qð Þ dqþ 1

1�p

Z 1

pQ qð Þ dq�2Q pð Þ;

which is a function of the derivative of BðQ ; pÞ with respect to p. The zero of G coincides with the maximum of the parabolicsplit function B. The linear statistical functional (of F) G proffers a simple way to determine the split point when F isinvertible as opposed to Hartigan's parabolic split function which accommodated a general F.

If XðiÞ; i¼ 1;…;n are order statistics corresponding to i.i.d. real-valued observations Xi obtained from F, then the empiricalcounterpart of G, christened, the Empirical Cross-over Function (ECF), was defined in Bharath et al. (2013) as

GnðpÞ ¼ 1k

∑k

j ¼ 1XðjÞ �XðkÞ þ

1n�k

∑n

j ¼ kþ1XðjÞ �Xðkþ1Þ;

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–9486

for ðk�1Þ=nrpok=n and

GnðpÞ ¼1n

∑n

j ¼ 1XðjÞ �XðnÞ;

for ðn�1Þ=nrpo1, where 1rkrn�1. The “zero” of Gn was the estimate of the theoretical split point and it was shownthat the estimate was consistent, a CLT was proven and its utility was demonstrated on a real-world dataset.

Our chief interest in this paper is to examine an alternative definition for G and consequently Gn—we change the domainof the functions from ð0;1Þ to an appropriate subset of the real line. This then permits us to come up with a sample versioncomprising truncated sums, at fixed level, as opposed to trimmed sums used in the definition of Gn. We first redefine thecriterion function.

Definition 1. Consider the extended real line R [ f�1;1g. Let xþ ¼ supfx : FðxÞo1g and x� ¼ inffx : FðxÞ40g, the upperand lower endpoints of F—they can both be infinite. Then, for tA ½x� ; xþ �DR, the cross-over function is defined as

T tð Þ ¼ 1FðtÞ

Z t

�1x dFþ 1

1�FðtÞZ 1

tx dF�2t: ð2:2Þ

Suppose p0 is the zero of G, then the zero of T, say t0, would be equal to Q ðp0Þ. Prior to defining the sample-based versionof T, let us recall the definition of the empirical distribution function. For a sequence Xi;1r irn (not necessarily i.i.d.),consider the usual empirical distribution function

Fn tð Þ ¼

0 if toXð1Þkn

if XðkÞrtoXðkþ1Þ

1 if tZXðnÞ

8>>><>>>:

where Xð1Þr⋯rXðnÞ a.s. are the order statistics.

Definition 2. For tA ½Xð1Þ;XðnÞÞ the sample cross-over function is defined as

Tn tð Þ ¼ 1FnðtÞ

Z t

�1x dFnþ 1

1�FnðtÞZ 1

tx dFn�2t

¼ 1nFnðtÞ

∑n

i ¼ 1XiIXi r tþ

1nð1�FnðtÞÞ

∑n

i ¼ 1XiIXi 4 t�2t; ð2:3Þ

where IB is the indicator function of the set B.

We now extend the definition of Tn onto ðx� ; xþ Þ by defining a¼ TnðXð1ÞÞ and b¼ limt↑XðnÞ Tn so that Tn is now defined on½a; b� with �1rx� raobrxþ r1. The right end-point b exists since Tn, by virtue of its definition, is cádlág. All resultswill be proved for tAðx� ; xþ Þ since with probability 1 all limits lie within the compact set ½x� ; xþ �. Unless stated otherwise,we shall assume throughout that Tn (and its functionals) are all defined on tA ðx�; xþ Þ.

The problem usually associated with truncated sums is with the decision regarding the level of truncation. In our setup,however, since all levels of truncations are investigated for determining the sample split point, we are untroubled by theissue. The advantage with the truncated sums representation is in the fact that if Xi are observations from a stationarysequence, then Tn is basically a combination of partial sums of the random variables themselves for a fixed truncation level tand not the order statistics. Indeed, literature is rife with limit results for stationary sequences which come in handy whileexamining the asymptotic behavior of Tn.

Remark 1. The function G (and T on its appropriate domain), starting positive, crosses over zero at a point p0A ð0;1Þ, thesplit point. The point of crossing is of chief interest and the conditions on the distribution function F which guarantee theuniqueness of p0 remain an open question (see Remark 1 in Bharath et al., 2013). However, it is vitally important to controlthe behavior of G in the vicinity of 0 and 1 since results involving the sample split point for two clusters are provenassuming only one crossing of G. While it was noted in Bharath et al. (2013) that the condition lim supp↑1 GðpÞo0 ensuredthe good behavior of G near 1, the limit results for the sample split point were proved in ½a′; b′� for 0oa′ob′o1. Thefunction T, on the other hand, by virtue of its definition on ½x� ; xþ � quite readily satisfies

lim supt↑xþ

TðtÞo0 and lim inft↓x�

TðtÞ40:

This property assists us in our definition of the sample split point.

Denote by t0 the solution of TðtÞ ¼ 0. We will assume throughout that the solution, t0, is unique. We are now ready todefine its empirical counterpart.

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–94 87

Definition 3. For nZ1, the sample split point pertaining to Tn is defined as

tn ¼�1 if TnðtÞo0 8tAðx�; xþ Þ;þ1 if TnðtÞ40 8tAðx�; xþ Þ;supftA ðx� ; xþ Þ : TnðtÞZ0g otherwise:

8><>:

Armed with alternative definitions for the theoretical criterion function, its sample version and the sample split point, wenow extend the clustering framework developed in Bharath et al. (2013) to dependent observations arising from apopulation with a smooth and invertible distribution function. The partial-sums nature of Tn offers a platform to consideralmost any kind of dependence structure for which a law of large numbers and a central limit theorem of some sort exist.

3. Assumptions

Let X¼ fXigiZ1 be a strictly stationary sequence of real valued random variables with distribution function F. In certaincases, if there is no confusion, for ease of notation, we shall denote by X the canonical random variable from F. For thesequence X, let

sm ¼ sðX1;…;XmÞbe the s field generated by the random variables X1;…;Xm. In a similar fashion let

smþk ¼ sðXmþk;Xmþkþ1;…Þ:We assume that the random variables satisfy a β-mixing condition with the β-mixing coefficient βk for the sequenceX defined as

βkðXÞ ¼ supmZ1

E supfjPðBjsmÞ�PðBÞj : BAsmþkg: ð3:1Þ

Suppose one obtains a strictly stationary β-mixing subsequence X1;…;Xn of X as observations. We make the followingassumptions:

A1.

F is invertible for 0opo1 and absolutely continuous with density f with respect to the Lebesgue measure. A2. EðXÞ ¼ 0 and EðX2Þ ¼ 1. A3. F is twice continuously differentiable at any tAðx� ; xþ Þ. A4. For r41, βk ¼ oðk� rÞ.

Assumption A4 does not represent the weakest possible condition on the mixing rate guaranteeing the validity of ourresults. Our objective in this paper is not to refine or develop the limit theory for β-mixing sequences; we primarily intend touse it as a device to incorporate dependence within observations which are to be clustered. We do not claim to prove ourresults under the weakest possible conditions. Indeed, to our knowledge, there does not exist a weak, universal condition onthe rate of decay of the β-mixing coefficients which would guarantee convergences in most cases—unless, of course, thestrongest known hitherto is assumed. Assumption A2 will be clarified in the subsequent section upon noting the invarianceof the “zero” of Tn to scaling and translations of the observations.

4. Some remarks on T and Tn

In this paper, as with Bharath et al. (2013), we only consider the case of two clusters. The criterion function T can bethought of as a statistical functional on the space of distribution functions, which induces a functional on D½x� ; xþ �, thespace of cádlág functions on ½x� ; xþ �. In other words, T can be represented as a statistical functional of F defined as, fortA ½x� ; xþ �,

V F; tð Þ ¼ 1FðtÞ

Z t

x�s�tð Þ dF sð Þþ 1

1�FðtÞZ xþ

ts�tð Þ dF sð Þ:

Equivalently, for fixed t, we have

VðF; tÞ ¼ V1ðF1; tÞþV2ðF2; tÞ;where

F1ðsÞ ¼FðsÞFðtÞ if x� rsrt

1 if tosorxþ ;

8><>: F2 sð Þ ¼

FðsÞ�FðtÞ1�FðtÞ if tosrxþ

0 if x� rsrt

8><>:

with V1ðF1; tÞ ¼R ðx�tÞ dF1ðxÞ and V2ðF2; tÞ ¼

R ðx�tÞ dF2ðxÞ. Hence, for fixed tA ½x� ; xþ �, F1 and F2 are the distributionfunctions of a real-valued random variable truncated above and below at t, respectively. Then, V1 and V2 are the respective

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–9488

mean functionals with t subtracted and one is perhaps entitled to believe that existing limit results for differentiablestatistical functionals can be employed with minimal fuss. Unfortunately, if t is finite and x� is �1, for example in V1, thenthe functional F↦

R½x� ;t�s dF is not Hadamard differentiable with respect to the supremum norm from the domain of all

distribution functions to ½x� ; t�; the same is true for V2 too (see van der Vaart, 2000, problem 7, p. 303). This is so since thesupremum norm offers us no control over the tails of the distributions.

The intuition behind the sample criterion function is simple: Tn is based on the distances between a point t and themeans of the observations bounded above and below by t. Then, by checking the distances for all possible t, one hopes toascertain the t at which the distances match up, viz., the function Tn becomes 0. Based on that particular t, one is then able toinfer if the sample perhaps was obtained from a population with ‘more than one mean’ or estimate the split point.

As in the case of Gn used in Bharath et al. (2013), we note that the “zero” of Tn is invariant to scaling and translation oforiginal data. Suppose fXig1r irn is the sequence of interest and if Zi ¼ uXiþv for vAðx� ; xþ Þ and u40, and we denote by Tznthe sample criterion function based on Zi with empirical cdf Fzn, then

Tzn tð Þ ¼ 1

nFznðtÞ∑n

i ¼ 1ZiIZi r tþ

1nð1�FznðtÞÞ

∑n

i ¼ 1ZiIZi 4 t�2t:

It is an easy exercise to check that

Tzn tð Þ ¼ uTn

t�vu

� �;

and therefore, the “zero” of Tn remain unchanged. This elucidates the role played by Fn in the definition of Tn: the presence ofFn guarantees the invariance of tn with respect to scaling and translation.

5. Preparatory results

In this section, in anticipation of the subsequent arguments to be employed in the proofs, we prove a few lemmas andstate two results from the literature on empirical processes for dependent sequences. We first state a result from Yu (1994)which will be used in the proof of weak convergence of Tn.

Theorem 1 (Yu, 1994). Suppose F is a class of functions for which Nðε; d;F Þ ¼Opðε�wÞ for some w40. If 0okr1, then for any0osok and ε40, as n-1,

P supf AF

ns=ð1þ sÞ 1n

∑n

i ¼ 1f Xið Þ�Ef Xið Þ

��������4ε

" #-0:

The following theorem by Arcones and Yu (1994) proves a functional limit theorem for empirical process of β-mixingsequences indexed by a VC-class (Vapnik–C˘ervonenkis class) of functions.

Theorem 2 (Arcones and Yu, 1994). Suppose F is a measurable uniformly bounded VC class of functions. If assumption A4 issatisfied, then there is a Gaussian process fGðf Þgf AF which has a version with uniformly bounded and uniformly continuous pathswith respect to the J � J2 norm such that

n�1=2 ∑n

i ¼ 1ðf ðXiÞ�Ef ðXiÞÞ

( )f AF

) fGðf Þgf AF in l1ðF Þ:

Lemma 1. If fXigiZ1 is a stationary sequence satisfying a β-mixing condition, then for any measurable function h, the sequencefhðXiÞgiZ1 satisfies the β-mixing condition (Yu, 1994).

Proof. For any measurable function h, observe that the s-field of h(X) is contained in the s-field of X. Therefore, the functionh has its β-mixing rate bounded by the corresponding rate of the original sequence and the sequence fhðXiÞgiZ1 satisfies theβ-mixing condition. □

One way of quantifying the ‘size’ of a class of functions is by the notion of covering number; we recall its definition (seep. 275 of van der Vaart, 2000, for instance).

Definition 4. The covering number Nðε;F ; dÞ, related to semi-metric d on a class F of functions, is defined as

Nðε;F ; dÞ ¼minm

there are g1;…; gm in L1 such that minj

dðf ; gjÞrε for any f in F� �

:

A well-known result from empirical process theory states that Vapnik–C̆ervonenkis (VC) classes are examples ofpolynomial classes in the sense that their covering numbers are bounded by a polynomial in ε�1 uniformly over allprobability measures for which the class F is not identically zero (see Lemma 19.15, p. 275 of van der Vaart, 2000). The

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–94 89

following lemma asserts a Glivenko–Cantelli-type result for the empirical distribution function defined using β-mixingrandom variables.

Lemma 2. For any ε40, under assumptions A1–A4,

P suptA ½x� ;xþ �

jFnðtÞ�FðtÞj4ε

" #-0 as n-1:

Proof. Note that the class fIð�1;t� : tA ½x� ; xþ �g is a VC class with index bounded above by ðnþ1Þ since Fn(t) is based on nrandom variables. The class hence has a covering number which is Opðε�1Þ for ε40. From Theorem 1, we have the result forβ-mixing sequences. Therefore, the classical Glivenko–Cantelli property for sums of random variables is preserved underβ-mixing. □

Using the result from the preceding lemma, we quantify the size of the classes of functions which are of interest in thepaper.

Lemma 3. The classes of functions G1 ¼ fXIXr t ; tA ½x� ; xþ �g and G2 ¼ fXIX4 t ; tA ½x� ; xþ �g are VC with VC-index2.

Proof. Note that the subgraphs of G1, fðt;uÞ : XIXr toug are increasing in t. This implies that they cannot shatter any set of 2points. More precisely, from Lemma 2, since fIð�1;t� : tA ½x� ; xþ �g is a VC class of sets and the subgraphs of G1 are increasing,the smallest number of points k, for which no set of size k is shattered is 2. Since the VC index of G1 is finite, it is VC.Furthermore, owing to the fact that G2 ¼ Gc

1, the VC-property is preserved. □

Corollary 1. For any ε40 and for a fixed t, under assumptions, A1–A4,

1.

P½suptA ½x� ;xþ �jð1=nÞ∑ni ¼ 1XiIXi r t�EXIXr t j4ε�-0 as n-1.

2.

P½suptA ½x� ;xþ �jð1=nÞ∑ni ¼ 1XiIXi>t�EXIX>t j4ε�-0 as n-1.

Proof. The proof is an immediate consequence of Lemma 3 and Theorem 1. □

It is therefore the case that the classes G1 and G2 consisting of truncated-at-fixed level functions have covering numberswhich are Opðε�1Þ and are therefore VC. Observe that the classes G1 and G2 are uniformly bounded by X which is in L2 owingto assumption A2.

6. Asymptotics

6.1. Weak convergence of Tn

We now have all the ingredients required to prove the weak convergence of Tn. Suppose we denote UnðtÞ ¼ffiffiffin

p ðTnðtÞ�TðtÞÞ. The presence of Fn in the definition of Tn renders Un(t) unsuitable for representation as an empiricalprocess indexed by classes of functions; we are hence unable to use results from empirical processes under β-mixing todirectly obtain the asymptotic normality of Un. For convenience, we define a few quantities first. Let μlðtÞ ¼ ð1=FðtÞÞEXIXr t

and μuðtÞ ¼ ð1=ð1�FðtÞÞÞEXIX>t . Also, let

ξt ¼1

FðtÞ X�μl tð Þ �

IXr tþ1

1�FðtÞ X�μu tð Þ �IX>t : ð6:1Þ

Note that based on Lemma 1, for fixed t, fξ0t ; ξðiÞt giZ1 is a β-mixing sequence. Denote by l1½x� ; xþ � the space of boundedfunctions on ½x� ; xþ �.

Theorem 3. Under assumptions A1–A4, there is a centered Gaussian process fUðtÞ; tA ½x� ; xþ �g which has a version withuniformly bounded and uniformly continuous paths with respect to the J � J2 norm such that

UnðtÞ ) UðtÞ in l1½x� ; xþ �;with non-negative real valued covariance function given by the absolutely convergent series

st ¼ ∑iAZ

Eðξ0t ξðiÞt Þ: ð6:2Þ

Proof. We follow the technique espoused in Billingsley (1968) by first showing the convergence of the finite dimensionaldistributions and then verifying that the sequence fUng is asymptotically tight and concentrates on a compact set with highprobability. We first prove the finite dimensional convergence by expressing Un as partial sums of β-mixing randomvariables; the result follows then from Doukhan et al. (1994). We then proceed to show that fUngnZ1 is tight in an indirectmanner by considering Un component-wise and establishing tightness of the individual components—indeed, then, the sumof tight component sequences would be tight.

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–9490

Finite dimensional convergence: As mentioned earlier, our intention is to represent Un as partial sums of β-mixing randomvariables plus an error term which goes to zero in probability:

Un tð Þ ¼ ffiffiffin

pTn tð Þ�T tð Þð Þ

¼ ffiffiffin

p 1nFnðtÞ

∑n

i ¼ 1XiIXi r t�

1FðtÞ EXIXr t

" #

� ffiffiffin

p 1nð1�FnðtÞÞ

∑n

i ¼ 1XiIXi>tþ

11�FðtÞ EXIX>t

" #:

Now, observe that

ffiffiffin

p 1nFnðtÞ

∑n

i ¼ 1XiIXi r t�

1FðtÞ EXIXr t

" #

¼ ffiffiffin

p 1nFnðtÞ

∑n

i ¼ 1XiIXi r t�

1FnðtÞ

EXIXr t

" #þ ffiffiffi

np

EXIXr t1

FnðtÞ� 1

FðtÞ

� �� �: ð6:3Þ

Following some cumbersome algebra, the RHS of (6.3) can be expressed as

n�1=2

FnðtÞ∑n

i ¼ 1XiIXi r t�

1FðtÞ EXIXr tIXi r t

� � !¼ n�1=2

FnðtÞ∑n

i ¼ 1τi;

where

τi ¼ ðXi�μlðtÞÞIXi r t

are stationary β-mixing random variables and Eτi ¼ 0 for each i¼ 1;…;n. In a similar manner, we have that

ffiffiffin

p 1nð1�FnðtÞÞ

∑n

i ¼ 1XiIXi>t�

11�FðtÞ EXIX>t

" #¼ n�1=2

1�FnðtÞ∑n

i ¼ 1ηi;

where

ηi ¼ ðXi�μutÞIXi>t

are stationary β-mixing random variable with zero mean. We thus have the convenient representation

Un tð Þ ¼ n�1=2 1FnðtÞ

∑n

i ¼ 1τiþ

11�FnðtÞ

∑n

i ¼ 1ηi

" #:

At this point, we still do not have the desired representation; we will, if we can ‘replace’ Fn(t) by F(t) in the precedingequation. We now demonstrate that we are, in fact, able to do precisely that. Observe that, owing to assumptions A1–A4,

n�1=2 ∑n

i ¼ 1τi

1FnðtÞ

� 1FðtÞ

� �-P0

since, n�1=2∑ni ¼ 1τi is asymptotically normal from CLT for β-mixing sequences (see Doukhan et al., 1994, for instance) and

hence bounded in probability and by continuity of the reciprocal, 1=FnðtÞ-P 1=FðtÞ. Thus,

n�1=2 1FnðtÞ

∑n

i ¼ 1τi

" #¼ n�1=2 1

FðtÞ ∑n

i ¼ 1τi

" #þop 1ð Þ:

By a similar argument we can claim that

n�1=2 11�FnðtÞ

∑n

i ¼ 1ηi

" #¼ n�1=2 1

1�FðtÞ ∑n

i ¼ 1ηi

" #þop 1ð Þ:

Finally, we have our desired representation: for each tAðx�; xþ Þ,

Un tð Þ ¼ n�1=2 ∑n

i ¼ 1

1FðtÞ τiþ

11�FðtÞ ηi

� � !þop 1ð Þ;

¼ n�1=2 ∑n

i ¼ 1ξðiÞt þopð1Þ; ð6:4Þ

where ξðiÞt is as defined in (6.1). Now, ðUnðt1Þ;…;UnðtkÞÞ can be represented as sums of k-dimensional random vectorsðξðiÞt1 ;…; ξðiÞtk Þ for i¼1,…,n by virtue of the representation in (6.4). Using the multivariate version of the CLT for β-mixingsequences (see Doukhan et al., 1994), ðUnðt1Þ;…;UnðtkÞÞ convergences in distribution to a multivariate normal vector.

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–94 91

Tightness: Tightness is verified in an indirect manner. Instead of proving directly that fUng concentrates on a compact setwith high probability, we note that Tn comprises two truncated mean processes in t,

Mð1Þn tð Þ ¼ 1ffiffiffi

np ∑

n

i ¼ 1XiIXi r t�EXIXr t �

: tA x� ; xþ½ �( )

Mð2Þn tð Þ ¼ 1ffiffiffi

np ∑

n

i ¼ 1XiIXi>t�EXIX>t �

: tA x� ; xþ½ �( )

:

Consider the two classes of functions G1 and G2 defined in Lemma 3. Since G1 and G2 are both VC class, from Theorem 2 we havethatMð1Þ

n ðtÞ andMð2Þn ðtÞ converge weakly to Gaussian processesM1ðtÞ andM2ðtÞ in l1ðG1Þ and l1ðG2Þ, respectively, with respect to

the J � J2 norm. This informs us that the two sequences of truncated mean processes are relatively compact. Since l1 is completeand separable with respect to the J � J2 norm and the corresponding metric, using the converse of Prohorov's theorem (seeBillingsley, 1968, p. 115) we can claim that each of the truncated mean sequences is tight. Furthermore, since the sum of tightsequences concentrates on a compact set, we can claim that the process Mð1Þ

n þMð2Þn is tight. The only unresolved issue is the

presence of Fn and ð1�FnÞ in the denominator of Un. Since both the quantities converge uniformly in ðx� ; xþ Þ owing to Lemma2, we can use Slutsky's lemma to claim that Mð1Þ

n =Fn and Mð2Þn =ð1�FnÞ are tight and consequently, so is their sum. □

The functional CLT for Tn provides us with a slew of related results which will assist us considerably while consideringthe asymptotic behavior of tn.

Corollary 2. Under assumptions A1–A4, for any ε40, we have

P suptA ðx� ;xþ Þ

jTnðtÞ�TðtÞj4ε

!-0 as n-1:

Proof. Note that C½x� ; xþ � �D½x� ; xþ � � l1½x� ; xþ �. While convergence of Un to U is in l1½x� ; xþ �, from Theorem 3 it is thecase that UAC½x� ; xþ � with probability one (in fact, U has uniformly continuous paths and belongs to a smaller class offunctions). Therefore the functional suptA ðx� ;xþ ÞjUðtÞj is continuous in C½x� ; xþ � with respect to metric induced by the J � J2norm inherited from l1½x� ; xþ �. As a consequence, using the mapping theorem (see Pollard, 1984, for instance)

supx� o toxþ

ffiffiffin

p jTnðtÞ�TðtÞj ) supx� o toxþ

jUðtÞj

and it is hence true that Tn converges to T uniformly in ðx� ; xþ Þ. □

An important consequence of the functional CLT which will be used in the sequel is stochastic equicontinuity of Un. Theproof is straightforward and is omitted.

Corollary 3. Under assumptions A1–A4, for every ε40 and η40, there exists δ40 such that

lim supn

P sups;tA ðx� ;xþ Þ:jt� sjo δ

jUnðtÞ�UnðsÞj4η

!oε:

In other words, fUnðtÞ : tA ðx� ; xþ Þg is stochastically equicontinuous.

6.2. Asymptotics of tn

At this juncture the hard part has been done and we are in a convenient position of using arguments from Bharath et al.(2013) directly, with minimal changes, in the limit theorems for tn. This is so since the proofs of results for the empirical splitpoint pn in Bharath et al. (2013) are based solely on the weak convergence and uniform convergence of the sample criterionfunction; the results in the preceding section play a similar role in that regard and we can therefore repeat arguments fromtheir proofs with very little change. We are interested in showing consistency of tn and proving a CLT. Let us enumerate thesequence of steps which will lead us to the required results.

(i)

It is first shown that tn-Pt0, where Tðt0Þ ¼ 0—i.e., t0 is the true split point.

(ii)

Under the assumption that T ′ðt0Þo0 (see Theorem 2 in Hartigan, 1978), it is then shown that tn belongs to a Opðn�1=2Þneighborhood of t0.

(iii)

Then, using stochastic equicontinuity of the process Un, it is shown that in a Oðn�1=2Þ neighborhood of t0, Tn can beapproximated uniformly by a line with slope T ′ðt0Þ and intercept Tnðt0Þ up to opðn�1=2Þ. It is then shown that the t atwhich the line crosses over zero is within opðn�1=2Þ of where Tn crosses over zero, which is tn.

(iv)

Using the preceding result, tn is shown to have a convenient representation in terms of t0 and Tnðt0Þ which immediatelyleads to the CLT for tn.

Usually stochastic equicontinuity of Un would have been used in conjunction with empirical process techniques to obtainthe asymptotics of tn via an argmax or argmin reasoning. In Bharath et al. (2013) it was noted that their empirical criterion

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–9492

function Gn was not in a form which would have facilitated the representation of the empirical split point pn as anM-estimator. We face a similar problem here in the sense that Tn cannot be expressed as an estimate of the statisticalfunctional T using the empirical distribution function Fn; empirical process techniques, directly, do not aid us as was noted inthe arguments for weak convergence. Therefore, we proceed along similar lines as in Bharath et al. (2013) and use ageometric argument to obtain the results for tn.

We note now how proofs for asymptotics of the sample split point in Bharath et al. (2013) can be employed with veryminimal alterations. The crucial observation lies in the fact that the asymptotics for tn are entirely based on the weakconvergence result for Tn. At this point, one can quite safely forget the dependence amongst the observations Xi and proceedby minimally altering proofs for pn in Bharath et al. (2013). What is pertinent to keep in mind, however, is thattnA ½x� ; xþ �DR whereas in their case the empirical split point pn was in ½a; b� � ð0;1Þ. The technical arguments, however,remain the same once asymptotics of Tn are provided.

In Bharath et al. (2013), stochastic equicontinuity of Un defined in terms of trimmed sums was not used in any of thearguments. Instead, the increments of Un were quantified in their Lemma 2, which is a weaker result in comparison withstochastic equicontinuity. Proof of item (i) follows without change from Theorem 2 in Bharath et al. (2013) with stochasticequicontinuity replacing their usage of Lemma 2. Proof of item (ii) is exactly the same as the proof of Lemma 3 in their paper.We state the results without proof.

Theorem 4. Assume that t0 is the unique split point. Under assumptions A1–A4, for any ε40,

Pðjtn�t0j4εÞ-0 as n-1:

Lemma 4. Assume A1–A4 hold. Suppose t0 is the unique split point in ðx� ; xþ Þ and T ′ðt0Þo0. Then

tn ¼ t0þOpðn�1=2Þ:

We shall now prove item (iii).

Lemma 5. Assume A1–A4 hold; suppose TðtÞ ¼ 0 has a unique solution t0 and T ′ðt0Þo0, then, uniformly for tA ½t0�εn; t0þεn�,TnðtÞ�Tnðt0Þ ¼ T ′ðt0Þðt�t0Þþopðn�1=2Þ

where εn ¼Oðn�1=2Þ.

Proof. The argument here, using stochastic equicontinuity of Un, is different from the one in Bharath et al. (2013). Using thefact that Un satisfies stochastic equicontinuity, choose ε40 and η40 such that

lim supn

P suptA In

jUnðtÞ�Unðt0Þj4η

!oε

where In ¼ ½t0�C=ffiffiffin

p; t0þC=

ffiffiffin

p � for some C40. Therefore,

suptA In

ffiffiffin

p jTnðtÞ�Tnðt0Þ�½TðtÞ�Tðt0Þ�j ¼ opð1Þ ð6:5Þ

Now, note that

TðtÞ�Tðt0Þ ¼ T′ðt0Þðt�t0ÞþOðn�1Þowing to assumption A3 for all tA In. Consequently, substituting in (6.5), we have the required result. □

From Lemma 4, although tn belongs to a Opðn�1=2Þ of t0, it is not immediately clear how a CLT follows. The followinglemma provides a convenient representation from which asymptotic normality of tn readily follows.

Lemma 6. Under assumptions A1–A4, suppose that t0 is the unique split point and T ′ðt0Þo0. Then as n-1,

tn ¼ t0�Tnðt0ÞT′ðt0Þ

þop n�1=2�

:

Proof. We adopt the argument employed in Lemma 5 of Bharath et al. (2013). Consider the line given by

LnðtÞ ¼ Tnðt0ÞþT ′ðt0Þðt�t0Þ:Denote by tnn the random variable which is the solution of

LnðtÞ ¼ 0;

that is, almost surely,

tnn ¼ t0�Tnðt0ÞT′ðt0Þ

: ð6:6Þ

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–94 93

From the weak convergence offfiffiffin

p ðTn�TÞ to a Gaussian process, we know thatffiffiffin

p ðTnðt0Þ�Tðt0ÞÞ ¼ffiffiffin

pTnðt0Þ ¼Opð1Þ;

and hence tnn ¼ t0þOpðn�1=2Þ. If we can “replace” tnn by tn in (6.6), asymptotic normality of tn would follow automatically.More precisely, it is required thatffiffiffi

np ðtn�tnnÞ ¼ opð1Þ:

From Lemma 4, tn ¼ t0þOpðn�1=2Þ and importantly from Lemma 5, we have for uniformly for t in Oðn�1=2Þ neighborhood oft0 ffiffiffi

np ðTnðtÞ�LnðtÞÞ ¼

ffiffiffin

pTnðtÞ�

ffiffiffin

pTnðt0Þ�

ffiffiffin

pT ′ðt0Þðt�t0Þ ¼ opð1Þ:

Since Ln is a line uniformly approximating Tn in Oðn�1=2Þ neighborhood of t0, its zero and the “zero” of Tn, tn, are within thesame order of deviation. Consequently, this leads us to claim that tn ¼ tnnþopðn�1=2Þ and by substituting tnn with tn is (6.6),the proof of the lemma is complete. □

Theorem 5. Assume A1–A4 hold and suppose that t0 is the unique split point and T ′ðt0Þo0. Then as n-1,

ffiffiffin

ptn�t0ð Þ ) N 0;

st0T′ðt0Þ2

!;

where the non-negative st is as defined in Theorem 3.

7. Numerical example

Note that the limit theorems presented in the preceding sections are valid for ϕ-mixing sequences too since ϕ-mixingrepresents a stronger condition than β-mixing, i.e. a ϕ-mixing sequence is also β-mixing. It is well-known that for stationaryGaussian sequences ϕ-mixing is equivalent to m-dependence for a positive integer m (see for e.g. Bradley, 2007). Bearing inmind assumptions A1–A3, we shall consider a 2-dependent Gaussian sequence defined as

Xk ¼ Yk�2þYk�1þYk

where Yk is an i.i.d. Nð0;1=3Þ sequence. Therefore, fXigiZ1 is a stationary sequence from Nð0;1Þ satisfying a ϕ-mixingcondition. Since the density is unimodal and symmetric it is easy to note that the true split point t0 ¼ 0, Fðt0Þ ¼ 0:5,μlð0Þ � �0:4 and μuð0Þ � 0:4. For the variance s0, we will employ the expression given in Billingsley (1968, p. 174) forϕ-mixing sequences: s0 ¼ Eðξ00Þ2þ2∑1

k ¼ 1Eðξ00ξk0Þ. Based on Eqs. (6.1) and (6.2) and keeping in mind that Xi is a 2-dependentsequence, straightforward calculation gives s0 ¼ 3. Now, note that

T′ tð Þ ¼ tf ðtÞFðtÞ �

f ðtÞ R t�1 x dF

F2ðtÞ� tf ðtÞ

1�FðtÞ þf ðtÞ R1t x dF

ð1�FðtÞÞ2�2;

and hence T ′ð0Þ ¼ 4f ð0Þ½R 0�1 x dFþ R10 x dF��2� �0:73; hence, assumption that T′ðt0Þo0 in the statement of Theorem 5 issatisfied. As a consequence, we have that the asymptotic variance of

ffiffiffin

p ðtn�t0Þ ¼ffiffiffin

ptn is approximately 5.67. Table 1

corroborates the theoretical results.

8. Concluding remarks

For the i.i.d. case, the results in this paper can be obtained without much effort since the framework is a story oftruncated-at-fixed-level means. The results can be found in the dissertation of Bharath (2012). The partial-sums nature ofthe sample criterion function is an attractive feature when exploring extensions of the present framework to problems inhigher dimensions; in contrast, the framework in Bharath et al. (2013) is conceivably harder owing to the usage of orderstatistics. Some interesting theoretical points arise; however, necessary and sufficient conditions on F guaranteeinguniqueness of the true split point are unknown and are of interest since all limit results are obtained assuming theiruniqueness; the presence of Fn and ð1�FnÞ in the denominator in the expression of Tn bring about some disagreeabletechnical issues regarding the definition of Tn. Their presence is justifiably necessary since they ensure that the sample splitpoint remains invariant to scaling and translations of the data.

Table 1Simulated mean and variance of

ffiffiffin

ptn for different sample sizes for the 2-dependent sequence; 1000 simulations were performed for each sample size.

Sample sizes n¼100 n¼300 n¼1000 n¼5000

Simulated mean 0.004 0.007 0.003 �0.001Simulated variance 5.63 5.76 5.71 5.69

K. Bharath / Journal of Statistical Planning and Inference 147 (2014) 84–9494

Acknowledgments

The author is grateful to Vladimir Pozdnyakov and Dipak Dey for the useful discussions and guidance. He would also liketo thank the referees for a detailed review; in particular, he thanks one referee for a meticulous review which improved theoverall exposition.

References

Arcones, M.A., Yu, B., 1994. Central limit theorems for empirical and U-processes of stationary mixing sequences. J. Theoret. Probab. 7, 47–71.Bharath, K., 2012. Asymptotics of Clustering Criteria for Smooth Distributions (Ph.D. dissertation). University of Connecticut.Bharath, K., Pozdnyakov, V., Dey, D.K., 2013. Asymptotics of a clustering criterion for smooth distributions. Electron. J. Statist. 7, 1078–1093.Billingsley, P., 1968. Convergence of Probability Measures. John Wiley and Sons, New York.Bradley, R.C., 2007. Introduction to Strong Mixing Conditions, vol. 1. Kendrick Press, Heber City, UT.Cortes, C., Vapnik, V., 1995. Support-vector networks. Mach. Learn. 20, 273–297.Doukhan, P., 1994. Mixing: Properties and Examples. Lecture Notes in Statistics. Springer-Verlag.Doukhan, P., Massart, P., Rio, E., 1994. The functional central limit theorem for strongly mixing processes. Ann. Inst. H. Poincare 30, 63–82.Hartigan, J., 1978. Asymptotic distributions for clustering criteria. Ann. Statist. 6, 117–131.Mohri, M., Rostamizadeh, A., 2010. Stability bounds for stationary ϕ-mixing and β-mixing processes. J. Mach. Learn. Res. 11, 661–686.Pollard, D., 1984. Convergence of Stochastic Processes. Springer-Verlag, New York.Rio, E., 2000. Théorie asymptotique des processus aléatoires faiblement dépenedantes. Collection Mathématiques and Applications. Springer, Berlin.Saunders, C., Gammerman, A., Vovk, V., 1998. Proceedings of the Fifteenth International Conference on Machine Learning. Morgan-Kaufmann Publishers, Inc.van der Vaart, A.W., 2000. Asymptotic Statistics. Cambridge University Press, Cambridge.Vapnik, V., 1998. Statistical Learning Theory. Wiley-Interscience, New York.Vidyasagar, M., 2003. Learning and Generalization: With Applications to Neural Networks. Springer.Yu, B., 1994. Rates of convergence for empirical processes of stationary mixing sequences. Ann. Probab. 22, 94–116.