Lecture11-Dichotomyandgrowthfunctionbloch.ece.gatech.edu/ece6254sp19/lecture11-v1.pdf ·...

ECE 6254 Spring 2019 Matthieu Bloch - February 12, 2019 - revised March 19, 2019

Lecture 11 - Dichotomy and growth function

1 Motivation

For a hypothesis set H with |H| = M and h∗ = argmaxh∈H R̂N (h), we have shown earlier that

∀ϵ > 0 P(∣∣∣R̂N (h∗)−R(h∗)

∣∣∣ ⩾ ϵ)⩽ 2M exp(−2Nϵ2). (1)

In particular, the factor M is the result of the union bound, which we used to show that for ϵ > 0

P(∣∣∣R̂N (h∗)−R(h∗)

∣∣∣ ⩾ ϵ)⩽ P

(maxh∈H

∣∣∣R̂N (h)−R(h)∣∣∣ ⩾ ϵ

)(2)

⩽M∑j=1

P(∣∣∣R̂N (hj)−R(hj)

∣∣∣ ⩾ ϵ). (3)

The second inequality is tight when the events Ej ≜ {∣∣∣R̂N (hj)−R(hj)

∣∣∣ ⩾ ϵ} are disjoint, but thisis rarely the case in our classification setup. This is illustrated in Fig. 1 below, where the two classifiershown are distinct but have exactly the same empirical risk on the training set.

h1<latexit sha1_base64="RrYeGxR1nv9WBvIMYkiso1hFLdg=">AAACinicnVFNb9NAEN24QEso0MKRi0WExAHFdqnUIi6V4MCBQxFNGymNovFkHK+6H+7uGDWy+hN6hd/Gv2Ht5EBbToy0q6c37+3MzuSVkp7T9Hcv2njw8NHm1uP+k+2nz57v7L449bZ2SCO0yrpxDp6UNDRiyYrGlSPQuaKz/OJTmz/7Qc5La054WdFUw8LIQiJwoL6Xs2y2M0iHaRfxfZCtwUCs43i227s8n1usNRlGBd5PsrTiaQOOJSq67p/XnirAC1jQJEADmvy06Xq9jt8EZh4X1oVjOO7Yvx0NaO+XOg9KDVz6u7mWfOdZg1u6+b9Ek5qLw2kjTVUzGVxVLGoVs43bCcRz6QhZLQMAdDI0HWMJDpDDnG6VWzioSolX4Uvdc00y8kGThOlaLJOvcELjhOkqUdDe4NDq1T0zlrsJD0PiP+xFbbC1+5W/H5aU3V3JfXC6N8zeD9Nv+4Ojw/W6tsQr8Vq8FZk4EEfiizgWI4FiIW7ET/Er2o72og/Rx5U06q09L8WtiD7/Ad8IyCY=</latexit>

h2<latexit sha1_base64="/81gil22yZ4If2ZvBOor6gybdic=">AAACinicnVFNb9NAEN2YrxIKtHDsxSJC4oBiO1SiiEslOHDooYimjZRG0XgyjlfdD3d3jBpZ/Qlc4bfxb1g7OdCWEyPt6unNezuzM3mlpOc0/d2L7t1/8PDR1uP+k+2nz57v7L449bZ2SGO0yrpJDp6UNDRmyYomlSPQuaKz/OJTmz/7Ts5La054VdFMw9LIQiJwoL6V89F8Z5AO0y7iuyDbgIHYxPF8t3d5vrBYazKMCryfZmnFswYcS1R03T+vPVWAF7CkaYAGNPlZ0/V6Hb8OzCIurAvHcNyxfzsa0N6vdB6UGrj0t3Mt+dazBrdyi3+JpjUXB7NGmqpmMriuWNQqZhu3E4gX0hGyWgUA6GRoOsYSHCCHOd0ot3RQlRKvwpe655pk7IMmCdO1WCZHcEKThOkqUdDe4NDq9T03lrsJD0PiP+xFbbC1+7W/H5aU3V7JXXA6GmbvhunX/cHhwWZdW2JPvBJvRCbei0PxRRyLsUCxFD/ET/Er2o5G0Yfo41oa9Tael+JGRJ//AOEsyCc=</latexit>

Figure 1: Two distinct classsifiers with the same empirical risk

This observations suggests that our bound might be extremely loose and that |H| may not nec-essarily be the right measure of the richness of the hypothesis set H. Most of our work in the nextfew lectures will be devoted to finding a suitable replacement for |H|, which will enable use to provea generalization bound even in settings for which |H| = ∞, as is the case for linear classifiers.

2 Dichotomy and growth function

Motivated by the situation in Fig. 1, where many classifier have the same empirical risk, we willattempt to assess the number of hypotheses that lead to distinct labelings for a given dataset. In-tuitively, we are hoping that the number of distinct labelings is quantity that better captures therichness of the hypothesis class H. Formally, we introduce the notion of dichotomy.

Definition 2.1 (Dichotomy). For a datasetD ≜ {xi}Ni=1 and set of hypothesesH, the set of dichotomiesgenerated by H on D is the set of labelings that can be generated by classfiers in H on the dataset, i.e.,

H({xi}Ni=1) ≜ {{h(xi)}Ni=1 : h ∈ H}. (4)

1


Note that many sets {{h(xi)}Ni=1 for distinct h are actually identical because the labelings in-duced on the dataset are identical. By definition, for our binary labeling problem,

∣∣H({xi}Ni=1)∣∣ ⩽

2N and in general∣∣H({xi}Ni=1)

∣∣ ≪ |H|. Unfortunately,∣∣H({xi}Ni=1)

∣∣ is not a particularly usefulquantity because it is not only potentially difficult to compute but also dependent on a specificdataset. This motivates the definition of the growth function as follows.

Definition 2.2 (Growth function). For a set of hypotheses H, the growth function of H is

mH(N) ≜ max{xi}N

i=1

∣∣H({xi}Ni=1)∣∣ . (5)

Note that the growth function depends on the number of datapoints N but not on the exactdatapoints {xi}Ni=1. The growth function measures the maximum number of dichotomies that Hcan generate over all possible datasets, and by definition, it still holds that mH(N) ⩽ 2N .

Example 2.3 (Positive rays). Consider a binary classification problem in R with the set of positive rays

H ≜ {ha : R → {±1} : x 7→ sign(x− a)|a ∈ R}. (6)

As illustrated below, the threshold a defines a classifier such that all points to the left are assigned label−1 while all points to the right are assigned label +1.

h(x) = +1<latexit sha1_base64="kC6zGsGH/3h1YhUvcuaF/NP8pF8=">AAACkXicnVFNa9tAEF2raZu4H0maYy8ippDSYkltoekhYNJLoTkkECemtjGj8cpash/q7qjYiPyLXNP/1X+TlexDvk4d2OXx5r2d2Zm0kMJRHP9rBU/Wnj57vr7RfvHy1evNre03Z86UFnkfjTR2kILjUmjeJ0GSDwrLQaWSn6cX3+v8+R9unTD6lBYFHyuYaZEJBPLUr3xvlGbz9wcfkslWJ+7GTYQPQbICHbaK48l26/doarBUXBNKcG6YxAWNK7AkUPLL9qh0vAC8gBkfeqhBcTeumpYvw3eemYaZsf5oChv2tqMC5dxCpV6pgHJ3P1eTHx0psAs7fUw0LCnbH1dCFyVxjcuKWSlDMmE9iHAqLEeSCw8ArfBNh5iDBSQ/rjvlZhaKXODcf6l5ror6zmsiP2SDeXQEp3wQEZ9HEuobLBq1vCfaUDPork/8hz0rNdZ2t/S3/ZKS+yt5CM4+dZPP3fjkS6e3v1rXOnvLdtkeS9hX1mM/2DHrM2SaXbFr9jfYCb4FveBwKQ1aK88OuxPBzxsmY8pi</latexit>h(x) = �1

<latexit sha1_base64="vSzcr0ZlzButph2AwaWF//27KBE=">AAACkXicnVFNa9tAEF2raZu4H0maYy8ippBCa0ltoekhYNJLoTkkECemtjGj8cpash/q7qjYiPyLXNP/1X+TlexDvk4d2OXx5r2d2Zm0kMJRHP9rBU/Wnj57vr7RfvHy1evNre03Z86UFnkfjTR2kILjUmjeJ0GSDwrLQaWSn6cX3+v8+R9unTD6lBYFHyuYaZEJBPLUr3xvlGbz9wcfk8lWJ+7GTYQPQbICHbaK48l26/doarBUXBNKcG6YxAWNK7AkUPLL9qh0vAC8gBkfeqhBcTeumpYvw3eemYaZsf5oChv2tqMC5dxCpV6pgHJ3P1eTHxwpsAs7fUw0LCnbH1dCFyVxjcuKWSlDMmE9iHAqLEeSCw8ArfBNh5iDBSQ/rjvlZhaKXODcf6l5ror6zmsiP2SDeXQEp3wQEZ9HEuobLBq1vCfaUDPork/8hz0rNdZ2t/S3/ZKS+yt5CM4+dZPP3fjkS6e3v1rXOnvLdtkeS9hX1mM/2DHrM2SaXbFr9jfYCb4FveBwKQ1aK88OuxPBzxsqrcpk</latexit>

x1<latexit sha1_base64="rXlEeD1HA+8JeR+6WHLp1acK6yc=">AAACjXicnVFLb9NAEN6YVwmPPjhysYiQOKDYpkX0gKpKHODAoUhNGimJovFknCzdh9kdo0RW/wNX+Gf8G9ZODrTlxEi7+vTN9+3MzuSlkp7T9HcnunP33v0HOw+7jx4/ebq7t38w9LZySAO0yrpRDp6UNDRgyYpGpSPQuaKL/PJDk7/4Ts5La855XdJUw8LIQiJwoIaTvFjNstleL+2nbcS3QbYFPbGNs9l+59tkbrHSZBgVeD/O0pKnNTiWqOiqO6k8lYCXsKBxgAY0+WndtnsVvwzMPC6sC8dw3LJ/O2rQ3q91HpQaeOlv5hrytWcNbu3m/xKNKy6Op7U0ZcVkcFOxqFTMNm6GEM+lI2S1DgDQydB0jEtwgBxGda3cwkG5lLgKX2qfq5OBD5okDNjiMvkM5zRKmFaJguYGh1Zv7pmx3A65HxL/YS8qg43db/zdsKTs5kpug+GbfnbYT78c9U6Pt+vaEc/FC/FKZOKdOBWfxJkYCBRfxQ/xU/yKdqO30fvoZCONOlvPM3Etoo9/AOfJyXg=</latexit>

x2<latexit sha1_base64="026RsCYQOQS9oRoYTAuWKc07tZ0=">AAACjXicnVFNb9NAEN2YrxI+2sKRi0WExAHFdgG1B4QqcYADhyI1aaQkisaTcbLtfri7Y5TI6n/gCv+Mf8PayYG2nBhpV09v3tuZnclLJT2n6e9OdOfuvfsPdh52Hz1+8nR3b//Z0NvKIQ3QKutGOXhS0tCAJSsalY5A54rO8otPTf7sOzkvrTnldUlTDQsjC4nAgRpO8mI1O5jt9dJ+2kZ8G2Rb0BPbOJntdy4nc4uVJsOowPtxlpY8rcGxREVX3UnlqQS8gAWNAzSgyU/rtt2r+FVg5nFhXTiG45b921GD9n6t86DUwEt/M9eQbzxrcGs3/5doXHFxNK2lKSsmg5uKRaVitnEzhHguHSGrdQCAToamY1yCA+QwqmvlFg7KpcRV+FL7XJ0MfNAkYcAWl8lXOKVRwrRKFDQ3OLR6c8+M5XbI/ZD4D3tRGWzsfuPvhiVlN1dyGwwP+tnbfvrtXe/4aLuuHfFCvBSvRSYOxbH4Ik7EQKA4Fz/ET/Er2o3eRx+ijxtp1Nl6notrEX3+A+ntyXk=</latexit>

xN<latexit sha1_base64="mrpuzgfM+oSMURdq7VzVWeTgGdM=">AAACjXicnVFNb9NAEN2Yj5bw0RaOXCwiJA4otguIHhCqxAEOCBWpSSMlUTSejJOl++HujlEiq/+BK/wz/g1rJwfacmKkXT29eW9ndiYvlfScpr870a3bd+7u7N7r3n/w8NHe/sHjobeVQxqgVdaNcvCkpKEBS1Y0Kh2BzhWd5ecfmvzZd3JeWnPK65KmGhZGFhKBAzWc5MVq9mW230v7aRvxTZBtQU9s42R20LmYzC1WmgyjAu/HWVrytAbHEhVddieVpxLwHBY0DtCAJj+t23Yv4+eBmceFdeEYjlv2b0cN2vu1zoNSAy/99VxDvvSswa3d/F+iccXF0bSWpqyYDG4qFpWK2cbNEOK5dISs1gEAOhmajnEJDpDDqK6UWzgolxJX4Uvtc3Uy8EGThAFbXCaf4ZRGCdMqUdDc4NDqzT0zltsh90PiP+xFZbCx+42/G5aUXV/JTTA87Gev+unX173jo+26dsVT8Uy8EJl4K47FJ3EiBgLFN/FD/BS/or3oTfQuer+RRp2t54m4EtHHPyXsyZU=</latexit>

xN�1<latexit sha1_base64="vktV1VZbG1xh8suBPZq30+sWwHg=">AAACkXicnVFNb9NAEN2YrxK+WnrkYhEhcYDYLkiUWwQXJBAqUtNGJFE0noyTVffD7I5RIiv/giv8L/4NaycH2nJipF09vXlvZ3YmL5X0nKa/O9GNm7du39m72713/8HDR/sHj8+8rRzSEK2ybpSDJyUNDVmyolHpCHSu6Dy/eN/kz7+T89KaU16XNNWwMLKQCByor5O8WM3qzy+zzWy/l/bTNuLrINuBntjFyeyg820yt1hpMowKvB9nacnTGhxLVLTpTipPJeAFLGgcoAFNflq3LW/iZ4GZx4V14RiOW/ZvRw3a+7XOg1IDL/3VXEO+8KzBrd38X6JxxcXxtJamrJgMbisWlYrZxs0g4rl0hKzWAQA6GZqOcQkOkMO4LpVbOCiXElfhS+1zdTL0QZOEIVtcJp/glEYJ0ypR0Nzg0OrtPTOW20H3Q+I/7EVlsLH7rb8blpRdXcl1cHbUz1710y+ve4Pj3br2xBPxVDwXmXgjBuKDOBFDgcKIH+Kn+BUdRm+jQfRuK406O8+huBTRxz+jwssT</latexit>

a<latexit sha1_base64="ZRV5Kupa9N7DS52N8Z/Y7am4U1Q=">AAACiHicnVFNT9tAEN0YSmlKKdBjLxYRUg8otttK0BtqLxx6AIlApBBF48k4WbEf7u64IrL4Bb22P67/hrWTQ4GeGGlHTzPv7XzlpZKe0/RvJ1pbf7HxcvNV9/XWm+23O7t7l95WDmmAVlk3zMGTkoYGLFnRsHQEOld0ld98a/JXP8l5ac0FL0oaa5gZWUgEDqFzmOz00n7aWvwUZCvQEys7m+x2flxPLVaaDKMC70dZWvK4BscSFd11rytPJeANzGgUoAFNfly3nd7FByEyjQvrwjMct9F/FTVo7xc6D0wNPPePc03w0LMGt3DT/5FGFRfH41qasmIyuKxYVCpmGzfzx1PpCFktAgB0MjQd4xwcIIctPSg3c1DOJd6Gkdrv6mTgAycJu7U4T77DBQ0TpttEQePBodVLPzGW2/32Q+IZ8qIy2Mj9Ut8NR8oen+QpuPzYzz710/PPvZPj1bk2xXuxLz6ITByJE3EqzsRAoCDxS/wWf6JulEZH0ZclNeqsNO/EA4u+3gNPhMd7</latexit>

Although H = ∞, the number of dichotomies is still finite, and one can actually compute the growthfunction exactly. In general, this is challenging because we need to identify the worst case dataset thatgenerates the highest number of dichotomies; here, this is only tractable because the situation is simple.

Without losing generality, we can assume that all N points {xi}Ni=1 are distinct. Let us introducex0 ≜ −∞ and xN+1 ≜ ∞. For any i ⩾ 0, all classifiers ha with xi ⩽ a < xi+1 induce thesame labeling. Consequently, the number of distinct labelings is at most N + 1 and mH(N) = N +1. Interestingly, the growth function is growing polynomially in N , which is much slower than theexponential growth 2N allowed by the upper bound.

Example 2.4 (Positive intervals). Consider a binary classification in R with the set of positive intervals

H ≜ {ha,b : R → {±1} : x 7→ 1{x ∈ [a; b]} − 1{x /∈ [a; b]} |a < b ∈ R}. (7)

As illustrated below, the thresholds a < b define a classifier such that all points with [a; b] are assignedlabel +1 while all points outside are assigned label −1.

h(x) = +1<latexit sha1_base64="kC6zGsGH/3h1YhUvcuaF/NP8pF8=">AAACkXicnVFNa9tAEF2raZu4H0maYy8ippDSYkltoekhYNJLoTkkECemtjGj8cpash/q7qjYiPyLXNP/1X+TlexDvk4d2OXx5r2d2Zm0kMJRHP9rBU/Wnj57vr7RfvHy1evNre03Z86UFnkfjTR2kILjUmjeJ0GSDwrLQaWSn6cX3+v8+R9unTD6lBYFHyuYaZEJBPLUr3xvlGbz9wcfkslWJ+7GTYQPQbICHbaK48l26/doarBUXBNKcG6YxAWNK7AkUPLL9qh0vAC8gBkfeqhBcTeumpYvw3eemYaZsf5oChv2tqMC5dxCpV6pgHJ3P1eTHx0psAs7fUw0LCnbH1dCFyVxjcuKWSlDMmE9iHAqLEeSCw8ArfBNh5iDBSQ/rjvlZhaKXODcf6l5ror6zmsiP2SDeXQEp3wQEZ9HEuobLBq1vCfaUDPork/8hz0rNdZ2t/S3/ZKS+yt5CM4+dZPP3fjkS6e3v1rXOnvLdtkeS9hX1mM/2DHrM2SaXbFr9jfYCb4FveBwKQ1aK88OuxPBzxsmY8pi</latexit>

h(x) = �1<latexit sha1_base64="vSzcr0ZlzButph2AwaWF//27KBE=">AAACkXicnVFNa9tAEF2raZu4H0maYy8ippBCa0ltoekhYNJLoTkkECemtjGj8cpash/q7qjYiPyLXNP/1X+TlexDvk4d2OXx5r2d2Zm0kMJRHP9rBU/Wnj57vr7RfvHy1evNre03Z86UFnkfjTR2kILjUmjeJ0GSDwrLQaWSn6cX3+v8+R9unTD6lBYFHyuYaZEJBPLUr3xvlGbz9wcfk8lWJ+7GTYQPQbICHbaK48l26/doarBUXBNKcG6YxAWNK7AkUPLL9qh0vAC8gBkfeqhBcTeumpYvw3eemYaZsf5oChv2tqMC5dxCpV6pgHJ3P1eTHxwpsAs7fUw0LCnbH1dCFyVxjcuKWSlDMmE9iHAqLEeSCw8ArfBNh5iDBSQ/rjvlZhaKXODcf6l5ror6zmsiP2SDeXQEp3wQEZ9HEuobLBq1vCfaUDPork/8hz0rNdZ2t/S3/ZKS+yt5CM4+dZPP3fjkS6e3v1rXOnvLdtkeS9hX1mM/2DHrM2SaXbFr9jfYCb4FveBwKQ1aK88OuxPBzxsqrcpk</latexit>





a<latexit sha1_base64="ZRV5Kupa9N7DS52N8Z/Y7am4U1Q=">AAACiHicnVFNT9tAEN0YSmlKKdBjLxYRUg8otttK0BtqLxx6AIlApBBF48k4WbEf7u64IrL4Bb22P67/hrWTQ4GeGGlHTzPv7XzlpZKe0/RvJ1pbf7HxcvNV9/XWm+23O7t7l95WDmmAVlk3zMGTkoYGLFnRsHQEOld0ld98a/JXP8l5ac0FL0oaa5gZWUgEDqFzmOz00n7aWvwUZCvQEys7m+x2flxPLVaaDKMC70dZWvK4BscSFd11rytPJeANzGgUoAFNfly3nd7FByEyjQvrwjMct9F/FTVo7xc6D0wNPPePc03w0LMGt3DT/5FGFRfH41qasmIyuKxYVCpmGzfzx1PpCFktAgB0MjQd4xwcIIctPSg3c1DOJd6Gkdrv6mTgAycJu7U4T77DBQ0TpttEQePBodVLPzGW2/32Q+IZ8qIy2Mj9Ut8NR8oen+QpuPzYzz710/PPvZPj1bk2xXuxLz6ITByJE3EqzsRAoCDxS/wWf6JulEZH0ZclNeqsNO/EA4u+3gNPhMd7</latexit>

h(x) = �1<latexit sha1_base64="vSzcr0ZlzButph2AwaWF//27KBE=">AAACkXicnVFNa9tAEF2raZu4H0maYy8ippBCa0ltoekhYNJLoTkkECemtjGj8cpash/q7qjYiPyLXNP/1X+TlexDvk4d2OXx5r2d2Zm0kMJRHP9rBU/Wnj57vr7RfvHy1evNre03Z86UFnkfjTR2kILjUmjeJ0GSDwrLQaWSn6cX3+v8+R9unTD6lBYFHyuYaZEJBPLUr3xvlGbz9wcfk8lWJ+7GTYQPQbICHbaK48l26/doarBUXBNKcG6YxAWNK7AkUPLL9qh0vAC8gBkfeqhBcTeumpYvw3eemYaZsf5oChv2tqMC5dxCpV6pgHJ3P1eTHxwpsAs7fUw0LCnbH1dCFyVxjcuKWSlDMmE9iHAqLEeSCw8ArfBNh5iDBSQ/rjvlZhaKXODcf6l5ror6zmsiP2SDeXQEp3wQEZ9HEuobLBq1vCfaUDPork/8hz0rNdZ2t/S3/ZKS+yt5CM4+dZPP3fjkS6e3v1rXOnvLdtkeS9hX1mM/2DHrM2SaXbFr9jfYCb4FveBwKQ1aK88OuxPBzxsqrcpk</latexit>

b<latexit sha1_base64="68M/pnHiaHBAPoJHxflz08B9SiU=">AAACiHicnVFNT9tAEN0YSmlKKdBjLxYRUg8otttK0BtqLxx6AIlApBBF48k4WbEf7u64IrL4Bb22P67/hrWTQ4GeGGlHTzPv7XzlpZKe0/RvJ1pbf7HxcvNV9/XWm+23O7t7l95WDmmAVlk3zMGTkoYGLFnRsHQEOld0ld98a/JXP8l5ac0FL0oaa5gZWUgEDqHzfLLTS/tpa/FTkK1AT6zsbLLb+XE9tVhpMowKvB9lacnjGhxLVHTXva48lYA3MKNRgAY0+XHddnoXH4TINC6sC89w3Eb/VdSgvV/oPDA18Nw/zjXBQ88a3MJN/0caVVwcj2tpyorJ4LJiUamYbdzMH0+lI2S1CADQydB0jHNwgBy29KDczEE5l3gbRmq/q5OBD5wk7NbiPPkOFzRMmG4TBY0Hh1Yv/cRYbvfbD4lnyIvKYCP3S303HCl7fJKn4PJjP/vUT88/906OV+faFO/FvvggMnEkTsSpOBMDgYLEL/Fb/Im6URodRV+W1Kiz0rwTDyz6eg9RqMd8</latexit>

Again, this is a situation for which we can compute the growth function exactly. Without loss of generality,we assume that all N datapoints are distinct and we introduce x0 ≜ −∞ and xN+1 ≜ ∞. We need tobe a bit more careful when counting dichotomies:

• If x0 < a < b ⩽ x1, all classifiers hab induce an all-−1 labeling;

• for any 0 ⩽ i < j ⩽ N , all classifiers hab such that xi ⩽ a ⩽ xi+1 < xj ⩽ b ⩽ xj+1 inducethe same labelings;

2


• for any 0 ⩽ i ⩽ N , all classifiers hab such that xi ⩽ a < b < xi+1 induce again an all-−1labeling.

Consequently, the number of classifiers is 1 +(N+12

)and mH(N) = N2

2 + N2 + 1, which grows again

polynomially in N .

Example 2.5 (Convex sets). Consider a binary classification in R2 with the set

H ≜ {h : R2 → {±1}|{x ∈ R2 : h(x) = +1} is convex}. (8)

Consider a set of N distinct points distributed on the unit circle, as illustrated below.





h(x) = +1<latexit sha1_base64="kC6zGsGH/3h1YhUvcuaF/NP8pF8=">AAACkXicnVFNa9tAEF2raZu4H0maYy8ippDSYkltoekhYNJLoTkkECemtjGj8cpash/q7qjYiPyLXNP/1X+TlexDvk4d2OXx5r2d2Zm0kMJRHP9rBU/Wnj57vr7RfvHy1evNre03Z86UFnkfjTR2kILjUmjeJ0GSDwrLQaWSn6cX3+v8+R9unTD6lBYFHyuYaZEJBPLUr3xvlGbz9wcfkslWJ+7GTYQPQbICHbaK48l26/doarBUXBNKcG6YxAWNK7AkUPLL9qh0vAC8gBkfeqhBcTeumpYvw3eemYaZsf5oChv2tqMC5dxCpV6pgHJ3P1eTHx0psAs7fUw0LCnbH1dCFyVxjcuKWSlDMmE9iHAqLEeSCw8ArfBNh5iDBSQ/rjvlZhaKXODcf6l5ror6zmsiP2SDeXQEp3wQEZ9HEuobLBq1vCfaUDPork/8hz0rNdZ2t/S3/ZKS+yt5CM4+dZPP3fjkS6e3v1rXOnvLdtkeS9hX1mM/2DHrM2SaXbFr9jfYCb4FveBwKQ1aK88OuxPBzxsmY8pi</latexit>

Notice that irrespective of the labeling of the datapoints, the datapoints for which h(xi) = +1 define thevertices of a polytope, which is convex. Said differently, irrespective of the labeling there exists h ∈ H thatgenerates the labeling. Therefore, by definition, mH ⩾ 2N ; since we also know that mH ⩽ 2N , it musthold that mH = 2N .

The three previous examples are not at all representative of a general situation because it is nearlyimpossible to compute the growth function exactly in most practical cases. As shown next, even forlinear classifiers this can become a formidable task.

Example 2.6 (Linear classifiers). Consider a binary classification in R2 with the set of linear classifiers

H ≜ {h : R2 → {±1} : x 7→ sign(w⊺x+ b)|w ∈ R2, b ∈ R} (9)

The challenge again is to identify the worst case dataset that generates the most dichotomies. We first notethat {x : w⊺x+ b = 0} = {x : −w⊺x+ b = 0}, so that a single line actually defines two classifiers.

For N = 3, we need to distinguish two cases. If all three points are aligned, all dichotomies exceptthose illustrated below are possible, we therefore obtain six dichotomies.



x3<latexit sha1_base64="sZRIh15Nrh4RPvrDhdfgN9iEUCU=">AAACjXicnVFNb9NAEN2YrxI+2sKRi0WExAHFNgW1B4QqcYADhyI1aaQkisaTcbLtfri7Y5TI6n/gCv+Mf8PayYG2nBhpV09v3tuZnclLJT2n6e9OdOfuvfsPdh52Hz1+8nR3b//Z0NvKIQ3QKutGOXhS0tCAJSsalY5A54rO8otPTf7sOzkvrTnldUlTDQsjC4nAgRpO8mI1O5jt9dJ+2kZ8G2Rb0BPbOJntdy4nc4uVJsOowPtxlpY8rcGxREVX3UnlqQS8gAWNAzSgyU/rtt2r+FVg5nFhXTiG45b921GD9n6t86DUwEt/M9eQbzxrcGs3/5doXHFxNK2lKSsmg5uKRaVitnEzhHguHSGrdQCAToamY1yCA+QwqmvlFg7KpcRV+FL7XJ0MfNAkYcAWl8lXOKVRwrRKFDQ3OLR6c8+M5XbI/ZD4D3tRGWzsfuPvhiVlN1dyGwzf9rODfvrtXe/4aLuuHfFCvBSvRSYOxbH4Ik7EQKA4Fz/ET/Er2o3eRx+ijxtp1Nl6notrEX3+A+wRyXo=</latexit>




If the three points are not aligned, they form the vertices of a polytope and any hyperplane cutting thepolytope will isolate one point. In addition, any hyperplane no cutting the polytope will assign the samelabel to all three points. Consequently, the number of dichotomies generated is 8 = 23.




Consequently, mH(3) = 8.

For N = 4, we need to distinguish even more cases. If all four points are aligned, all dichotomiesexcept those illustrated below are possible we therefore obtain 10 dichotomies.

3





x4<latexit sha1_base64="uV2CM2bfuxVhqGC6kqsuSQi+IAQ=">AAACjXicnVFNb9NAEN2YrxI+2sKRi0WExAHFNhS1B4QqcYADhyI1aaQkisaTcbLtfri7Y5TI6n/gCv+Mf8PayYG2nBhpV09v3tuZnclLJT2n6e9OdOfuvfsPdh52Hz1+8nR3b//Z0NvKIQ3QKutGOXhS0tCAJSsalY5A54rO8otPTf7sOzkvrTnldUlTDQsjC4nAgRpO8mI1O5jt9dJ+2kZ8G2Rb0BPbOJntdy4nc4uVJsOowPtxlpY8rcGxREVX3UnlqQS8gAWNAzSgyU/rtt2r+FVg5nFhXTiG45b921GD9n6t86DUwEt/M9eQbzxrcGs3/5doXHFxNK2lKSsmg5uKRaVitnEzhHguHSGrdQCAToamY1yCA+QwqmvlFg7KpcRV+FL7XJ0MfNAkYcAWl8lXOKVRwrRKFDQ3OLR6c8+M5XbI/ZD4D3tRGWzsfuPvhiVlN1dyGwzf9rN3/ezbQe/4aLuuHfFCvBSvRSYOxbH4Ik7EQKA4Fz/ET/Er2o3eRx+ijxtp1Nl6notrEX3+A+6HyXw=</latexit>





















If three out of four points are aligned, the four points form a 3-vertex polytope, and one point, say x2, ison the edge, say defined by x1 and x3. Any hyperplane cutting through the hyperplane cannot assign alabel to x2 that is distinct of both x1 and x3. Consequently, the dichotomies illustrated below cannot begenerated and we obtain 14 dichotomies.

















If no three out of four points are aligned, the four points could form a 4-vertex polytope, in which case ahyperplane cutting through the polytope cannot assign distinct labels to a vertex and all its neighbors. Thefour points could also form a 3-vertex polytope with a point in the interior, in which case a hyperplanecutting through the polytope cannot assign a label to the interior point distinct from all the vertices.Consequently, the dichotomies illustrated below cannot be generated and we obtain 14 dichotomies.

















Consequently, mH(4) = 14.

This last example illustrates the essentially combinatorial nature of the calculation of the growthfunction. As we will soon seen, we will conveniently only care about the scaling of the growthfunction with N in particular whether it is polynomial or exponential.

4

Lecture11-Dichotomyandgrowthfunctionbloch.ece.gatech.edu/ece6254sp19/lecture11-v1.pdf ·...

Documents

Transcript of Lecture11-Dichotomyandgrowthfunctionbloch.ece.gatech.edu/ece6254sp19/lecture11-v1.pdf ·...