Ι/Ο Data Εngineering

Kyriakos C. Chatzidimitriou - http://kyrcha.info 1

I/O Data Engineering“Garbage in, garbage out”

November 2016


Preamble• This work is a fusion of ideas and work (slides, text, images etc.) I found on the internet or had

and wrote on my own regarding the area of input/output data engineering in data mining and machine learning.

• Attribution:• The slides are based on the PowerPoint accompanying slides of the Data Mining, Practical Machine

Learning Tools, Witten et al., 4th ed., 2017 and in particular Chapter 8, available at: http://www.cs.waikato.ac.nz/ml/weka/book.html

• Slides from the Machine Learning MOOC by Prof. Andrew Ng: http://ml-class.org (PCA parts)• Slides from Learning from Data MOOC by Prof. Yaser S. Abu-Mustafa its support site:

http://work.caltech.edu/telecourse.html (The digits dataset and the non-linear transformation)• Slides from the Pattern Recognition class by Prof. Andreas L. Symeonidis, ECE department, Aristotle

University of Thessaloniki• A tutorial on Principal Component Analysis by Lindsay I. Smith, February 2002 (

http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf) • Introduction to Data Mining, Tan et al., 2006: http://www-users.cs.umn.edu/~kumar/dmbook/

November 2016

http://www.cs.waikato.ac.nz/ml/weka/book.html

http://ml-class.org/

http://work.caltech.edu/telecourse.html

http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf

http://www-users.cs.umn.edu/~kumar/dmbook/


Successful data mining: Just apply a learner! No?• Select the learning algorithm• Scheme/parameter selection/tuning

• Treat selection process as part of the learning process to avoid optimistic performance estimates

• Estimate the expected true performance of a learning scheme• Split• Cross-validation

• Data Engineering• Engineering the input data into a form suitable for the learning scheme chosen

• Data engineering to make learning possible or easier• Engineering the output to make it more effective

• Converting multi-class problems into two-class ones• Re-calibrating probability estimates

November 2016


Data Transformations – OutlineTopics covered:1. Attribute Selection2. Discretizing Numeric Attributes3. Data Projection4. Data Cleansing5. Transforming Multiple Classes to Binary Ones

November 2016


It is a jungle out there!

Data Transformations

Feature SelectionFeature Engineering

Data Engineering Dimensionality Reduction

Principal Components Analysis

Pre- and post-processing

Data CleansingETL

Feature LearningWrapper methods

Filter methods

Independent Component Analysis

Outlier Detection

November 2016


Attribute/Feature SelectionRemoving attributes that are not useful to the task at hand

November 2016


Motivation• Experiments have showed that adding useless attributes causes the performance of learning schemes

(decision tree and rules, linear regression, instance-based learners) to deteriorate• Adding a random binary variable effects:

• Divide-and-conquer tree learners and separate-and-conquer rule learners• If you reach depths at which only a small amount of data is available for picking a split, the random attribute will look good by

chance• C4.5 deterioration in performance 5-10% for 1 random variable

• Instance-based learners• Susceptible as well, reason: work in local neighborhoods• The number of training instances needed to produce a predetermined level of performance for instance-based learning increases

exponentially with the number of irrelevant attributes present• Naive Bayes

• Not susceptible• It assumes by design that all attributes are independent of one another, an assumption that is just right for random “distracter”

attributes• On the other hand: pays a heavy price in other ways because its operation is damaged by adding redundant attributes

• Independence “thrown out of the window”• Conclusion: Relevant attributes can also be harmful if they mislead the learning algorithm

November 2016


Advantages of Attribute Selection• Improves performance of learning algorithms• Speeds them up• Outweighed by the computation involved in attribute selection

• Yields a more compact, more easily interpretable representation

November 2016


Attribute Selection Types• Manually• The best way• Requires deep understanding of the learning problem and what the attributes

actually mean

• Filter-method – Scheme-Independent Attribute Selection• Make an independent assessment based on general characteristics of the

data

• Wrapper method – Scheme-Dependent Attribute Selection• Evaluate the subset using the machine learning algorithm that will ultimately

be employed for learning

November 2016


Scheme-Independent Attribute Selection• aka Filter approach to attribute selection:

assess attributes based on general characteristics of the data

• Attributes are selected in a manner that is independent of the target machine learning scheme

• One method: find smallest subset of attributes that separates data

• Another method: use a fast learning scheme that is different from the target learning scheme to find relevant attributes• E.g., use attributes selected by C4.5, or

coefficients of linear model, possibly applied recursively (recursive feature elimination)

November 2016

By Lucien Mousin - Own work, GFDL, https://commons.wikimedia.org/w/index.php?curid=37776286


Recursive Feature Elimination12…

F1 F2 F3

LearningAlgorithm

Ranking: F2 F1 F3

12…

F1 F2

LearningAlgorithm

Ranking: F1 F2 Final Ranking: F1 F2 F3

Learning algorithm should produce a ranking, i.e. a linear SVM, where ranks are based on the size of the coefficients

November 2016


Correlation-based Feature Selection (CFS)• Correlation between attributes measured by symmetric uncertainty:

where H is the entropy function:

• Goodness of subset of attributes measured by

where C is the class attribute, breaking ties in favour of smaller subsets.

]1,0[)()(

),()()(2),(

BHAHBAHBHAHBAU

j i j jij AAUCAU ),(),(

𝐻 ( 𝑋 ,𝑌 )=−∑𝑆𝑋

∑𝑆𝑌

𝑝 (𝑥 , 𝑦 ) log (𝑝 (𝑥 , 𝑦 ))𝐻 ( 𝑋 )=−∑𝑆𝑋

𝑝 (𝑥 ) log (𝑝 (𝑥 ))

November 2016


The Weather DataOutlook Temperature Humidity Windy Play

Sunny Hot High False No

Sunny Hot High True No

Overcast Hot High False Yes

Rainy Mild High False Yes

Rainy Cool Normal False Yes

Rainy Cool Normal True No

Overcast Cool Normal True Yes

Sunny Mild High False No

Sunny Cool Normal False Yes

Rainy Mild Normal False Yes

Sunny Mild Normal True Yes

Overcast Mild High True Yes

Overcast Hot Normal False Yes

Rainy Mild High True NoNovember 2016


Symmetric Uncertainty Example Calculation• H(Outlook) = - 5/14log(5/14) – 4/14log(4/14) – 5/14 log(5/14) = 1.577• H(Temperature) = - 4/14log(4/14) – 6/14log(6/14) – 4/14 log(4/14) = 1.556• H(Outlook, Temperature) =

- p(s,h)logp(s,h) - p(s,m)logp(s,m) - p(s,c)logp(s,c) - p(o,h)logp(o,h) - p(o,m)logp(o,m) - p(o,c)logp(o,c) - p(r,h)logp(r,h) - p(r,m)logp(r,m) - p(r,c)logp(r,c) = - 2/14log(2/14) – 2/14log(2/14) – 1/14 log(1/14) - 2/14log(2/14) – 1/14log(1/14) – 1/14 log(1/14) - 0/14log(0/14) – 3/14log(3/14) – 2/14 log(2/14) = 2.896

• U(Outlook, Temperature) = 2*(1.577 + 1.566 – 2.896)/(1.577 + 1.566) = 0.1512927

November 2016


Attribute subsets for weather dataThe number of possible attribute

subsets increases exponentially with the number of attributes, making

exhaustive search impractical on all but the simplest problems.

November 2016


Scheme-specific selection• Wrapper approach to attribute selection:

attributes are selected with target scheme in the loop

• Implement “wrapper” around learning scheme• Evaluation criterion: cross-validation performance

• Time consuming in general• greedy approach, k attributes, evaluation time

multiplied by a factor of k2, worst case• prior ranking of attributes, complexity linear in k

• Can use significance test (paired t-test) to stop cross-validation for a subset early if it is unlikely to “win” (race search)• Can be used with forward, backward selection,

prior ranking, or special-purpose schemata search• Efficient for decision tables and Naïve Bayes

(Selective Naïve Bayes)

November 2016

By Lastdreamer7591 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=37208688


Selective Naïve Bayes• Use the forward selection algorithm• Better able to detect a redundant attribute than backward elimination

• Use as metric the quality of an attribute to be simply the performance on the training set• We know that: Training set performance not a reliable indicator of

test set performance• But Naïve Bayes is less likely to overfit• Plus, as discussed, robust to random variables

November 2016


Complexity example• If I do 10-fold CV I must train the algorithm 10 times = 10• I should do also the 10-fold CV 10 times to obtain a more reliable

estimate = 10*10• If I have 10 features the total search space is 2^10 = 1024 different

subsets = 10*10*1024 = 102,400 • Then I should also tune the parameters of the learning algorithm…• Or should I do that before…

November 2016


Searching the attribute space• Number of attribute subsets is exponential in

the number of attributes• Common greedy approaches:

• forward selection• backward elimination

• More sophisticated strategies:• Bidirectional search• Best-first search:

• can find optimum solution, • does not just terminate when the performance starts

to drop keeps a list of all attribute subsets evaluated so far, sorted in order of the performance measure, so that it can revisit an earlier configuration

• Beam search: approximation to best-first search, keeps a truncated list

• Genetic algorithms

November 2016


DiscretizationTransforming numeric attributes into discrete ones

November 2016


Motivation• Essential if the task involves numeric attributes but the chosen

learning scheme can only handle categorical ones• Schemes that can handle numeric attributes often produce better

results, or work faster, if the attributes are pre-discretized.• The converse situation, in which categorical attributes must be

represented numerically, also occurs (although less often)

November 2016


Attribute discretization• Discretization can be useful even if a learning algorithm can be run on numeric

attributes directly• Avoids normality assumption in Naïve Bayes and clustering• Examples of discretization we have already encountered:

• Decision trees perform local discretization

• Global discretization can be advantageous because it is based on more data• Apply learner to

• k-valued discretized attribute or to• k – 1 binary attributes that code the cut points

• The latter approach often works better when learning decision trees or rule sets

November 2016


Discretization: unsupervised• Unsupervised discretization: determine

intervals without knowing class labels• When clustering, the only possible way!

• Two well-known strategies:• Equal-interval binning• Equal-frequency binning

(also called histogram equalization)• Unsupervised discretization is normally

inferior to supervised schemes when applied in classification tasks• But equal-frequency binning works well with

Naïve Bayes if the number of intervals is set to the square root of the size of dataset (proportional k-interval discretization)

Data Equal interval width

Equal frequency K-means

November 2016


Discretization: supervised• Classic approach to supervised discretization is entropy-based• This method builds a decision tree with pre-pruning on the attribute being discretized

• Uses entropy as splitting criterion• Uses the minimum description length principle as the stopping criterion for pre-pruning

• Works well: still the state of the art• To apply the minimum description length principle, the “theory” is

• the splitting point (can be coded in log2[N – 1] bits)• plus class distribution in each subset (a more involved expression)

• Description length is the number of bits needed for coding both the splitting point and the class distributions

• Compare description lengths before/after adding split

November 2016


Example: temperature attributePlayTemperature

Yes No Yes Yes Yes No No Yes Yes Yes No Yes Yes No64 65 68 69 70 71 72 72 75 75 80 81 83 85

November 2016


Final

It can be shown theoretically that a cut point that minimizes the information value will never occur between two instances of the same class

November 2016


Formula for MDL stopping criterion• Can be formulated in terms of the information gain• Assume we have N instances

• Original set: k classes, entropy E• First subset: k1 classes, entropy E1

• Second subset: k2 classes, entropy E2

• If the information gain is greater than the expression on the right, we continue splitting

• Results in no discretization intervals for the temperature attribute in the weather data• Fail to play a role in the final decision structure

NEkEkkE

NNgain

k221122 )23(log)1(log

November 2016


Supervised discretization: other methods• Can replace top-down procedure by bottom-up method• This bottom-up method has been applied in conjunction with the chi-squared

test• Continue to merge intervals until they become significantly different

• Can use dynamic programming to find optimum k-way split for given additive criterion• Requires time quadratic in the number of instances• But can be done in linear time if error rate is used instead of entropy• Error rate: count the number of errors that a discretization makes when predicting each

training instance’s class, assuming that each interval receives the majority class.• However, using error rate is generally not a good idea when discretizing an attribute as

we will see

November 2016


Error-based vs. entropy-based• Question:

could the best discretization ever have two adjacent intervals with the same class?• Wrong answer: No. For if so,• Collapse the two• Free up an interval• Use it somewhere else• (This is what error-based discretization will do)

• Right answer: Surprisingly, yes.• (and entropy-based discretization can do it)

November 2016


Error-based vs. entropy-basedA 2-class,2-attribute problem

Entropy-based discretization can detect change of class distribution (from 100% to 50%)

Class 1: a1 < 0.3 or if a1 < 0.7 and a2 < 0.5Class 2: otherwise

Best discretization

a2: no problema1: middle will have whatever labelhappens to occur most

November 2016


Data Projections and Dimensionality ReductionProjecting data into a more suitable space

November 2016


Motivation• Curse of Dimensionality• Visualization• Add new, synthetic attributes whose purpose is to present existing

information in a form that is suitable for the machine learning scheme to pick up on.

November 2016


Curse of Dimensionality• When dimensions increase, data

become increasingly sparse• Density and distance between

points which are important criteria for clustering and outlier detection loose their importance

• Create 500 points• Calculate the max and min distance between

any pair of points

November 2016


Projections• Definition: a projection is a kind of function or mapping that transforms data in

some way• Simple transformations can often make a large difference in performance• Example transformations (not necessarily for performance improvement):

• Difference of two date attributes age• Ratio of two numeric (ratio-scale) attributes

• Useful for algorithms doing axis parallel splits• Concatenating the values of nominal attributes• Encoding cluster membership• Adding noise to data• Removing data randomly or selectively• Obfuscating the data

• Anonymising

November 2016


Digits dataset

November 2016From: Learning from data MOOC, http://work.caltech.edu/telecourse.html


Input representation• ‘raw’ input x = (x0, x1, x2, …,

x256)• Linear model: (w0, w1, w2, …,

w256)

• Features: extract useful information, e.g.,• Intensity and symmetry x = (x0, x1,

x2)• Linear model: (w0, w1, w2)

From: Learning from data MOOC, http://work.caltech.edu/telecourse.htmlNovember 2016


Illustration of features

November 2016From: Learning from data MOOC, http://work.caltech.edu/telecourse.html


Another one

November 2016

From: Learning from data MOOC, http://work.caltech.edu/telecourse.html


Methods• Unsupervised• Principal Components Analysis (PCA)• Independent Component Analysis (ICA)• Random Projections

• Supervised• Partial Least Squares (PLS)• Linear Discriminant Analysis (LDA)

November 2016


Principal Components Analysisaka PCA

November 2016


Principal component analysis in a glance• Unsupervised method for identifying the important directions in a dataset• We can then rotate the data into the new coordinate system that is given by

those directions• Finally we can keep the new dimension that are of more importance• PCA is a method for dimensionality reduction• Algorithm:

1. Find direction (axis) of greatest variance2. Find direction of greatest variance that is perpendicular to previous direction and

repeat

• Implementation: find eigenvectors of the covariance matrix of the data• Eigenvectors (sorted by eigenvalues) are the directions

November 2016


PCA problem formulation

Reduce from 2-dimension to 1-dimension: Find a direction (a vector )onto which to project the data so as to minimize the projection error.Reduce from n-dimension to k-dimension: Find vectors onto which to project the data, so as to minimize the projection error.

November 2016


Data in a matrix form• Let n instances with d attributes. Every instance is described by d

numerical values.• We represent our data as a nd matrix A with real numbers.• We can use linear algebra to process the matrix

• Our goal is to produce a new nk matrix B such as:• It contains as much information as the original matrix A• To reveal something about the structure of data of A

November 2016


Principal Components• The first principal component is the direction of the axis with the

largest variance in the data• The second principal component is the next orthogonal direction with

the largest variance in the data• And so on..• The 1st PC contains the largest variance• The kth PC contains the kth fraction of variance• For n original dimensions, the covariance matrix is nxn and has up to n

eigenvectors. Thus, n PC.

November 2016


Example: 10-dimensional data

• Data is normally standardized or mean-centered for PCA• Can also apply this recursively in a tree learnerNovember 2016


PCA example• Dataset with 2 attributes x1 and x2.

November 2016


PCA example – Step 1: Get some data

November 2016

X - Data:x1 x22.5 2.40.5 0.72.2 2.91.9 2.23.1 3.02.3 2.7

2 1.61 1.1

1.5 1.61.1 0.9


Data Pre-processingPreprocessing (feature scaling/mean normalization):

Replace each with .If different features on different scales (e.g., size of house, number of bedrooms), scale features to have comparable range of values.

November 2016


PCA example – Step 2: Subtract the mean

November 2016

X’- Mean normalization:x1 x2

.69 .49-1.31 -1.21.39 .99.09 .29

1.29 1.09.49 .79.19 -.31-.81 -.81-.31 -.31

-.71 -1.01


Principal Component Analysis (PCA) algorithmReduce data from -dimensions to -dimensionsCompute “covariance matrix”:

Compute “eigenvectors” of matrix :[U,S,V] = svd(Sigma);

November 2016


PCA Example – Step 3: Calculate the covariance matrix

• Given that non-diagonal values are positive, we expect that x1 and x2 will increase together (+ sign of cov(x1, x2))

November 2016

S


Principal Component Analysis (PCA) algorithm

From , we get: [U,S,V] = svd(Sigma)

November 2016


PCA Example – Step 4: Compute the eigenvectors of S

• [U, D, V] = SVD(S)• The 1st eigenvector has an

eigenvalue of 1.2840277, while the 2nd an eigenvalue of 0.0490834• Eigenvectors are perpendicular to

each other: orthogonal

November 2016

U


Choosing k number of principal componentsTypically, choose to be smallest value so that

“99% of variance is retained”

(1%)

November 2016


Choosing k number of principal components[U,S,V] = svd(Sigma)Pick smallest value of for which

(99% of variance retained)

November 2016


PCA Example – Step 5: Choosing components• Choosing the 1st component will

retain more than 95% of the variance

November 2016


Principal Component Analysis (PCA) algorithmAfter mean normalization (ensure every feature has zero mean) and optionally feature scaling:

Sigma =

[U,S,V] = svd(Sigma);Ureduce = U(:,1:k);z = Ureduce’*x;

November 2016


PCA Example – Step 6: Deriving the new data• FinalData = RowFeatureVector x RowDataAdjust• RowDataAdjust = X’T

• Normalized with zero mean with every row being a dimension and every column a point (inverted form)

• RowFeatureVector = U’T

• Eigenvectors are in rows with the most important in the first row

November 2016


Supervised learning speedup

Extract inputs:Unlabeled dataset:

New training set:

Note: Mapping should be defined by running PCA only on the training set. This mapping can be applied as well to the examples and in the cross validation and test sets.

November 2016


ApplicationsApplication of PCA

- Compression- Reduce memory/disk needed to store data- Speed up learning algorithm

- Visualization

November 2016


Bad use of PCA: To prevent overfittingUse instead of to reduce the number of features to Thus, fewer features, less likely to overfit.

This might work OK, but isn’t a good way to address overfitting. Use regularization instead.

November 2016


PCA is sometimes used where it shouldn’t beDesign of ML system:

- Get training set- Run PCA to reduce in dimension to get- Train logistic regression on- Test on test set: Map to . Run on

How about doing the whole thing without using PCA?Before implementing PCA, first try running whatever you want to do with the original/raw data . Only if that doesn’t do what you want, then implement PCA and consider using .

November 2016


Data CleansingData Cleaning, Data Scrubbing, or Data Reconciliation

November 2016


What is data cleansing?• "Detect and remove errors and inconsistencies from data in order

to improve the quality of data" [Rahm]• "The process of detecting and correcting (or removing) corrupt or

inaccurate records from a record set, table, or database" [Wikipedia]

• Integral part of data processing and maintenance• Usually semi-automatic process, highly application specific

November 2016


How to• Necessity of getting to know your data: understanding the meaning of

all the different attributes, the conventions used in coding them, the significance of missing values and duplicate data, measurement noise, typographical errors, and the presence of systematic errors—even deliberate ones.• There are also automatic methods of cleansing data, of detecting

outliers, and of spotting anomalies, which we describe—including a class of techniques referred to as “one-class learning” in which only a single class of instances is available at training time.

November 2016


Data Cleansing vs. Data Validation• Data validation almost invariably means data is rejected from the

system at entry and is performed at entry time, rather than on batches of data.

• Example: Data validation in web forms

November 2016


Anomalies Classification• Syntactical Anomalies

• Lexical Errors (Gender: {M, M, F, 5' 8})• Domain format errors (Smith, John vs. Smith John)• Irregularities: non-uniform use of values, units, abbreviations (examples: different currencies in

the salaries, different use of abbreviations)

• Semantic Anomalies• Integrity constraint violations (AGE >= 0)• Contradictions (AGE vs. CURRENT_DATE - DATE_OF_BIRTH)• Duplicates• Invalid tuples

• Coverage Anomalies• Missing values• Missing tuples

November 2016


Data Quality Criteria: Accuracy + Uniqueness• Accuracy = Integrity + Consistency + Density• Integrity = Completeness + Validity• Completeness: M in D / M i.e. I should have 1000 tuples and I have 500 => 50% (Missing

values)• Validity: M in D / D, i.e. From the 500 tuples I have in D the 400 are valid => 80% (Illegal

values)• Consistency = Schema conformance + Uniformity• Schema conformance: tuples conforming to syntactical structure / overall number of tuples (if

in the database then it conforms)• Uniformity: attributes with no irregularities (non-uniform use of values) / total number of

attributes• Density: missing values in the tuples in D / total values in D

• Uniqueness: tuples of the same entity / total number of tuples

November 2016


Data Cleansing Operations1. Format adaptation for tuples and values2. Integrity constraint enforcement3. Derivation of missing values from existing ones4. Removing contradictions within or between tuples5. Merging and eliminating duplicates6. Detection of outliers, i.e. tuples and values having a high potential

of being invalid

November 2016


Data Cleansing Process1. Data auditing2. Workflow specification, i.e. choose appropriate methods to

automatically detect and remove them3. Workflow execution, apply the methods to the tuples in the data

collection4. Post-processing / Control

November 2016


Data Auditing• Data Profiling: Instance analysis• Data Mining: Whole data collection analysis• Examples

• Minimal, Maximal values• Value range• Variance• Uniquness• Null value occurences• Typical string patterns (through RegExps for example)

• Search for characteristics that could be used for the correction of anomalies

November 2016


Methods for Data Cleansing• Parsing (Syntax Errors)• Data Transformation (source to target format)• Integrity Constraint Enforcement (checking & maintenance)• Duplicate Elimination• Statistical Methods

• Outliers • Detection: mean, std, range, clustering, association rules• Remedy: set to average or other statistical value, censored, truncated (dropped)

• Missing• Detection: It's missing :)• Remedy: Filling-in (imputing) by a number of ways (mean, median, regression, propensity score, Markov-

Chain-Monte-Carlo method)

November 2016


Outlier (Error) Detection in Datasets• Statistical: mean (μ), std (σ), range (Chebyshev's theorem)• Accept (μ - ε * σ) < f < (μ + ε σ), where ε=5, else reject• Needs training/testing data for finding the best ε {3,4,5,6, etc.}

• Boxplots (univariate data)• Clustering, high-computational burden• Pattern-based, i.e. find a pattern where 90% of data exhibit the

same characteristics• Association rules: pattern = association rule with high confidence

and support

November 2016


Detecting anomalies• Visualization can help to detect anomalies• Automatic approach: apply committee of different learning schemes,

e.g.,• decision tree• nearest-neighbor learner• linear discriminant function

• Conservative consensus approach: delete instances incorrectly classified by all of them• Problem: might sacrifice instances of small classes

November 2016


One-Class Learning• Usually training data is available for all classes• Some problems exhibit only a single class at training time• Test instances may belong to this class or a new class not present at

training time• This the problem of one-class classification

• Predict either target or unknown

• Note that, in practice, some one-class problems can be re-formulated into two-class ones by collecting negative data

• Other applications truly do not have negative data, e.g., password hardening, nuclear plant operational status

November 2016


Outlier detection• One-class classification is often used for outlier/anomaly/novelty detection• First, a one-class models is built from the dataset• Then, outliers are defined as instances that are classified as unknown• Another method: identify outliers as instances that lie beyond distance d

from percentage p of training data• Density estimation is a very useful approach for one-class classification and

outlier detection• Estimate density of the target class and mark low probability test instances as

outliers• Threshold can be adjusted to calibrate sensitivity

November 2016


Transforming multiple classes to binary onesOutput processing

November 2016


Transforming multiple classes to binary ones• Some learning algorithms only work with two class problems

• e.g., standard support vector machines—only work with two-class problems.

• Sophisticated multi-class variants exist in many cases but can be very slow or difficult to implement

• A common alternative is to transform multi-class problems into multiple two-class ones

• Simple methods:• Discriminate each class against the union of the others – one-vs.-rest• Build a classifier for every pair of classes – pairwise classification

• We will discuss error-correcting output codes, which can often improve on these

November 2016


Error-correcting output codes• Multiclass problem multiple binary

problems• Simple one-vs.-rest scheme:

One-per-class coding• 1010??

• Idea: use error-correcting codes instead• base classifiers predict

1011111, true class = ??• Use bit vectors (codes) so that we

have large Hamming distancebetween any pair of bit vectors:• Can correct up to (d – 1)/2 single-bit errors

0001d0010c0100b1000aclass

vectorclass

0101010d0011001c0000111b1111111a

class vector

class

November 2016


The EndKyriakos C. Chatzidimitriouhttp://[email protected]

November 2016

http://kyrcha.info/

Ι/Ο Data Εngineering

Science

Transcript of Ι/Ο Data Εngineering