Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. ·...

39
Boosting, Bagging, etc. Based mostly on notes by Greg Ridgeway David Madigan

Transcript of Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. ·...

Page 1: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

Boosting, Bagging, etc.

Based mostly on notes by Greg Ridgeway

David Madigan

Page 2: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

ε ε

Page 3: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

ε

Page 4: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

ε

Page 5: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 6: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 7: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 8: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 9: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 10: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 11: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 12: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

Random Forests“

”

Page 13: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 14: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 15: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

> spam7.rf <- randomForest(yesno ~ ., data=spam7, importance=TRUE)> print(spam7.rf)

Call: randomForest(formula = yesno ~ ., data = spam7, importance = TRUE) Type of random forest: classification Number of trees: 500No. of variables tried at each split: 2

OOB estimate of error rate: 11.89%Confusion matrix: n y class.errorn 2642 146 0.05236729y 401 1412 0.22118036>

in R: library(randomForest)

Page 16: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

> tuneRF(x=spam7[,-7], y=spam7$yesno, trace=FALSE)-0.0582878 0.05 -0.01821494 0.05 mtry OOBError1 1 0.12627692 2 0.11932194 4 0.1214953>

> importance(spam7.rf) n y MeanDecreaseAccuracy MeanDecreaseGinicrl.tot 0.6355890 0.8020131 0.5177274 241.5807dollar 0.6494899 0.7958824 0.5210265 437.7034bang 0.6810718 0.8493533 0.5349533 575.5270money 0.5864884 0.7986897 0.5048771 211.2677n000 0.6500864 0.4718939 0.5107114 123.9895make 0.3840864 0.5809521 0.4117850 41.0645

Page 17: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 18: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 19: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 20: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 21: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 22: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

Andreas Buja’s home page:

Page 23: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 24: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 25: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 26: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 27: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 28: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 29: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 30: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 31: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 32: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 33: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 34: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 35: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 36: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 37: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 38: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning
Page 39: Boosting, Bagging, etc. - Columbia Universitymadigan/G6101/notes/boostbag.pdf · 2007. 12. 9. · Bootstrap aggregation ("bagging") is a device for reducing the variance of learning

http://www.cs.princeton.edu/~schapire/boost.html


Decision Tree and Boosting - CILVR at NYU · Decision Tree and Boosting Tong Zhang Rutgers University T. Zhang (Rutgers) Boosting 1 / 29

Decision Tree and Boosting - CILVR at NYU · Decision Tree and Boosting Tong Zhang Rutgers University T. Zhang (Rutgers) Boosting 1 / 29

Boosting global citizenship education using digital ... · Η παρούσα εργασία είναι σύμφωνα με τα δικαιώματα Creative Commons Attribution-NonCommercial-ShareAlike

Boosting global citizenship education using digital ... · Η παρούσα εργασία είναι σύμφωνα με τα δικαιώματα Creative Commons Attribution-NonCommercial-ShareAlike

-Catenin Signaling Pathway β Wnt/ Reducing E-Cadherin and ...

-Catenin Signaling Pathway β Wnt/ Reducing E-Cadherin and ...

A novel lineage of sulfate-reducing microorganisms ... · PDF fileA novel lineage of sulfate-reducing microorganisms: Thermodesulfobiaceae fam. nov., Thermodesulfobium narugense, gen.

A novel lineage of sulfate-reducing microorganisms ... · PDF fileA novel lineage of sulfate-reducing microorganisms: Thermodesulfobiaceae fam. nov., Thermodesulfobium narugense, gen.

Joe Felsenstein Department of Genome Sciences and Department …evolution.gs.washington.edu/gs541/2004/lecture27.pdf · 2007. 2. 26. · Bootstrap sample #1 Bootstrap sample #2 Estimate

Joe Felsenstein Department of Genome Sciences and Department …evolution.gs.washington.edu/gs541/2004/lecture27.pdf · 2007. 2. 26. · Bootstrap sample #1 Bootstrap sample #2 Estimate

Bootstrap and Resampling Methodsluke/classes/STAT7400/...Bootstrap and Resampling Methods Often we have an estimator T of a parameter q and want to know its sampling properties –

Bootstrap and Resampling Methodsluke/classes/STAT7400/...Bootstrap and Resampling Methods Often we have an estimator T of a parameter q and want to know its sampling properties –

Bootstrap Methods and the Accuracy of Large-Scale Estimatorsstatistics.rutgers.edu/joomlatools-files/docman-files/Efron.pdfLeukemia Microarray Study (Golub et al., 1999) 72 leukemia

Bootstrap Methods and the Accuracy of Large-Scale Estimatorsstatistics.rutgers.edu/joomlatools-files/docman-files/Efron.pdfLeukemia Microarray Study (Golub et al., 1999) 72 leukemia

Phylogenetic Analysis · 2013-10-16 · Phylogenetic Analysis Han Liang, Ph.D. ... • Bootstrap Analysis . What is Phylogenetics • The term phylogenetics derives from Greek ...

Phylogenetic Analysis · 2013-10-16 · Phylogenetic Analysis Han Liang, Ph.D. ... • Bootstrap Analysis . What is Phylogenetics • The term phylogenetics derives from Greek ...

Bootstrap Methods for Time Series: A Selective Overview ...politis/TALKS/slidesCyprusBOOT2004.pdf · Bootstrap distribution is automatically centered correctly. • Circular Block

Bootstrap Methods for Time Series: A Selective Overview ...politis/TALKS/slidesCyprusBOOT2004.pdf · Bootstrap distribution is automatically centered correctly. • Circular Block

Inference Based on the Wild Bootstrap - Carleton University€¦ · Bootstrap methods are sometimes used to estimate standard errors and covariance matrices. ... Might be useful for

Inference Based on the Wild Bootstrap - Carleton University€¦ · Bootstrap methods are sometimes used to estimate standard errors and covariance matrices. ... Might be useful for

Bagging and Boosting - Computer Science and Boosting Amit Srinet Dave Snyder Outline Bagging Definition Variants Examples Boosting Definition Hedge(β) AdaBoost Examples Comparison

Bagging and Boosting - Computer Science and Boosting Amit Srinet Dave Snyder Outline Bagging Definition Variants Examples Boosting Definition Hedge(β) AdaBoost Examples Comparison

07 - Bootstrap and Splines › SYS6018 › lectures › 07-bootstrap.pdf · 07 - Bootstrap and Splines SYS 6018 | Fall 2019 6/17 1 #-- Bootstrap Distribution 2 M =2000 # number of

07 - Bootstrap and Splines › SYS6018 › lectures › 07-bootstrap.pdf · 07 - Bootstrap and Splines SYS 6018 | Fall 2019 6/17 1 #-- Bootstrap Distribution 2 M =2000 # number of

Perioperative Myocardial Infarction Myocardial... · at reducing perioperative acute myocardial infarction. ... βBlockers Increase Perioperative Stroke? Bangalore Lancet 2008 β-Blockers

Perioperative Myocardial Infarction Myocardial... · at reducing perioperative acute myocardial infarction. ... βBlockers Increase Perioperative Stroke? Bangalore Lancet 2008 β-Blockers

ϑ Proportional pressure reducing 1/16 valve, pilot operated · Valves of type DRE(M) are pilot controlled pressure reducing valves. They are used for reducing an operating pressure.

ϑ Proportional pressure reducing 1/16 valve, pilot operated · Valves of type DRE(M) are pilot controlled pressure reducing valves. They are used for reducing an operating pressure.

Bagging, Random Forest und Boosting - mi.uni-koeln.de · Die Random Forest Methode o ist eine Modifikation der Bagging-Methode o Trainingsdatensatz wird ebenfalls in sogenannte Bootstrap

Bagging, Random Forest und Boosting - mi.uni-koeln.de · Die Random Forest Methode o ist eine Modifikation der Bagging-Methode o Trainingsdatensatz wird ebenfalls in sogenannte Bootstrap

Inference in first- order logic. Outline Reducing first-order inference to propositional inference Unification Generalized Modus Ponens Forward chaining.

Inference in first- order logic. Outline Reducing first-order inference to propositional inference Unification Generalized Modus Ponens Forward chaining.

Lecture 27+. The Bootstrap Method: Simulation Resultszuev/teaching/2013Spring... · Math 408 - Mathematical Statistics Lecture 27+. The Bootstrap Method: Simulation Results April

Lecture 27+. The Bootstrap Method: Simulation Resultszuev/teaching/2013Spring... · Math 408 - Mathematical Statistics Lecture 27+. The Bootstrap Method: Simulation Results April

 · Web viewσυμμετέχει στο εταιρικό σχήμα που υλοποιεί το έργο “BOOSTing the innovation potential of the triple helix of Adriatic-Ionian

 · Web viewσυμμετέχει στο εταιρικό σχήμα που υλοποιεί το έργο “BOOSTing the innovation potential of the triple helix of Adriatic-Ionian

Boosting 2: Classification - GitHub Pages

Boosting 2: Classification - GitHub Pages

Inference Based on the Wild Bootstrap · Bootstrap Testing Consider the heteroskedasticity-robust t statistic l 0 l) = l 0 q l (X>X) 1X>X^ (X>X) 1 ll: (17) To calculate wild bootstrap

Inference Based on the Wild Bootstrap · Bootstrap Testing Consider the heteroskedasticity-robust t statistic l 0 l) = l 0 q l (X>X) 1X>X^ (X>X) 1 ll: (17) To calculate wild bootstrap