Statistical Machine Learning

Statistical Machine Learning is a second graduate level course in advanced machine learning, assuming students have taken Machine Learning (10-715) and Intermediate Statistics (36-705). The term “statistical” in the title reflects the emphasis on statistical theory and methodology. The course combines methodology with theoretical foundations. Theorems are presented together with practical aspects of methodology and intuition to help students develop tools for selecting appropriate methods and approaches to problems in their own research. The course includes topics in statistical theory that are important for researchers in machine learning, including nonparametric theory, consistency, minimax estimation, and concentration of measure.

1. Review: probability, bias/variance, mle, regression, classification. 2. Theoretical Foundations (a) Function Spaces: Holder spaces, Sobolev spaces, reproducing kernel Hilbert spaces (RKHS) (b) Concentration of Measure (c) Minimax Theory 3. Supervised Learning (a) Linear Regression: low dimensional, ridge regression, lasso, greedy regression (b) Nonpar Regression: kernel regression, local polynomials, additive, RKHS regression (c) Linear Classification: linear, logistic, SVM, sparse logistic (d) Nonpar Classification: NN, naive Bayes, plug-in, kernelized SVM (e) Conformal Prediction (f) Cross Validation 4. Unsupervised Learning (a) Nonpar Density Estimation (b) Clustering: k-means, mixtures, single-linkage, density clustering, spectral clustering (c) Measures of Dependence (d) Graphical Models: correlation graphs, partial correlation graphs, cond. indep. graphs 5. Other Topics (a) Nonparametric Bayesian Inference (b) Bootstrap and subsampling (c) Interactive Data Analysis (d) Robustness (e) Active Learning (f) Differential Privacy (g) Deep Learning (h) Distributed Learning (i) Streaming