Course Overview
This graduate level course covers the bias-variance tradeoff, matrix spaces, matrix factorizations, eigenvalues and eigenvectors, Singular Value Decomposition (SVD), the Eckart-Young Theorem, vector and matrix norms, Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), the least squares method, regularization, logistc regression, exponential matrices, quadratic forms, derivatives of matrices, saddle points, minmax problems, function minimization, alternating least squares (ALS), Alternative Directions Method of Multipliers (ADMM), Non-negative Matrix Factorization (NMF), gradient descent algorithms, stochastic gradient descent (SGD), sparse representations, artificial neural networks (ANN), the back-propagation algorithm, partial derivatives, convolutional neural networks (CNN) and learning functions.
Syllabus & Lecture Topics
| Week | Topic & Detailed Content |
|---|---|
| Week 1 | Bias-Variance Tradeoff, Matrix Spaces, Matrix Factorizations, and Orthogonal Matrices |
| Week 2 | Eigenvalues, Eigenvectors, Positive/Positive Semi-Definite Matrices, and Singular Value Decomposition (SVD) |
| Week 3 | Vector/Matrix Norms, Principal Component Analysis, Linear Discriminant Analysis |
| Week 4 | The Least Squares Method, Ridge, LASSO and Elastic-Net |
| Week 5 | Logistic regression, cross-entropy loss, regularization |
| Week 6 | Quadratic forms, Jacobian and Hessian matrices |
| Week 7 | ALS, IRLS and ADMM techniques for optimization |
| Week 8 | Midterm exam |
| Week 9 | Non-negative matrix factorization |
| Week 10 | Function Minimization, Gradient Descent Algorithm, and Acceleration Methods |
| Week 11 | Stochastic Gradient Descent (SGD) Algorithm and Its Analysis |
| Week 12 | Sparse representations and dictionary learning |
| Week 13 | Artificial Neural Networks, Back-Propagation Algorithm, and Partial Derivatives |
| Week 14 | Convolutional Neural Networks (CNN) and Learning Functions |