Data Science and Machine Learning: Mathematical and Statistical Methods

Description

“This textbook is a well-rounded, rigorous, and informative work presenting the mathematics behind modern machine learning techniques. It hits all the right notes: the choice of topics is up-to-date and perfect for a course on data science for mathematics students at the advanced undergraduate or early graduate level. This book fills a sorely-needed gap in the existing literature by not sacrificing depth for breadth, presenting proofs of major theorems and subsequent derivations, as well as providing a copious amount of Python code. I only wish a book like this had been around when I first began my journey!” -Nicholas Hoell, University of Toronto

“This is a well-written book that provides a deeper dive into data-scientific methods than many introductory texts. The writing is clear, and the text logically builds up regularization, classification, and decision trees. Compared to its probable competitors, it carves out a unique niche. -Adam Loy, Carleton College

The purpose of Data Science and Machine Learning: Mathematical and Statistical Methods is to provide an accessible, yet comprehensive textbook intended for students interested in gaining a better understanding of the mathematics and statistics that underpin the rich variety of ideas and machine learning algorithms in data science.

Key Features

• Focuses on mathematical understanding.
• Presentation is self-contained, accessible, and comprehensive.
• Extensive list of exercises and worked-out examples.
• Many concrete algorithms with Python code.
• Full color throughout.

Table of Contents

1. Preface Notation
2. 1. Importing, Summarizing, and Visualizing Data
   1. Introduction
   2. Structuring Features According to Type
   3. Summary Tables
   4. Summary Statistics
   5. Visualizing Data
   6. Plotting Qualitative Variables
   7. Plotting Quantitative Variables
   8. Data Visualization in a Bivariate Setting Exercises
3. 2. Statistical Learning
   1. Introduction
   2. Supervised and Unsupervised Learning
   3. Training and Test Loss
   4. Tradeoffs in Statistical Learning
   5. Estimating Risk
   6. In-Sample Risk
   7. Cross-Validation
   8. Modeling Data
   9. Multivariate Normal Models
   10. Normal Linear Models
   11. Bayesian Learning Exercises
4. 3. Monte Carlo Methods
   1. Introduction
   2. Monte Carlo Sampling
   3. Generating Random Numbers
   4. Simulating Random Variables
   5. Simulating Random Vectors and Processes
   6. Resampling
   7. Markov Chain Monte Carlo
   8. Monte Carlo Estimation
   9. Crude Monte Carlo
   10. Bootstrap Method
   11. Variance Reduction
   12. Monte Carlo for Optimization
   13. Simulated Annealing
   14. Cross-Entropy Method
   15. Splitting for Optimization
   16. Noisy Optimization Exercises
5. 4. Unsupervised Learning
   1. Introduction
   2. Risk and Loss in Unsupervised Learning
   3. Expectation–Maximization (EM) Algorithm
   4. Empirical Distribution and Density Estimation
   5. Clustering via Mixture Models
   6. Mixture Models
   7. EM Algorithm for Mixture Models
   8. Clustering via Vector Quantization
   9. K-Means
   10. Clustering via Continuous Multiextremal Optimization
   11. Hierarchical Clustering
   12. Principal Component Analysis (PCA)
   13. Motivation: Principal Axes of an Ellipsoid
   14. PCA and Singular Value Decomposition (SVD) Exercises
6. 5. Regression
   1. Introduction
   2. Linear Regression
   3. Analysis via Linear Models
   4. Parameter Estimation
   5. Model Selection and Prediction
   6. Cross-Validation and Predictive Residual Sum of Squares
   7. In-Sample Risk and Akaike Information Criterion
   8. Categorical Features
   9. Nested Models
   10. Coefficient of Determination
   11. Inference for Normal Linear Models
   12. Comparing Two Normal Linear Models
   13. Confidence and Prediction Intervals
   14. Nonlinear Regression Models
   15. Linear Models in Python
   16. Modeling
   17. Analysis
   18. Analysis of Variance (ANOVA)
   19. Confidence and Prediction Intervals
   20. Model Validation
   21. Variable Selection
   22. Generalized Linear Models Exercises
7. 6. Regularization and Kernel Methods
   1. Introduction
   2. Regularization
   3. Reproducing Kernel Hilbert Spaces
   4. Construction of Reproducing Kernels
   5. Reproducing Kernels via Feature Mapping
   6. Kernels from Characteristic Functions
   7. Reproducing Kernels Using Orthonormal Features
   8. Kernels from Kernels 6.5 Representer Theorem
   9. Smoothing Cubic Splines
   10. Gaussian Process Regression
   11. Kernel PCA Exercises
8. 7. Classification
   1. Introduction
   2. Classification Metrics
   3. Classification via Bayes’ Rule
   4. Linear and Quadratic Discriminant Analysis
   5. Logistic Regression and Softmax Classification
   6. K-nearest Neighbors Classification
   7. Support Vector Machine
   8. Classification with Scikit-Learn Exercises
9. 8. Decision Trees and Ensemble Methods
   1. Introduction
   2. Top-Down Construction of Decision Trees
   3. Regional Prediction Functions
   4. Splitting Rules
   5. Termination Criterion
   6. Basic Implementation
   7. Additional Considerations
   8. Binary Versus Non-Binary Trees
   9. Data Preprocessing
   10. Alternative Splitting Rules
   11. Categorical Variables
   12. Missing Values
   13. Controlling the Tree Shape
   14. Cost-Complexity Pruning
   15. Advantages and Limitations of Decision Trees
   16. Bootstrap Aggregation
   17. Random Forests
   18. Boosting Exercises
10. Deep Learning
    1. Introduction
    2. Feed-Forward Neural Networks
    3. Back-Propagation
    4. Methods for Training
    5. Steepest Descent
    6. Levenberg–Marquardt Method
    7. Limited-Memory BFGS Method
    8. Adaptive Gradient Methods
    9. Examples in Python
    10. Simple Polynomial Regression
    11. Image Classification Exercises
11. A. Linear Algebra and Functional Analysis
    1. Vector Spaces, Bases, and Matrices
    2. Inner Product
    3. Complex Vectors and Matrices
    4. Orthogonal Projections
    5. Eigenvalues and Eigenvectors
    6. Left- and Right-Eigenvectors
    7. Matrix Decompositions
    8. (P)LU Decomposition
    9. Woodbury Identity
    10. Cholesky Decomposition
    11. QR Decomposition and the Gram–Schmidt Procedure
    12. Singular Value Decomposition
    13. Solving Structured Matrix Equations
    14. Functional Analysis
    15. Fourier Transforms
    16. Discrete Fourier Transform
    17. Fast Fourier Transform
12. B. Multivariate Differentiation and Optimization
    1. Multivariate Differentiation
    2. Taylor Expansion
    3. Chain Rule
    4. Optimization Theory
    5. Convexity and Optimization
    6. Lagrangian Method
    7. Duality
    8. Numerical Root-Finding and Minimization
    9. Newton-Like Methods
    10. Quasi-Newton Methods
    11. Normal Approximation Method
    12. Nonlinear Least Squares
    13. Constrained Minimization via Penalty Functions
13. C. Probability and Statistics
    1. Random Experiments and Probability Spaces
    2. Random Variables and Probability Distributions
    3. Expectation
    4. Joint Distributions
    5. Conditioning and Independence
    6. Conditional Probability
    7. Independence
    8. Expectation and Covariance
    9. Conditional Density and Conditional Expectation
    10. Functions of Random Variables
    11. Multivariate Normal Distribution
    12. Convergence of Random Variables
    13. Law of Large Numbers and Central Limit Theorem
    14. Markov Chains
    15. Statistics
    16. Estimation
    17. Method of Moments
    18. Maximum Likelihood Method
    19. Confidence Intervals
    20. Hypothesis Testing
14. D. Python Primer
    1. Getting Started
    2. Python Objects
    3. Types and Operators
    4. Functions and Methods
    5. Modules
    6. Flow Control
    7. Iteration
    8. Classes
    9. Files
    10. NumPy
    11. Creating and Shaping Arrays
    12. Slicing
    13. Array Operations
    14. Random Numbers
    15. Matplotlib
    16. Creating a Basic Plot
    17. Pandas
    18. Series and DataFrame
    19. Manipulating Data Frames
    20. Extracting Information
    21. Plotting
    22. Scikit-learn
    23. Partitioning the Data
    24. Standardization
    25. Fitting and Prediction
    26. Testing the Model
    27. System Calls, URL Access, and Speed-Up
15. Bibliography
16. Index

Reviews

“The first impression when handling and opening this book at a random page is superb. A big format (A4) and heavy weight, because the paper quality is high, along with a spectacular style and large font, much colour and many plots, and blocks of python code enhanced in colour boxes. This makes the book attractive and easy to study…The book is a very well-designed data science course, with mathematical rigor in mind. Key concepts are highlighted in red in the margins, often with links to other parts of the book…This book will be excellent for those that want to build a strong mathematical foundation for their knowledge on the main machine learning techniques, and at the same time get python recipes on how to perform the analyses for worked examples.“
– Victor Moreno, ISCB News, December 2020

Authors Biography

Dirk P. Kroese, PhD, is a Professor of Mathematics and Statistics at The University of Queensland. He has published over 120 articles and five books in a wide range of areas in mathematics, statistics, data science, machine learning, and Monte Carlo methods. He is a pioneer of the well-known Cross-Entropy method—an adaptive Monte Carlo technique, which is being used around the world to help solve difficult estimation and optimization problems in science, engineering, and finance.

Zdravko Botev, PhD, is an Australian Mathematical Science Institute Lecturer in Data Science and Machine Learning with an appointment at the University of New South Wales in Sydney, Australia. He is the recipient of the 2018 Christopher Heyde Medal of the Australian Academy of Science for distinguished research in the Mathematical Sciences.

Thomas Taimre, PhD, is a Senior Lecturer of Mathematics and Statistics at The University of Queensland.
His research interests range from applied probability and Monte Carlo methods to applied physics and the remarkably universal self-mixing effect in lasers. He has published over 100 articles, holds a patent, and is the coauthor of Handbook of Monte Carlo Methods (Wiley).

Radislav Vaisman, PhD, is a Lecturer of Mathematics and Statistics at The University of Queensland. His research interests lie at the intersection of applied probability, machine learning, and computer science. He has published over 20 articles and two books.