Some resources for ML research
Personal and biased selection of ML resources.
Disclaimer: I'm a noivce in ML research, and I read only a few of the list.
Table of Contents
- Machine Learning: A Probabilistic Perspective (Murphy)
- Deep Learning (Goodfellow et al.)
- Reinforcement Learning: An Introduction (Sutton & Barto)
- Convex Optimization (Boyd & Vandenberghe)
- Graphical Models, Exponential Families, and Variational Inference (Wainwright & Jordan)
- Learning from Data (Abu-Mostafa) -> for whom interested in learning theory
- Read research blogs (e.g., OpenAI, BAIR, CMU)
- Read lectures from Berkeley, Stanford, CMU or UofT (e.g., unsupervised learning)
- There are lots of good sources, but I stop updating them up-to-date
There are many ML books, but most of them are encyclopedic.
I recommend to take a course using Murphy or Bishop book.
- Machine Learning: A Probabilistic Perspective (Murphy) :sparkles:
- Pattern Recognition and Machine Learning (Bishop) :sparkles:
- The Elements of Statistical Learning (Hastie et al.)
- Pattern Classification (Duda et al.)
- Bayesian Reasoning and Machine Learning (Barber)
Goodfellow et al. is the new classic.
For vision and NLP, Stanford lectures would be helpful.
- Deep Learning (Goodfellow et al.) :sparkles:
I seperated generative model as an independent topic,
since I think it is big and important area.
For classic (non-deep) RL, Sutton & Barto is the classic.
For deep RL, lectures from Berkeley/CMU looks good.
- Reinforcement Learning: An Introduction (Sutton & Barto) :sparkles:
Koller & Friedman is comprehensive, but too encyclopedic.
I recommend to take an introductory course using Koller & Friedman book.
Wainwright & Jordan only focuses on variational inference,
but it gives really good intuition for probabilistic models.
- Probabilistic Graphical Models: Principles and Techniques (Koller & Friedman)
- Graphical Models, Exponential Families, and Variational Inference (Wainwright & Jordan) :sparkles:
Boyd & Vandenberghe is the classic, but I think it's too boring.
Reading chapter 2-5 would be enough.
Bertsekas more concentrates on convex analysis.
Nocedal & Wright more concentrates on optimization.
- Convex Optimization (Boyd & Vandenberghe) :sparkles:
- Convex Optimization Theory (Bertsekas)
- Numerical Optimization (Nocedal & Wright)
In my understanding, there are two major topics in learning theory:
Learning Theory: VC-dimension, PAC-learning
Online Learning: regret bound, multi-armed bandit
For learning theory, Kearns & Vazirani is the classic; but it's too old-fashined.
Abu-Mostafa is a good introductory book, and I think it's enough for most people.
For online learning, Cesa-Bianchi & Lugosi is the classic.
For multi-armed bandit, Bubeck & Cesa-Bianchi provides a good survey.
Textbook (Learning Theory)
- Learning from Data (Abu-Mostafa) :sparkles:
- Foundations of Machine Learning (Mohri et al.)
- An Introduction to Computational Learning Theory (Kearns & Vazirani)
Textbook (Online Learning)
- Prediction, Learning, and Games (Cesa-Bianchi & Lugosi)
- Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems (Bubeck & Cesa-Bianchi)
Statistics is a broad area; hence, I listed only a few of them.
For advanced topics, lectures from Berkeley/Stanford/CMU/MIT looks really cool.
Textbook (Statistical Inference)
- All of Statistics (Wasserman)
- Computer Age Statistical Inference (Efron & Hastie) :sparkles:
- Time Series Analysis and Its Applications: With R Examples (Shumway & Stoffer)
- All of Nonparametric Statistics (Wasserman)
- Introduction to Nonparametric Estimation (Tsybakov)
- Gaussian Process and Machine Learning (Rasmussen & Williams) :sparkles:
- Bayesian Nonparametrics (Ghosh & Ramamoorthi) :sparkles:
Textbook (Advanced Topics)
- High-Dimensional Statistics: A Non-Asymptotic Viewpoint (Wainwright) :sparkles:
- Statistics for High-Dimensional Data (Bühlmann & van de Geer)
- Asymptotic Statistics (van der Vaart)
- Empirical Processes in M-Estimation (van der Vaart)
Topics in Machine Learning
Miscellaneous topics related to machine learning.
There are much more subfields, but I'll not list them all.
- Elements of Information Theory (Cover & Thomas)
- Information Theory, Inference, and Learning Algorithms (MacKay)
- Networks, Crowds, and Markets (Easley & Kleinberg)
- Social and Economic Networks (Jackson)
- Markov Chains (Norris)
- Markov Chains and Mixing Times (Levin et al.)
- Algorithmic Game Theory (Nisan et al.)
- Multiagent Systems (Shoham & Leyton-Brown)
- The Probabilistic Method (Alon & Spencer)
- A First Course in Combinatorial Optimization (Lee)
- Introduction to Algorithms (Cormen et al.)
- Randomized Algorithms (Motwani & Raghavan)
- Approximation Algorithms (Vazirani)
- Topological Data Analysis (Wasserman)
- Methods of Information Geometry (Amari & Nagaoka)
- Algebraic Geometry and Statistical Learning Theory (Watanabe)
I selected essential topics for machine learning.
Personally, I think more analysis / matrix / geometry never hurts.
- Probability: Theory and Examples (Durrett)
- Theoretical Statistics (Keener)
- Stochastic Processes (Bass)
- Probability and Statistics Cookbook (Vallentin)
- Linear Algebra (Hoffman & Kunze)
- Matrix Analysis (Horn & Johnson)
- Matrix Computations (Golub & Van Loan)
- The Matrix Cookbook (Petersen & Pedersen)
- Concentration Inequalities and Martingale Inequalities (Chung & Lu)
- An Introduction to Matrix Concentration Inequalities (Tropp)