Some resources for ML research
Personal and biased selection of ML resources.
Disclaimer: I'm a noivce in ML research, and I read only a few of the list.
Table of Contents
Beginner's Guide
Must Read
 Machine Learning: A Probabilistic Perspective (Murphy)
 Deep Learning (Goodfellow et al.)
 Reinforcement Learning: An Introduction (Sutton & Barto)
Recommended
 Convex Optimization (Boyd & Vandenberghe)
 Graphical Models, Exponential Families, and Variational Inference (Wainwright & Jordan)
 Learning from Data (AbuMostafa) > for whom interested in learning theory
Recent Topics
 Read research blogs (e.g., OpenAI, BAIR, CMU)
 Read lectures from Berkeley, Stanford, CMU or UofT (e.g., unsupervised learning)
 There are lots of good sources, but I stop updating them uptodate
Machine Learning
There are many ML books, but most of them are encyclopedic.
I recommend to take a course using Murphy or Bishop book.
Textbook
 Machine Learning: A Probabilistic Perspective (Murphy) :sparkles:
 Pattern Recognition and Machine Learning (Bishop) :sparkles:
 The Elements of Statistical Learning (Hastie et al.)
 Pattern Classification (Duda et al.)
 Bayesian Reasoning and Machine Learning (Barber)
Lecture
Deep Learning
Goodfellow et al. is the new classic.
For vision and NLP, Stanford lectures would be helpful.
Textbook
 Deep Learning (Goodfellow et al.) :sparkles:
Lecture (Practice)
Lecture (Theory)
Generative Model
I seperated generative model as an independent topic,
since I think it is big and important area.
Lecture
Reinforcement Learning
For classic (nondeep) RL, Sutton & Barto is the classic.
For deep RL, lectures from Berkeley/CMU looks good.
Textbook
 Reinforcement Learning: An Introduction (Sutton & Barto) :sparkles:
Lecture
Graphical Model
Koller & Friedman is comprehensive, but too encyclopedic.
I recommend to take an introductory course using Koller & Friedman book.
Wainwright & Jordan only focuses on variational inference,
but it gives really good intuition for probabilistic models.
Textbook
 Probabilistic Graphical Models: Principles and Techniques (Koller & Friedman)
 Graphical Models, Exponential Families, and Variational Inference (Wainwright & Jordan) :sparkles:
Lecture
Optimization
Boyd & Vandenberghe is the classic, but I think it's too boring.
Reading chapter 25 would be enough.
Bertsekas more concentrates on convex analysis.
Nocedal & Wright more concentrates on optimization.
Textbook
 Convex Optimization (Boyd & Vandenberghe) :sparkles:
 Convex Optimization Theory (Bertsekas)
 Numerical Optimization (Nocedal & Wright)
Lecture
Learning Theory
In my understanding, there are two major topics in learning theory:

Learning Theory: VCdimension, PAClearning

Online Learning: regret bound, multiarmed bandit
For learning theory, Kearns & Vazirani is the classic; but it's too oldfashined.
AbuMostafa is a good introductory book, and I think it's enough for most people.
For online learning, CesaBianchi & Lugosi is the classic.
For multiarmed bandit, Bubeck & CesaBianchi provides a good survey.
Textbook (Learning Theory)
 Learning from Data (AbuMostafa) :sparkles:
 Foundations of Machine Learning (Mohri et al.)
 An Introduction to Computational Learning Theory (Kearns & Vazirani)
Textbook (Online Learning)
 Prediction, Learning, and Games (CesaBianchi & Lugosi)
 Regret Analysis of Stochastic and Nonstochastic Multiarmed Bandit Problems (Bubeck & CesaBianchi)
Lecture
Statistics
Statistics is a broad area; hence, I listed only a few of them.
For advanced topics, lectures from Berkeley/Stanford/CMU/MIT looks really cool.
Textbook (Statistical Inference)
 All of Statistics (Wasserman)
 Computer Age Statistical Inference (Efron & Hastie) :sparkles:
 Time Series Analysis and Its Applications: With R Examples (Shumway & Stoffer)
Textbook (Nonparametrics)
 All of Nonparametric Statistics (Wasserman)
 Introduction to Nonparametric Estimation (Tsybakov)
 Gaussian Process and Machine Learning (Rasmussen & Williams) :sparkles:
 Bayesian Nonparametrics (Ghosh & Ramamoorthi) :sparkles:
Textbook (Advanced Topics)
 HighDimensional Statistics: A NonAsymptotic Viewpoint (Wainwright) :sparkles:
 Statistics for HighDimensional Data (Bühlmann & van de Geer)
 Asymptotic Statistics (van der Vaart)
 Empirical Processes in MEstimation (van der Vaart)
Lecture
Topics in Machine Learning
Miscellaneous topics related to machine learning.
There are much more subfields, but I'll not list them all.
Information Theory
 Elements of Information Theory (Cover & Thomas)
 Information Theory, Inference, and Learning Algorithms (MacKay)
Network Science
 Networks, Crowds, and Markets (Easley & Kleinberg)
 Social and Economic Networks (Jackson)
Markov Chain
 Markov Chains (Norris)
 Markov Chains and Mixing Times (Levin et al.)
Game Theory
 Algorithmic Game Theory (Nisan et al.)
 Multiagent Systems (Shoham & LeytonBrown)
Combinatorics
 The Probabilistic Method (Alon & Spencer)
 A First Course in Combinatorial Optimization (Lee)
Algorithm
 Introduction to Algorithms (Cormen et al.)
 Randomized Algorithms (Motwani & Raghavan)
 Approximation Algorithms (Vazirani)
Geometric View
 Topological Data Analysis (Wasserman)
 Methods of Information Geometry (Amari & Nagaoka)
 Algebraic Geometry and Statistical Learning Theory (Watanabe)
Some Lectures
Math Backgrounds
I selected essential topics for machine learning.
Personally, I think more analysis / matrix / geometry never hurts.
Probability
 Probability: Theory and Examples (Durrett)
 Theoretical Statistics (Keener)
 Stochastic Processes (Bass)
 Probability and Statistics Cookbook (Vallentin)
Linear Algebra
 Linear Algebra (Hoffman & Kunze)
 Matrix Analysis (Horn & Johnson)
 Matrix Computations (Golub & Van Loan)
 The Matrix Cookbook (Petersen & Pedersen)
Large Deviations
 Concentration Inequalities and Martingale Inequalities (Chung & Lu)
 An Introduction to Matrix Concentration Inequalities (Tropp)
Blogs