An open course on reinforcement learning in the wild.
Taught on-campus at HSE and YSDA and maintained to be friendly to online students (both english and russian).
Optimize for the curious. For all the materials that aren’t covered in detail there are links to more information and related materials (D.Silver/Sutton/blogs/whatever). Assignments will have bonus sections if you want to dig deeper.
Practicality first. Everything essential to solving reinforcement learning problems is worth mentioning. We won't shun away from covering tricks and heuristics. For every major idea there should be a lab that makes you to “feel” it on a practical problem.
Git-course. Know a way to make the course better? Noticed a typo in a formula? Found a useful link? Made the code more readable? Made a version for alternative framework? You're awesome! Pull-request it!
The syllabus is approximate: the lectures may occur in a slightly different order and some topics may end up taking two weeks.
- Lecture: RL problems around us. Decision processes. Stochastic optimization, Crossentropy method. Parameter space search vs action space search.
- Seminar: Welcome into openai gym. Tabular CEM for Taxi-v0, deep CEM for box2d environments.
- Homework description - see week1/README.md.
week02valuebased Value-based methods
- Lecture: Discounted reward MDP. Value-based approach. Value iteration. Policy iteration. Discounted reward fails.
- Seminar: Value iteration.
- Homework description - see week2/README.md.
week03modelfree Model-free reinforcement learning
- Lecture: Q-learning. SARSA. Off-policy Vs on-policy algorithms. N-step algorithms. TD(Lambda).
- Seminar: Qlearning Vs SARSA Vs Expected Value SARSA
- Homework description - see week3/README.md.
recapdeeplearning - deep learning recap
- Lecture: Deep learning 101
- Seminar: Intro to pytorch/tensorflow, simple image classification with convnets
week04approxrl Approximate (deep) RL
- Lecture: Infinite/continuous state space. Value function approximation. Convergence conditions. Multiple agents trick; experience replay, target networks, double/dueling/bootstrap DQN, etc.
- Seminar: Approximate Q-learning with experience replay. (CartPole, Atari)
- Lecture: Contextual bandits. Thompson Sampling, UCB, bayesian UCB. Exploration in model-based RL, MCTS. "Deep" heuristics for exploration.
- Seminar: bayesian exploration for contextual bandits. UCB for MCTS.
week06policybased Policy Gradient methods
- Lecture: Motivation for policy-based, policy gradient, logderivative trick, REINFORCE/crossentropy method, variance reduction(baseline), advantage actor-critic (incl. GAE)
- Seminar: REINFORCE, advantage actor-critic
week07_seq2seq Reinforcement Learning for Sequence Models
- Lecture: Problems with sequential data. Recurrent neural netowks. Backprop through time. Vanishing & exploding gradients. LSTM, GRU. Gradient clipping
- Seminar: character-level RNN language model
week08_pomdp Partially Observed MDP
- Lecture: POMDP intro. POMDP learning (agents with memory). POMDP planning (POMCP, etc)
- Seminar: Deep kung-fu & doom with recurrent A3C and DRQN
week09policyII Advanced policy-based methods
- Lecture: Trust region policy optimization. NPO/PPO. Deterministic policy gradient. DDPG
- Seminar: Approximate TRPO for simple robot control.
week10_planning Model-based RL & Co
- Lecture: Model-Based RL, Planning in General, Imitation Learning and Inverse Reinforcement Learning
- Seminar: MCTS for toy tasks
yetanotherweek Inverse RL and Imitation Learning
- All that cool RL stuff that you won't learn from this course :)
Course materials and teaching by: [unordered]
- Pavel Shvechikov - lectures, seminars, hw checkups, reading group
- Nikita Putintsev - seminars, hw checkups, organizing our hot mess
- Alexander Fritsler - lectures, seminars, hw checkups
- Oleg Vasilev - seminars, hw checkups, technical support
- Dmitry Nikulin - tons of fixes, far and wide
- Mikhail Konobeev - seminars, hw checkups
- Ivan Kharitonov - seminars, hw checkups
- Ravil Khisamov - seminars, hw checkups
- Anna Klepova - hw checkups
- Fedor Ratnikov - admin stuff