Practical_RL

by yandexdataschool

yandexdataschool / Practical_RL

A course in reinforcement learning in the wild

4.2K Stars 1.3K Forks Last release: 3 months ago (spring20) The Unlicense 1.3K Commits 2 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Practical_RL

An open course on reinforcement learning in the wild. Taught on-campus at HSE and YSDA and maintained to be friendly to online students (both english and russian).

Manifesto:

  • Optimize for the curious. For all the materials that aren’t covered in detail there are links to more information and related materials (D.Silver/Sutton/blogs/whatever). Assignments will have bonus sections if you want to dig deeper.
  • Practicality first. Everything essential to solving reinforcement learning problems is worth mentioning. We won't shun away from covering tricks and heuristics. For every major idea there should be a lab that makes you to “feel” it on a practical problem.
  • Git-course. Know a way to make the course better? Noticed a typo in a formula? Found a useful link? Made the code more readable? Made a version for alternative framework? You're awesome! Pull-request it!

Github contributors

Course info

Additional materials

Syllabus

The syllabus is approximate: the lectures may occur in a slightly different order and some topics may end up taking two weeks.

  • week01_intro Introduction

    • Lecture: RL problems around us. Decision processes. Stochastic optimization, Crossentropy method. Parameter space search vs action space search.
    • Seminar: Welcome into openai gym. Tabular CEM for Taxi-v0, deep CEM for box2d environments.
    • Homework description - see week1/README.md.
  • week02valuebased Value-based methods

    • Lecture: Discounted reward MDP. Value-based approach. Value iteration. Policy iteration. Discounted reward fails.
    • Seminar: Value iteration.
    • Homework description - see week2/README.md.
  • week03modelfree Model-free reinforcement learning

    • Lecture: Q-learning. SARSA. Off-policy Vs on-policy algorithms. N-step algorithms. TD(Lambda).
    • Seminar: Qlearning Vs SARSA Vs Expected Value SARSA
    • Homework description - see week3/README.md.
  • recapdeeplearning - deep learning recap

    • Lecture: Deep learning 101
    • Seminar: Intro to pytorch/tensorflow, simple image classification with convnets
  • week04approxrl Approximate (deep) RL

    • Lecture: Infinite/continuous state space. Value function approximation. Convergence conditions. Multiple agents trick; experience replay, target networks, double/dueling/bootstrap DQN, etc.
    • Seminar: Approximate Q-learning with experience replay. (CartPole, Atari)
  • week05_explore Exploration

    • Lecture: Contextual bandits. Thompson Sampling, UCB, bayesian UCB. Exploration in model-based RL, MCTS. "Deep" heuristics for exploration.
    • Seminar: bayesian exploration for contextual bandits. UCB for MCTS.
  • week06policybased Policy Gradient methods

    • Lecture: Motivation for policy-based, policy gradient, logderivative trick, REINFORCE/crossentropy method, variance reduction(baseline), advantage actor-critic (incl. GAE)
    • Seminar: REINFORCE, advantage actor-critic
  • week07_seq2seq Reinforcement Learning for Sequence Models

    • Lecture: Problems with sequential data. Recurrent neural netowks. Backprop through time. Vanishing & exploding gradients. LSTM, GRU. Gradient clipping
    • Seminar: character-level RNN language model
  • week08_pomdp Partially Observed MDP

    • Lecture: POMDP intro. POMDP learning (agents with memory). POMDP planning (POMCP, etc)
    • Seminar: Deep kung-fu & doom with recurrent A3C and DRQN
  • week09policyII Advanced policy-based methods

    • Lecture: Trust region policy optimization. NPO/PPO. Deterministic policy gradient. DDPG
    • Seminar: Approximate TRPO for simple robot control.
  • week10_planning Model-based RL & Co

    • Lecture: Model-Based RL, Planning in General, Imitation Learning and Inverse Reinforcement Learning
    • Seminar: MCTS for toy tasks
  • yetanotherweek Inverse RL and Imitation Learning

    • All that cool RL stuff that you won't learn from this course :)

Course staff

Course materials and teaching by: [unordered] - Pavel Shvechikov - lectures, seminars, hw checkups, reading group - Nikita Putintsev - seminars, hw checkups, organizing our hot mess - Alexander Fritsler - lectures, seminars, hw checkups - Oleg Vasilev - seminars, hw checkups, technical support - Dmitry Nikulin - tons of fixes, far and wide - Mikhail Konobeev - seminars, hw checkups - Ivan Kharitonov - seminars, hw checkups - Ravil Khisamov - seminars, hw checkups - Anna Klepova - hw checkups - Fedor Ratnikov - admin stuff

Contributions

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.