CNTK Examples: Reinforcement Learning

Overview

Data:	Dynamic Atari Learning Environment (ALE)
Purpose	This folder contains examples that demonstrate how to use CNTK to define Deep Q Neural Network (Mnih & al, 2013), a value-based Deep Reinforcement Learning method inspired by Q-Learning method
Network	DeepQNeuralNetwork (DQN).
Training	Adam.
Comments	See below.

Running the example

python DeepQNeuralNetwork.py

Some options are available:

• -e : Number of epochs to run (one epoch is 250.000 actions taken)
• -p : Turn on tensorboard plotting, to visualize training
• Environment name, provided as trailing parameter to easily change the ALE environment

Example:

python DeepQNeuralNetwork.py -e 200 -p SpaceInvaders-v3

Details

This example uses OpenAI Atari Learning Environment to train an agent on various Atari games using reinforcement learning. As an Action-Value based network, a Deep Q Neural Network will try to estimate the expected reward for each action by looking at the N last states (s_t, s_t-1, ..., s_t-N).

This agent has an exploration process, that allows it to take random actions to have a better understanding of the game dynamics. The exploration process we use here is called 'Epsilon Greedy' where the 'best' action is taken with a probability of 1 - epsilon. Otherwise, a random action is taken. During the training, epsilon will slowly decay to a minimum value, commonly 0.1.

Notes

This example is only available on Linux as OpenAI ALE doesn't provide Windows interface.