Reinforcement Learning and Artificial Intelligence (RLAI)
	Reinforcement Learning Toolkit

Toolkit Contents:

• The Reinforcement Learning Interface (RLI) (module RLinterface) - a standard for interconnecting RL agents and environments. This interface allows you to simply specify an  agent function and an environment function and then easily simulate steps or episodes of them interacting.
• Tile Coding (tiles) - tile coding is a very useful way to handle large or continuous state spaces. The software here includes collision handling, as well as wrap versions.
  
• Eligibility Traces (module traces) - ...
  
• Utilities (module utilities) - (probability, statistics, etc.)
• Graphical Utilities (quickgraph)
  • g - a low level graphics package using Python and Tk. This is a very easy to use graphics package which will enable you to quickly make graphical displays of your application. We've used it to make some fine demos.
    
  • graph - a quick graphing tool based on g
  • graph3d - a 3d drawing package based on g 
  Note: g is based on tk. You must have tk installed on your machine, and the Tkinter package for python. You must also be running in a windowing environment to be able to run these and create new windows.
• A variety of agents and environments ...
  
• Demos:
  • Mountain Car (both graphical and non graphical versions)
  • Maintenance Example - a simple mdp where the agent tries to maximize a machine running time.
    
  • Gridworld (both graphical and non graphical versions) - the agent must make its way through a gridworld with barriers and walls to the goal. In the GUI version, the squares change color to show learning. The agent may use one of 4 learning methods: one step Q, Q lambda with traces, sarsa lambda, and dyna.
    
  • Function Approximation - this GUI demo visually shows how function approximation works.
  • Tiles - this GUI demo shows for one particular x,y point the various tiles of different shapes that the point fits in. Use the space bar and arrow keys to highlight the various tiles. Clicking in the graph will show the tiles for the new point you click at.
    

RLToolkit downloads - Version 1.0:  

• Temporarily disabled
  

Requirements:

• Python (version 2.3)
• C++ (optional; needed only for the faster version of tiles)
  
• For the GUI version you must also have the following installed on your machine:
  
  • Tk
  • Tkinter (the Python interface for Tk)
  • In addition, you must be running in a windowing environment.
  • Questions? More information is available here.
• For help in using Python, and especially with Tkinter, see here.
  

Installation:

• Install the toolkit as a standard Python module:
  • Download the entire module. It unzips/tars into a folder called RLtoolkit.
  • Place the RLtoolkit folder in your Python site-packages folder. For example:
    • Linux Redhat 9: /usr/lib/python2.3/site-packages
    • Mac OS X: /Library/Python/2.3 or
      /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages
    • Windows XP: c:\Python23\Lib\site-packages
• Install the toolkit as a user level module.
  • Download the entire module. It unzips into a folder called RLtoolkit. Place the RLtoolkit folder wherever you like.
  • Set up your own site packages list, using the folder that you put the unzipped RLtoolkit folder into. Add your directory to the python path by either
    • setting the environment variable PYTHONPATH   or
      
    • executing the following python commands from within Python:

      import sys

      sys.path.append("dir")

    • Alternatively, just move to the folder which contains the new RLtoolkit folder and start python.
  • Note: if you have problems with python complaining that it can't find the toolkit when you know that you have the directory right, add an empty __init__.py file (two underscores, init, two more underscores, and .py) to the directory at the same level as RLtoolkit.
    
• Download the toolkit and use directly.
  
  • Download the entire module. It unzips into a folder called RLtoolkit. Place the RLtoolkit folder wherever you like.
    
  • You will be able to use some of the tools, but there may be problems with others, and with the demos if you do not set it up as a python package.
• If you wish to use the c version of tiles, you must also compile the c code located in RLtoolkit.CTiles. There is a Macintosh Makefile available there to help you do this.
  
• If you have installed the toolkit and it tells you that one of the packages or modules does not exist, check the __init__.py files for the package or subpackage involved. If the file is missing, you can add an empty one by that name, but then imports with "*" may not work as desired.
  

Usage:

• If you have not installed the toolkit as a package, you must move to the directory where you put the RLtoolkit folder before starting python. You will only be able to load specific components. Since most of the demos refer to RLtoolkit as a package, you may not be able to run them.
• You can also move one level deeper into the directory (inside RLtoolkit), and start python from there. Then you  can load things with just their name, without having to prefix RLtoolkit. (This is not recommended for the demos) For example:
  
• from RLinterface import *

Importing the Whole Toolkit at Once

• To have all the major tools available for use, with module names:
• from RLtoolkit import *
  makes the major tools (tiles, traces, g, graph, RLinterface, utilities) as well as demo (which helps run the demos) available. You must put a module name in front to use them. For the tools which have their own folders (and thus are not simple modules), some have a shortcut set up so that you can use a module name which is the lower case version of the tool. Then you can do tiles.loadtiles(...) or g.gDrawCircle(...) as in previous versions. The graph and graph3d routines should be called with graph. and graph3d. (instead of quickgraph.) Note that the "extra" modules in each tool package (e.g. the tiles demo, the g tests) are NOT available with this import.
• To have all the major tools available for use, without module names:
• from RLtoolkit.guiuser import *
  then you can just call the tiles routines, g routines, etc by their names. This loads tiles, the tiles demo, traces, g, graph, graph3d, RLinterface, the demo function and utilities.
  
• from RLtoolkit.nonguiuser import *
  then you can just call the tiles routines etc by their names. This loads tiles, traces, RLinterface and utilities.

Importing Specific Tools

• To load the tiles package, do ONE of the following:
• from RLtoolkit.Tiles import *
  then you can call the various tile routines by prefixing the modules names in front. For example, tiles.loadtiles(...), tilesdemo.showtiles(...), fancytiles.diamondtiles(...)
  
• import RLtoolkit.Tiles.tiles as tiles
  then you can use the basic tile routines by putting tiles. in front (e.g. tiles.tiles, tiles.loadtiles, tiles.tileswrap, tiles.loadtileswrap, tiles.CollisionTable). The tiles demo and "fancy" tiles functions are NOT imported or available with this import.
  
• from RLtoolkit.Tiles.tiles import *
  then you can use the basic tile routines without having to prefix a module name. The demo and "fancy" tiles functions are NOT imported or available with this import.
  
• from RLtoolkit.tiles import *
  does exactly the same as the previous import. It is just a shortcut.
• To load the g package, do ONE of the following:
• from RLtoolkit.G import *
  then you can call the various g routines by prefixing the module name in front. For example, tiles.g(...)
  
• import RLtoolkit.G.g as g
  then you can use the basic tile routines by putting g. in front (e.g. g.Gwindow, g.gDrawCircle(...), g.gStartEventLoop). The g example and test functions are NOT imported or available with this import.
  
• from RLtoolkit.G.g import *
  then you can use the g functions without having to prefix a module name. The examples and test  functions are NOT imported or available with this import.
  
• from RLtoolkit.g import *
  does exactly the same as the previous import. It is just a shortcut.
• To load the quickgraph package, do ONE of the following:
• from RLtoolkit.Quickgraph import *
  then you can call the various graphing routines by prefixing the module name in front. For example, graph.graph(...), graph.xTickmarks(...), graph3d.graphSurface(...)
  
• import RLtoolkit.Quickgraph.graph as graph
  then you can use the 2d graphing routines by putting graph. in front (e.g. graph.graph(...), graph.xGraphLimits(...)). The graph3d functions are NOT imported or available with this import.
  
• from RLtoolkit.Quickgraph.graph import *
  then you can use the graph functions without having to prefix a module name. The graph3d  functions are NOT imported or available with this import.
  
• from RLtoolkit.graph import *
  does exactly the same as the previous import. It is just a shortcut.
• import RLtoolkit.Quickgraph.graph3d as graph3d
  then you can use the 3d graphing routines by putting graph3d. in front (e.g. graph3d.graphSurface(...)). The graph2d functions are NOT imported or available with this import.
  
• from RLtoolkit.Quickgraph.graph3d import *
  then you can use the graph3d functions without having to prefix a module name. The graph2d  functions are NOT imported or available with this import.
  
• from RLtoolkit.graph3d import *
  does exactly the same as the previous import. It is just a shortcut.
• To load any single module tools (RLinterface, traces, utilities) do ONE of the following:
• import RLtoolkit.toolname as toolname
  then you can use the tool's routines by putting toolname. in front. For example:
  
• import RLtoolkit.RLinterface as rli
• rl = rli.RLinterface(...)
• from RLtoolkit.toolname import *
  then you can call the tool's routines without prefixing the module name in front. For example:
• from RLtoolkit,traces import *
• th = TraceHolder(...)
  

Running the Demos

• The following demos are available:
• Mountain Car
• GUI version - in RLtoolkit.examples.mountainDemoG
• Nongui version - in RLtoolkit.examples.mountainDemoN
• Maintenance Example (running/maintaining machine to maximize reward)
• Nongui version only - in RLtoolkit.examples.maintenanceDemoN
• Gridworld
• GUI version - in RLtoolkit.gridworld.gwDemoG
• Nongui version - in RLtoolkit.gridworld.gwDemoN
• Function Approximation
• GUI version only - in RLtoolkit.fa.demo
• Tiles demo
• GUI version only - in RLtoolkit.Tiles.tilesdemo
• To run a specific demo:
• Using IDLE, open the specific demo file itself (in folder examples, fa, or gridworld within RLtoolkit) and run it (F5). The demos (usually) end in DemoN (for non graphical) or DemoG (for GUI). They will automatically load whatever tools and other files they need.
  
• You can import a demo. GUI demos may start automatically, but if they don't, use the following:
• from RLtoolkit.examples.mountainDemoG import runDemo
  then start the demo with runDemo()

• from RLtoolkit.fa.demo import faDemo
  then start the demo with faDemo()
• from RLtoolkit.gridworld.gwDemoG import *
  then run with runDemo() or runObjDemo()
• from RLtoolkit.Tiles.tilesdemo import showtiles
  then run with showtiles(numtilings, memct, floats, title, start, end, intervals)
  
• The nongui demos will print help information on how to run them when they are imported. Note that  the functions described by this help information require that you prefix them with a module name. Example imports:
• import RLtoolkit.examples.mountainDemoN as mcdemo
  then call the functions shown with mcdemo. in front (e.g. mcdemo.mcInit()).
• import RLtoolkit.examples.maintenanceDemoN as maint
  then call the functions shown with maint. in front (e.g. maint.maintTest(...)).
• import RLtoolkit.gridworld.gwDemoN as gw
  then call the functions shown with gw. in front (e.g. gw.gwInit(...))
• To use the demos function for easy access to the demos, do ONE of the following:
  
• Using IDLE, open the demo.py file from within the RLtoolkit folder, and run it (F5). This is the recommended way to run the demos. Then you can do:
• demos() - prints a list of available demos
• demos('demoname') - prints information about the specific demo
• demos('demoname', 'run') - loads the specific demo. If it is a GUI demo it will start up automatically. If it is not, help information on how to run it will be printed. 
  
• If you have imported the whole toolkit (from RLtoolkit import *), the demos function is available in module demo.
  
• from RLtoolkit import *
  
• demo.demos() - prints a list of available demos
• demo.demos('demoname') - prints information about the specific demo
• demo.demos('demoname', 'run') - loads the specific demo. If it is a GUI demo it will start up automatically. If it is not, help information on how to run it will be printed.  You must prefix the commands shown with demo. (e.g. demo.mcInit())
• Import the demos function from the file demo as follows:

  import RLtoolkit.demo as demo

  use the demos function as described above (e.g. demo.demos(...))
  

• If you have imported the whole toolkit (from RLtoolkit.guiuser import *), the demos function is available with no module prefix. Note that the nongui demos commands may not be available. The GUI ones should work though.
  
• from RLtoolkit.guiuser import *
• demos() - prints a list of available demos
• demos('demoname') - prints information about the specific demo
• demos('demoname', 'run') - loads the specific demo. If it is a GUI demo it will start up automatically. If it is not, help information on how to run it will be printed.  

version 1.0b6 only provides __init__.py.bin, not the __init__.py (or *.pyc) files, in the directories and therefore cannot be used under Windows. I had to switch to 1.0b5, which at least has *.pyc files. I'd appreciate if someone could include the init functions. Thanks!  

for the missing __init__.py files, all you need to do is to create empty files with that name in each directory.