Reinforcement Learning Benchmarks and Bake-offs
Friday December 17, 2004
http://rlbb.rlai.net
7:30 Benchmarks 1
Introduction (Rich Sutton/Satinder Singh) (30 minutes)
Panel discussion: Prior experience with benchmarks (40 minutes)
Satinder Singh
Mehryar Mohri (bandit problems)
Bill Smart (SourceForge model)
Michael Buro (TIELT)
Proposals
Martin Reidmiller (20 minutes)
"A software framework for RL benchmarking"
8:50 Coffee break
9:10 Benchmarks 2
More Proposals
Drew Bagnell and John Langford (20 minutes)
"RLbench: A benchmark suite for reinforcement learning"
Satinder Singh (10 minutes)
Mehryar Mohri (10 minutes)
"Multi-Armed Bandit Algorithms and Empirical Evaluation"
Michael Buro (30 minutes
"Open software for real-time strategy games as RL benchmarks"
Vincent Corruble (10 minutes)
"Games and multi-agent simulations, towards benchmarks for
reinforcement learning"
10:30 morning session ends
4:00 Competitions
Introduction (Michael Littman) (15 minutes)
"Should reinforcement learning have competitions?"
Panel discussion: Prior experience with competitions (40 minutes)
Will Uther (RoboCup)
Bill Smart (AAAI Robot Competition)
Satinder Singh (Trading Agent Competition)
Michael Littman (International Planning Competition)
Proposals (Discussion) (30 minutes)
5:20 Coffee break
5:40 Action items (benchmarks and competitions)
Detailed, specific proposals
Decisions
Assignments
