Reinforcement Learning and Artificial Intelligence (RLAI)

Mini-project information

Course Mini-project

items due:

1-page proposal	0 points	due Tuesday, April 4
3-page writeup + graphs	100 points	due Thursday, April 13

The idea in this project is to give you experience and training in how to formulate, test, and report results on a research hypothesis in the context of reinforcement learning. you will formulate a small, specific hypothesis, a research question, that is of some interest and that can be answered or significantly addressed in a simple computational experiment. you will then design and run that experiment and report the results. i am most interested in seeing you design a simple and clean experiment. do not try to do too much. do not try to resolve a fundamental question of reinforcement learning. but do try to do something. it is very important not to over claim or to be unclear about what you are claiming. you must delineate your claim -- establish matter of factly what it is and why it might be considered interesting. then explain your experimental design and how it bears on the question. finally, present your results and discuss their relationship to your question and to the broader issues that motivated the question.

your proposal should state the research hypothesis and an experimental design. the proposal is worth zero points but you must turn it in. essentially, it is for your benefit. i will look at your proposed hypothesis and experiment and advise you as to their clarity and feasibility. if you want to suggest more than one hypothesis-experiment pair, to get a recommendation as to which might work out better, that's fine, but keep the whole thing to less than one page.

your presentation (writeup) should contain the following parts in this order. here i am assuming you are running a comparison of learning algorithms on specific tasks. this is of course only a guide. make adjustments as needed.

a brief discussion of general issues motivating and leading, as quickly as possible, to your question.
a brief discussion of prior work, to the extent that you have not already done it, as part of further motivating your question and specific experiment.
an overview of the design of the experiment and the algorithms you will use. address how the experiment relates to the hypothesis, what you can and do learn from it.
now you are ready to get down to brass tacks, to present all the specific details of your experiment. the standard here is that you must specify everything -- everything sufficient that another knowledgeable researcher, say another student in the class, could reproduce your experiment and get exactly the same results. And you must do this very compactly and concisely. remember that your whole report should be no longer than three pages long, plus figures. this will require some advance planning to make sure that you can describe your experiment in a small space and still include all the details. be mentally prepared to run your experiment and then, when it comes time to write it up, realize that it would be sleeker and shorter if you had done the experiment a little differently. the right thing to do in this case is simply to do the experiment again. it happens all the time. be prepared. leave some time for it. if your experiment is clean and appropriately small (and you have left some time for this, then re-running should be no big deal, just some more work for the computer. OK, this bullet point is not really a section of your report, just a comment. now back to specifying contents of your report.
a complete specification of the task(s), the environment(s) you will apply your algorithms to. do not specify details of the experiment, just the environment(s).
details of the algorithms you will use in the experiment. parameter settings. algorithm variations.
details of the experiment. how many steps, episodes, and runs?
what data will be collected? what performance measures will be computed?
now present the results, in tables or graphs. what are the axes on the graphs? avoid those ugly "unix graphs" if you possibly can. and eschew legends. label the gosh darn lines so that it's not an exercise in visual cognition to figure out which goes with which.
make your final factual statements about the results. everything from the brass tacks through here should be written in past tense. it's what actually happened.
now you switch to present tense to make conclusions and draw inferences. what has been shown and not shown. have you answered your question? how much progress have you made, or not made, toward it?
what have you learned?

no more than three pages. less is more.