PaperBank

	Reinforcement Learning and Artificial Intelligence (RLAI)
	Xin X, He H. and Hu D. "Efficient Reinforcement Learning Using Recursive Least-Squares Methods", Journal of Artificial Intelligence Research, Vol.16,2002, pp:259-292

Download Here

Abstract:

The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD( lambda ) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. ...

Keywords: Reinforcement Learning, Least-Square Methods, Value-Function Approximation, Least Square Methods.

Bibtex:

@inproceedings{ xu02efficient,
    author = "Xin Xu and Han-gen He and Dewen Hu",
    title = "Efficient Reinforcement Learning Using Recursive Least-Squares Methods",
    booktitle = "Journal of Artificial Intelligence Research",
    pages = "259-292",
    year = "2002",
    volume = "16",
    url = "citeseer.ist.psu.edu/article/xu02efficient.html" }

Comments:

Dynamic programming is part of RL. Also RL includes all cases: with model and without model.
As lambda increases the error bound would be reduced. good to have in mind.
Why there is not much comparison between RLSTD and LSTD? Does it mean they are the same aside from the fact that one is recursive version of the other?
The tasks investigated in this paper are stationary, so why they also used \mu \neq 1?
The Boyan example has a relatively high SNR? how can one comment about the SNR of a problem?
What is the main difference of RLS-TD And LSTD only the recursive computation of matrix?
The effect of initialization seems huge. It is similar to the alpha parameter selection with iLSTD method when averaging the TD errors by number of steps.
According to the paper the internal link between actor and critic is internal reward which seems to me to be something we want to minimize not maximize. The world utility might be a better option.
\partial J/ \partial a (46) seems odd to me. Is it computable?
What is the time difference of fast-AHC and AHC? They should be different in order of magnitude w.r.t #of features.
They could have used SVD in order to use LSTD in first runs of AHC. (Alborz)