You are here

Robust Learning for Adaptive Programs by Leveraging Program Structure

TitleRobust Learning for Adaptive Programs by Leveraging Program Structure
Publication TypeConference Paper
Year of Publication2010
AuthorsPinto, J., A. Fern, T. Bauer, and M. Erwig
Conference NameInternational Conference on Machine Learning and Applications (ICMLA)
Pagination943 - 948
Date Published12/2010
Conference LocationWashington, DC
ISBN Number978-1-4244-9211-4
Keywordsadaptation-based programming, partial programming, reinforcement learning

We study how to effectively integrate reinforcement learning (RL) and programming languages via adaptation-based programming, where programs can include non-deterministic structures that can be automatically optimized via RL. Prior work has optimized adaptive programs by defining an induced sequential decision process to which standard RL is applied. Here we show that the success of this approach is highly sensitive to the specific program structure, where even seemingly minor program transformations can lead to failure. This sensitivity makes it extremely difficult for a non-RL-expert to write effective adaptive programs. In this paper, we study a more robust learning approach, where the key idea is to leverage information about program structure in order to define a more informative decision process and to improve the SARSA(λ) RL algorithm. Our empirical results show significant benefits for this approach.