You are here

Automatic discovery and transfer of MAXQ hierarchies

TitleAutomatic discovery and transfer of MAXQ hierarchies
Publication TypeConference Paper
Year of Publication2008
AuthorsMehta, N., S. Ray, P. Tadepalli, and T. G. Dietterich
Tertiary AuthorsCohen, W., A. McCallum, and S. Roweis
Conference NameProceedings of the 25th International Conference on Machine learning - ICML '08
Pagination648 - 655
Date Published07/2008
PublisherACM Press
Conference LocationHelsinki, Finland
ISBN Number9781605582054

We present an algorithm, HI-MAT (Hierarchy Induction via Models And Trajectories), that discovers MAXQ task hierarchies by applying dynamic Bayesian network models to a successful trajectory from a source reinforcement learning task. HI-MAT discovers subtasks by analyzing the causal and temporal relationships among the actions in the trajectory. Under appropriate assumptions, HI-MAT induces hierarchies that are consistent with the observed trajectory and have compact value-function tables employing safe state abstractions. We demonstrate empirically that HI-MAT constructs compact hierarchies that are comparable to manually-engineered hierarchies and facilitate significant speedup in learning when transferred to a target task.