Learning Hierarchical Control via Feasible Subgoal Prediction by Utsav Singh

Venue: SIT001

Abstract: Solving long-horizon tasks remains a central challenge in robotics because agents must explore efficiently, assign credit over long time scales, and act under sparse supervision. To address this, hierarchical reinforcement learning (HRL) offers a promising alternative to flat reinforcement learning (RL) by enabling a high-level policy to propose subgoals and a low-level policy to execute them. However, in practice, HRL suffers from a fundamental issue: the higher level can propose subgoals that are infeasible for the lower level to achieve, leading to training instability and sub-optimal performance. In this talk, I will present my research around a central idea: hierarchy is effective only when its high-level decisions are grounded in the capabilities of the lower-level policies. I will discuss methods for training high-level policies to predict feasible subgoals by leveraging expert demonstrations, preference-based feedback, and visually grounded reward synthesis. Across challenging navigation and manipulation tasks, these approaches improve training stability, mitigate non-stationarity, and enable agents to solve complex sparse-reward tasks in both simulation and real-world robotic settings. More broadly, this work argues that building intelligent robotic agents that can solve long-horizon tasks is not just a matter of better planning, but of ensuring that high-level decisions remain aligned with what lower-level policies can actually achieve.

Bio: Utsav Singh received his Ph.D. from the Department of Computer Science and Engineering at the Indian Institute of Technology (IIT) Kanpur, advised by Dr. Vinay P. Namboodiri and Dr. Sunil E. Simon. He will be joining the University of Central Florida as a Postdoctoral Researcher in June 2026, working with Dr. Mubarak Shah and Dr. Amrit Singh Bedi. Prior to his Ph.D., he received his M.Tech. from IIT Kanpur. His research interests span hierarchical reinforcement learning (HRL), embodied intelligence, and bilevel optimization, with a current focus on enabling language-guided robotic control and improving reasoning in large language models (LLMs) via bilevel actor-critic frameworks. His research has been published at leading venues including ICML, NeurIPS, ICLR, and AAAI.