Artificial intelligence
Inverse reinforcement learning
Definition
Inverse reinforcement learning infers a reward or cost function that could explain demonstrated behaviour in a known or estimated decision process. A policy can then be planned or learned from the inferred objective.
Also known as: Inverse RL, IRL
Updated
Work backwards from behaviour to objective
Ordinary reinforcement learning starts with a reward and seeks a policy. Inverse reinforcement learning starts with observed behaviour and asks which reward could make that behaviour preferable under the environment's dynamics.
Ng and Russell formulate algorithms for recovering reward functions from an optimal policy or observed trajectories in Markov decision processes. Once a reward is inferred, a planner or learner can use it in states that were not present in the demonstrations.
For a robot, the demonstrations might show how a person drives, reaches, or navigates. The intended result is an objective that explains trade-offs, rather than only a mapping from each observed state to the demonstrated action.
Reward inference differs from copying actions
Behavior cloning directly learns an action predictor from examples. IRL introduces an extra modelling step: infer a reward, then solve a control problem. That can support new initial states, but it also requires assumptions about the dynamics, the demonstrator, and how policies are evaluated.
IRL is also different from manual reward shaping. The reward is inferred from behaviour, although a designer still chooses its feature representation, prior, and ambiguity-handling method.
Demonstrations do not identify one true reward
Many reward functions can make the same policy optimal. Adding a constant, rescaling a reward, or changing rewards in states the demonstrator never visits may leave the observed choices unchanged. A suboptimal or inconsistent demonstrator creates further uncertainty.
The inferred reward can reproduce undesirable correlations in the examples and does not establish human intent or safety. It must be evaluated through the policies it induces, especially when a robot can reach states outside the demonstration distribution.
Sources
Related terms
Reinforcement learning
Reinforcement learning trains an agent to choose actions that maximize expected cumulative reward through experience with an environment. In robotics, the learned policy can select movements or higher-level behaviors from observations.
Imitation learning
Imitation learning learns behavior from examples supplied by a demonstrator. In robotics, demonstrations can teach a policy how to perform a task without requiring every action or objective to be programmed by hand.
Reward shaping
Reward shaping adds supplementary rewards to guide reinforcement learning toward useful behavior. Poorly chosen shaping can change which policy is optimal, so an easier training signal is not automatically equivalent to the original task objective.
Behavior cloning
Behavior cloning is an imitation-learning method that trains a policy to predict a demonstrator’s actions from recorded observations or states. It treats action prediction as a supervised-learning problem.