Artificial intelligence
Active preference learning
Definition
Active preference learning lets a robot choose which comparisons to ask a person so it can infer a reward or preference model with fewer, more informative answers. Its usefulness depends on the query model, the alternatives shown, and whether the person's answers remain meaningful for the deployed task.
Also known as: Active preference-based learning
Updated
Ask comparisons that reduce uncertainty
Preference learning replaces a request for a numeric reward with a comparison such as “Which of these two trajectories is better?” Active preference learning also chooses the comparison. The learner maintains uncertainty over possible reward functions or preference parameters and selects a query expected to remove useful uncertainty.
Sadigh and colleagues synthesised pairs of dynamically feasible trajectories and asked people to choose between them. Their method selected queries by the expected volume removed from a weighted hypothesis space, and their driving study compared it with non-active alternatives.
Put human judgement into robot learning
A robot can query choices between arm motions, grasp strategies, navigation paths, or assistance behaviours. The answers can fit a reward used by reinforcement learning, planning, or shared autonomy. This differs from inverse reinforcement learning, which commonly infers preferences from demonstrations rather than deliberately chosen comparisons.
The question itself matters. A 2026 preprint by Bobu and colleagues targets queries at weakly supervised reward features and adds explanations directing the person's attention to those features. The authors report simulation and user-study improvements over their active-learning baseline. Those results support that particular query strategy and study, not every explanation or reward-learning system.
A selected query can still be a poor question
The method can optimise only the uncertainty represented by its reward model and candidate-query generator. If both alternatives omit an important behaviour, no answer can reveal it. Queries can also be hard to compare, physically unsafe to execute, repetitive, or biased by how options are presented.
Human preferences may change with context, fatigue, or new information. A model that fits earlier answers can therefore become stale or internally inconsistent. Designers need stopping rules, checks for disagreement, a way to reject unclear comparisons, and validation on complete robot behaviour rather than query efficiency alone.
Sources
Related terms
Inverse reinforcement learning
Inverse reinforcement learning infers a reward or cost function that could explain demonstrated behaviour in a known or estimated decision process. A policy can then be planned or learned from the inferred objective.
Reward shaping
Reward shaping adds supplementary rewards to guide reinforcement learning toward useful behavior. Poorly chosen shaping can change which policy is optimal, so an easier training signal is not automatically equivalent to the original task objective.
Reinforcement learning
Reinforcement learning trains an agent to choose actions that maximize expected cumulative reward through experience with an environment. In robotics, the learned policy can select movements or higher-level behaviors from observations.
Shared autonomy
Shared autonomy is an interaction mode in which a human and a robot jointly influence the robot's actions. The robot interprets human input and supplies assistance while leaving some control or decision authority with the person.