Artificial intelligence
Hindsight experience replay
Definition
Hindsight experience replay is an off-policy reinforcement-learning technique that relabels a failed trajectory with a goal it actually achieved, then reuses that trajectory as successful experience for the relabelled goal. It helps learn goal-conditioned behaviour when rewards are sparse.
Also known as: HER
Updated
Learn from an achieved goal
A goal-conditioned policy may receive almost no positive reward early in training. HER stores a transition with its original goal, then creates additional training examples by substituting a state that the trajectory later achieved as the goal. The reward is recomputed for that substituted goal.
The original HER paper combines this relabelling with off-policy reinforcement learning. Its authors report simulated pushing, sliding, and pick-and-place experiments with sparse binary rewards, followed by transfer of trained policies to a physical robot.
For example, a robot that misses the requested object location still reaches some location. Relabelling that endpoint as the goal teaches which actions can reach it, instead of discarding the whole attempt as an undifferentiated failure.
Not ordinary replay or reward shaping
Standard experience replay samples stored transitions again with their original task labels. HER changes the goal attached to selected samples and recalculates the reward. It is compatible with off-policy learning because the learner can train on actions collected under another goal.
HER also differs from manually designed reward shaping. It can retain a sparse success test, but it changes the distribution of goals represented in training. The relabelling strategy, such as choosing future states from the same episode, is therefore part of the method.
Relabelled goals must still be valid
The technique assumes that an achieved state can be represented as a legitimate goal and that the reward can be recomputed. This is straightforward for some geometric manipulation goals but harder for language instructions, safety constraints, or tasks whose history matters.
Biased goal sampling can overrepresent easy outcomes, and relabelled successes do not remove exploration or sim-to-real problems. The original experiments support HER in their evaluated tasks; they do not show that relabelling alone solves sparse-reward learning for long-horizon or contact-sensitive robot behaviour.
Sources
Related terms
Reinforcement learning
Reinforcement learning trains an agent to choose actions that maximize expected cumulative reward through experience with an environment. In robotics, the learned policy can select movements or higher-level behaviors from observations.
Reward shaping
Reward shaping adds supplementary rewards to guide reinforcement learning toward useful behavior. Poorly chosen shaping can change which policy is optimal, so an easier training signal is not automatically equivalent to the original task objective.
Offline reinforcement learning
Offline reinforcement learning learns a reward-optimizing policy from previously collected experience without gathering new environment interactions during that learning stage. The data may come from earlier policies, demonstrations, or other collection procedures.
Robotic manipulation
Robotic manipulation is the use of a robot to change an object's position, orientation, or state through physical interaction. It includes grasping and moving objects as well as actions such as pushing or carrying them without a grasp.