A self-supervised Representation Learning method from DeepMind (2020) that learns without negative pairs.
Two networks: an online network and a target network. The target is not trained by backprop. Instead, it’s an exponential moving average (EMA) of the online network:
Training objective
Take two augmented views of the same image. Online network predicts the target network’s representation of the other view. Minimize their cosine similarity loss.
Why it doesn’t collapse
The EMA target acts as a slowly-moving “teacher” that’s always slightly ahead of the online network, creating a stable training signal. The asymmetry (only online has the predictor) + stop gradient + EMA together prevent collapse empirically, though the theoretical explanation is still debated.