A machine learning paradigm where models learn to generate their own labels from unlabeled data, bridging supervised and unsupervised learning.
How It Works
- Pretext Tasks: The model is trained to solve an auxiliary task that requires understanding the structure of the data, such as predicting obscured parts of a sentence or image (e.g., masking).
- Pseudo-labels: The algorithm generates target labels directly from the input data (e.g., predicting the next word, predicting color from grayscale)
- Representation Learning: The primary goal is to learn meaningful, high-quality feature representations from vast amounts of raw data.