Autoregressive (AR) models predict future values in a sequence based on a linear combination of previous, past values, acting as a regression of a variable against itself.
An AR model estimates the current value by using weighted sums of previous values () plus a stochastic shock/error term .
Examples and Applications
- Time Series Forecasting: Used to forecast economic indicators, stock prices, and weather patterns.
- Natural Language Processing (NLP): Large Language Models (LLMs) are autoregressive, generating text by predicting the next token based on previous tokens.
Limitations
- Error Propagation: Since they rely on past predictions, a small mistake early in a sequence can lead to compounding errors.
- Computational Cost: Generating long sequences can be slow because each step depends on the completion of the previous one.